visual analysis research questions

Perhaps what is missing in this context are the children’s direct voices and their own meaning-making through their work. As the dialogue with the teacher participants sometimes, takes place a period after the creation of their artwork and there was insufficient opportunity to take the time to dialogue directly with the children but rather to learn about the process through the teacher participants’ perspective at this stage. The meaning-making process here considers mainly the interpretation of the teacher and the researcher. The fact is that images should be acknowledged to be multi-vocal, having the ability to ‘speak’ to different audiences in a variety of contexts ( ; ).

In current times, digital media has reached into our mundane everyday existences, most obviously through the cell phone and modern-day gadgets, social media and these digital artefacts have engendered new forms of ethnographic enquiry. One of these includes what might be termed as the cultural politics of media and examines cultural identities, representations and imaginaries ( ). Fleer & Ridgway, (2014) outline and frame visual narrative data based on cultural-historical theory. Cultural historical theory acknowledges that the characteristics of individuals engaged in activities and interactions within a certain cultural setting can evolve and transform over a period of history. This can enable the researcher a better understanding on why certain practices and needs are defined as they are in a specific context and that different perspectives and priorities are taken in different cultures and times ( ; ).

Though field observation, particularly in Reggio-Emilia inspired centres and where children are given free reign of their imagination through encouragement and access to materials, it has been observed that the young can use the graphic and expressive languages of drawing, painting, collage and construction to record their ideas, observations, reflections, memories to further explore their understanding. Embedded in these activities are the processes of reconstructing and building on earlier knowledge so as to externalize their thoughts and what is learnt, to share their worlds with their peers and others ( ; ). The approach using ‘art as epistemology’ ( ) so that art experiences in the classroom can have both communicative and expressive goals, and the concept of art as a symbolic language is the subject of much debate. This highlights the potential of teachers facilitating children to develop the capacity in the ‘hundred language’ that is accessible to them so as to master the range of instruments and symbols ( ) that form the visual culture and an expressive language used in the curriculum. The potential for research based on visual methodologies is thus boundless.

I would like to thank niec, National Institute of Early Childhood Development, Singapore and the following teacher participant contributors Chandra Rai, Shauna Chen and Kavita Mogan.

SAGE. .)| false , M. (2011). Presenting visual research. In , 105–128. SAGE. 10.4135/9781526445933.n5.

SAGE. 10.4135/9781526445933.n5.)| false , U. , & Morris, P. A. (2006). The Bio ecological Model of Human Development. In . John Wiley.

John Wiley.)| false , K. (2015). (Second Ed, Vol. ). Elsevier.

Elsevier.)| false , V. L. P. , & Ivankova, N. V. (2016). How to Expand the use of Mixed Methods Research?: Intersecting Mixed Methods with Other Approaches. In , 135–160.

SAGE.)| false , R. , Griffin, C. , Hunter, J. , & Phoenix, A. (2008). Young people’s constructions of self: Notes on the use and analysis of the photo-elicitation methods. , (4), 345–356. .

Oxford University Press.)| false , M. , & Ridgway, A. (2014). . .

SAGE.)| false , K. (1991). Arts as Epistemology: Enabling Children to Know What They Know. , (1), 40–51. .

Polity Press.)| false , J. (2013). Visual research methods in education: In between difference and indifference. , (2), 63–78.

SAGE.)| false , Sarah . (2014). . .

& (Eds.), , (pp. – ).

Harvard University Press.)| false , S. (2007). Young children’s meaning-making through drawing and ‘telling’: Analogies to filmic textual features, , (4), 37–49. .

• 3.2 Researcher and Participant-generated Visual Material
• 3.3 Visual/Photo-elicited Data
• 4 Summary and Conclusions
• Acknowledgements

Reference Works

Primary source collections

COVID-19 Collection

How to publish with Brill

Open Access Content

Contact & Info

Sales contacts

Publishing contacts

Stay Updated

Newsletters

Social Media Overview

Terms and Conditions  

Privacy Statement  

Cookie Settings  

Accessibility

Legal Notice

Terms and Conditions   |   Privacy Statement   |  Cookie Settings   |   Accessibility   |  Legal Notice   |  Copyright © 2016-2024

Copyright © 2016-2024

• [91.193.111.216]
• 91.193.111.216

Character limit 500 /500

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Korean Med Sci
• v.37(16); 2022 Apr 25

A Practical Guide to Writing Quantitative and Qualitative Research Questions and Hypotheses in Scholarly Articles

Edward barroga.

1 Department of General Education, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan.

Glafera Janet Matanguihan

2 Department of Biological Sciences, Messiah University, Mechanicsburg, PA, USA.

The development of research questions and the subsequent hypotheses are prerequisites to defining the main research purpose and specific objectives of a study. Consequently, these objectives determine the study design and research outcome. The development of research questions is a process based on knowledge of current trends, cutting-edge studies, and technological advances in the research field. Excellent research questions are focused and require a comprehensive literature search and in-depth understanding of the problem being investigated. Initially, research questions may be written as descriptive questions which could be developed into inferential questions. These questions must be specific and concise to provide a clear foundation for developing hypotheses. Hypotheses are more formal predictions about the research outcomes. These specify the possible results that may or may not be expected regarding the relationship between groups. Thus, research questions and hypotheses clarify the main purpose and specific objectives of the study, which in turn dictate the design of the study, its direction, and outcome. Studies developed from good research questions and hypotheses will have trustworthy outcomes with wide-ranging social and health implications.

INTRODUCTION

Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses. 1 , 2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results. 3 , 4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the inception of novel studies and the ethical testing of ideas. 5 , 6

It is crucial to have knowledge of both quantitative and qualitative research 2 as both types of research involve writing research questions and hypotheses. 7 However, these crucial elements of research are sometimes overlooked; if not overlooked, then framed without the forethought and meticulous attention it needs. Planning and careful consideration are needed when developing quantitative or qualitative research, particularly when conceptualizing research questions and hypotheses. 4

There is a continuing need to support researchers in the creation of innovative research questions and hypotheses, as well as for journal articles that carefully review these elements. 1 When research questions and hypotheses are not carefully thought of, unethical studies and poor outcomes usually ensue. Carefully formulated research questions and hypotheses define well-founded objectives, which in turn determine the appropriate design, course, and outcome of the study. This article then aims to discuss in detail the various aspects of crafting research questions and hypotheses, with the goal of guiding researchers as they develop their own. Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points.

DEFINITIONS AND RELATIONSHIP OF RESEARCH QUESTIONS AND HYPOTHESES

A research question is what a study aims to answer after data analysis and interpretation. The answer is written in length in the discussion section of the paper. Thus, the research question gives a preview of the different parts and variables of the study meant to address the problem posed in the research question. 1 An excellent research question clarifies the research writing while facilitating understanding of the research topic, objective, scope, and limitations of the study. 5

On the other hand, a research hypothesis is an educated statement of an expected outcome. This statement is based on background research and current knowledge. 8 , 9 The research hypothesis makes a specific prediction about a new phenomenon 10 or a formal statement on the expected relationship between an independent variable and a dependent variable. 3 , 11 It provides a tentative answer to the research question to be tested or explored. 4

Hypotheses employ reasoning to predict a theory-based outcome. 10 These can also be developed from theories by focusing on components of theories that have not yet been observed. 10 The validity of hypotheses is often based on the testability of the prediction made in a reproducible experiment. 8

Conversely, hypotheses can also be rephrased as research questions. Several hypotheses based on existing theories and knowledge may be needed to answer a research question. Developing ethical research questions and hypotheses creates a research design that has logical relationships among variables. These relationships serve as a solid foundation for the conduct of the study. 4 , 11 Haphazardly constructed research questions can result in poorly formulated hypotheses and improper study designs, leading to unreliable results. Thus, the formulations of relevant research questions and verifiable hypotheses are crucial when beginning research. 12

CHARACTERISTICS OF GOOD RESEARCH QUESTIONS AND HYPOTHESES

Excellent research questions are specific and focused. These integrate collective data and observations to confirm or refute the subsequent hypotheses. Well-constructed hypotheses are based on previous reports and verify the research context. These are realistic, in-depth, sufficiently complex, and reproducible. More importantly, these hypotheses can be addressed and tested. 13

There are several characteristics of well-developed hypotheses. Good hypotheses are 1) empirically testable 7 , 10 , 11 , 13 ; 2) backed by preliminary evidence 9 ; 3) testable by ethical research 7 , 9 ; 4) based on original ideas 9 ; 5) have evidenced-based logical reasoning 10 ; and 6) can be predicted. 11 Good hypotheses can infer ethical and positive implications, indicating the presence of a relationship or effect relevant to the research theme. 7 , 11 These are initially developed from a general theory and branch into specific hypotheses by deductive reasoning. In the absence of a theory to base the hypotheses, inductive reasoning based on specific observations or findings form more general hypotheses. 10

TYPES OF RESEARCH QUESTIONS AND HYPOTHESES

Research questions and hypotheses are developed according to the type of research, which can be broadly classified into quantitative and qualitative research. We provide a summary of the types of research questions and hypotheses under quantitative and qualitative research categories in Table 1 .

Research questions in quantitative research

In quantitative research, research questions inquire about the relationships among variables being investigated and are usually framed at the start of the study. These are precise and typically linked to the subject population, dependent and independent variables, and research design. 1 Research questions may also attempt to describe the behavior of a population in relation to one or more variables, or describe the characteristics of variables to be measured ( descriptive research questions ). 1 , 5 , 14 These questions may also aim to discover differences between groups within the context of an outcome variable ( comparative research questions ), 1 , 5 , 14 or elucidate trends and interactions among variables ( relationship research questions ). 1 , 5 We provide examples of descriptive, comparative, and relationship research questions in quantitative research in Table 2 .

Hypotheses in quantitative research

In quantitative research, hypotheses predict the expected relationships among variables. 15 Relationships among variables that can be predicted include 1) between a single dependent variable and a single independent variable ( simple hypothesis ) or 2) between two or more independent and dependent variables ( complex hypothesis ). 4 , 11 Hypotheses may also specify the expected direction to be followed and imply an intellectual commitment to a particular outcome ( directional hypothesis ) 4 . On the other hand, hypotheses may not predict the exact direction and are used in the absence of a theory, or when findings contradict previous studies ( non-directional hypothesis ). 4 In addition, hypotheses can 1) define interdependency between variables ( associative hypothesis ), 4 2) propose an effect on the dependent variable from manipulation of the independent variable ( causal hypothesis ), 4 3) state a negative relationship between two variables ( null hypothesis ), 4 , 11 , 15 4) replace the working hypothesis if rejected ( alternative hypothesis ), 15 explain the relationship of phenomena to possibly generate a theory ( working hypothesis ), 11 5) involve quantifiable variables that can be tested statistically ( statistical hypothesis ), 11 6) or express a relationship whose interlinks can be verified logically ( logical hypothesis ). 11 We provide examples of simple, complex, directional, non-directional, associative, causal, null, alternative, working, statistical, and logical hypotheses in quantitative research, as well as the definition of quantitative hypothesis-testing research in Table 3 .

Research questions in qualitative research

Unlike research questions in quantitative research, research questions in qualitative research are usually continuously reviewed and reformulated. The central question and associated subquestions are stated more than the hypotheses. 15 The central question broadly explores a complex set of factors surrounding the central phenomenon, aiming to present the varied perspectives of participants. 15

There are varied goals for which qualitative research questions are developed. These questions can function in several ways, such as to 1) identify and describe existing conditions ( contextual research question s); 2) describe a phenomenon ( descriptive research questions ); 3) assess the effectiveness of existing methods, protocols, theories, or procedures ( evaluation research questions ); 4) examine a phenomenon or analyze the reasons or relationships between subjects or phenomena ( explanatory research questions ); or 5) focus on unknown aspects of a particular topic ( exploratory research questions ). 5 In addition, some qualitative research questions provide new ideas for the development of theories and actions ( generative research questions ) or advance specific ideologies of a position ( ideological research questions ). 1 Other qualitative research questions may build on a body of existing literature and become working guidelines ( ethnographic research questions ). Research questions may also be broadly stated without specific reference to the existing literature or a typology of questions ( phenomenological research questions ), may be directed towards generating a theory of some process ( grounded theory questions ), or may address a description of the case and the emerging themes ( qualitative case study questions ). 15 We provide examples of contextual, descriptive, evaluation, explanatory, exploratory, generative, ideological, ethnographic, phenomenological, grounded theory, and qualitative case study research questions in qualitative research in Table 4 , and the definition of qualitative hypothesis-generating research in Table 5 .

Qualitative studies usually pose at least one central research question and several subquestions starting with How or What . These research questions use exploratory verbs such as explore or describe . These also focus on one central phenomenon of interest, and may mention the participants and research site. 15

Hypotheses in qualitative research

Hypotheses in qualitative research are stated in the form of a clear statement concerning the problem to be investigated. Unlike in quantitative research where hypotheses are usually developed to be tested, qualitative research can lead to both hypothesis-testing and hypothesis-generating outcomes. 2 When studies require both quantitative and qualitative research questions, this suggests an integrative process between both research methods wherein a single mixed-methods research question can be developed. 1

FRAMEWORKS FOR DEVELOPING RESEARCH QUESTIONS AND HYPOTHESES

Research questions followed by hypotheses should be developed before the start of the study. 1 , 12 , 14 It is crucial to develop feasible research questions on a topic that is interesting to both the researcher and the scientific community. This can be achieved by a meticulous review of previous and current studies to establish a novel topic. Specific areas are subsequently focused on to generate ethical research questions. The relevance of the research questions is evaluated in terms of clarity of the resulting data, specificity of the methodology, objectivity of the outcome, depth of the research, and impact of the study. 1 , 5 These aspects constitute the FINER criteria (i.e., Feasible, Interesting, Novel, Ethical, and Relevant). 1 Clarity and effectiveness are achieved if research questions meet the FINER criteria. In addition to the FINER criteria, Ratan et al. described focus, complexity, novelty, feasibility, and measurability for evaluating the effectiveness of research questions. 14

The PICOT and PEO frameworks are also used when developing research questions. 1 The following elements are addressed in these frameworks, PICOT: P-population/patients/problem, I-intervention or indicator being studied, C-comparison group, O-outcome of interest, and T-timeframe of the study; PEO: P-population being studied, E-exposure to preexisting conditions, and O-outcome of interest. 1 Research questions are also considered good if these meet the “FINERMAPS” framework: Feasible, Interesting, Novel, Ethical, Relevant, Manageable, Appropriate, Potential value/publishable, and Systematic. 14

As we indicated earlier, research questions and hypotheses that are not carefully formulated result in unethical studies or poor outcomes. To illustrate this, we provide some examples of ambiguous research question and hypotheses that result in unclear and weak research objectives in quantitative research ( Table 6 ) 16 and qualitative research ( Table 7 ) 17 , and how to transform these ambiguous research question(s) and hypothesis(es) into clear and good statements.

a These statements were composed for comparison and illustrative purposes only.

b These statements are direct quotes from Higashihara and Horiuchi. 16

a This statement is a direct quote from Shimoda et al. 17

The other statements were composed for comparison and illustrative purposes only.

CONSTRUCTING RESEARCH QUESTIONS AND HYPOTHESES

To construct effective research questions and hypotheses, it is very important to 1) clarify the background and 2) identify the research problem at the outset of the research, within a specific timeframe. 9 Then, 3) review or conduct preliminary research to collect all available knowledge about the possible research questions by studying theories and previous studies. 18 Afterwards, 4) construct research questions to investigate the research problem. Identify variables to be accessed from the research questions 4 and make operational definitions of constructs from the research problem and questions. Thereafter, 5) construct specific deductive or inductive predictions in the form of hypotheses. 4 Finally, 6) state the study aims . This general flow for constructing effective research questions and hypotheses prior to conducting research is shown in Fig. 1 .

Research questions are used more frequently in qualitative research than objectives or hypotheses. 3 These questions seek to discover, understand, explore or describe experiences by asking “What” or “How.” The questions are open-ended to elicit a description rather than to relate variables or compare groups. The questions are continually reviewed, reformulated, and changed during the qualitative study. 3 Research questions are also used more frequently in survey projects than hypotheses in experiments in quantitative research to compare variables and their relationships.

Hypotheses are constructed based on the variables identified and as an if-then statement, following the template, ‘If a specific action is taken, then a certain outcome is expected.’ At this stage, some ideas regarding expectations from the research to be conducted must be drawn. 18 Then, the variables to be manipulated (independent) and influenced (dependent) are defined. 4 Thereafter, the hypothesis is stated and refined, and reproducible data tailored to the hypothesis are identified, collected, and analyzed. 4 The hypotheses must be testable and specific, 18 and should describe the variables and their relationships, the specific group being studied, and the predicted research outcome. 18 Hypotheses construction involves a testable proposition to be deduced from theory, and independent and dependent variables to be separated and measured separately. 3 Therefore, good hypotheses must be based on good research questions constructed at the start of a study or trial. 12

In summary, research questions are constructed after establishing the background of the study. Hypotheses are then developed based on the research questions. Thus, it is crucial to have excellent research questions to generate superior hypotheses. In turn, these would determine the research objectives and the design of the study, and ultimately, the outcome of the research. 12 Algorithms for building research questions and hypotheses are shown in Fig. 2 for quantitative research and in Fig. 3 for qualitative research.

EXAMPLES OF RESEARCH QUESTIONS FROM PUBLISHED ARTICLES

• EXAMPLE 1. Descriptive research question (quantitative research)
• - Presents research variables to be assessed (distinct phenotypes and subphenotypes)
• “BACKGROUND: Since COVID-19 was identified, its clinical and biological heterogeneity has been recognized. Identifying COVID-19 phenotypes might help guide basic, clinical, and translational research efforts.
• RESEARCH QUESTION: Does the clinical spectrum of patients with COVID-19 contain distinct phenotypes and subphenotypes? ” 19
• EXAMPLE 2. Relationship research question (quantitative research)
• - Shows interactions between dependent variable (static postural control) and independent variable (peripheral visual field loss)
• “Background: Integration of visual, vestibular, and proprioceptive sensations contributes to postural control. People with peripheral visual field loss have serious postural instability. However, the directional specificity of postural stability and sensory reweighting caused by gradual peripheral visual field loss remain unclear.
• Research question: What are the effects of peripheral visual field loss on static postural control ?” 20
• EXAMPLE 3. Comparative research question (quantitative research)
• - Clarifies the difference among groups with an outcome variable (patients enrolled in COMPERA with moderate PH or severe PH in COPD) and another group without the outcome variable (patients with idiopathic pulmonary arterial hypertension (IPAH))
• “BACKGROUND: Pulmonary hypertension (PH) in COPD is a poorly investigated clinical condition.
• RESEARCH QUESTION: Which factors determine the outcome of PH in COPD?
• STUDY DESIGN AND METHODS: We analyzed the characteristics and outcome of patients enrolled in the Comparative, Prospective Registry of Newly Initiated Therapies for Pulmonary Hypertension (COMPERA) with moderate or severe PH in COPD as defined during the 6th PH World Symposium who received medical therapy for PH and compared them with patients with idiopathic pulmonary arterial hypertension (IPAH) .” 21
• EXAMPLE 4. Exploratory research question (qualitative research)
• - Explores areas that have not been fully investigated (perspectives of families and children who receive care in clinic-based child obesity treatment) to have a deeper understanding of the research problem
• “Problem: Interventions for children with obesity lead to only modest improvements in BMI and long-term outcomes, and data are limited on the perspectives of families of children with obesity in clinic-based treatment. This scoping review seeks to answer the question: What is known about the perspectives of families and children who receive care in clinic-based child obesity treatment? This review aims to explore the scope of perspectives reported by families of children with obesity who have received individualized outpatient clinic-based obesity treatment.” 22
• EXAMPLE 5. Relationship research question (quantitative research)
• - Defines interactions between dependent variable (use of ankle strategies) and independent variable (changes in muscle tone)
• “Background: To maintain an upright standing posture against external disturbances, the human body mainly employs two types of postural control strategies: “ankle strategy” and “hip strategy.” While it has been reported that the magnitude of the disturbance alters the use of postural control strategies, it has not been elucidated how the level of muscle tone, one of the crucial parameters of bodily function, determines the use of each strategy. We have previously confirmed using forward dynamics simulations of human musculoskeletal models that an increased muscle tone promotes the use of ankle strategies. The objective of the present study was to experimentally evaluate a hypothesis: an increased muscle tone promotes the use of ankle strategies. Research question: Do changes in the muscle tone affect the use of ankle strategies ?” 23

EXAMPLES OF HYPOTHESES IN PUBLISHED ARTICLES

• EXAMPLE 1. Working hypothesis (quantitative research)
• - A hypothesis that is initially accepted for further research to produce a feasible theory
• “As fever may have benefit in shortening the duration of viral illness, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response when taken during the early stages of COVID-19 illness .” 24
• “In conclusion, it is plausible to hypothesize that the antipyretic efficacy of ibuprofen may be hindering the benefits of a fever response . The difference in perceived safety of these agents in COVID-19 illness could be related to the more potent efficacy to reduce fever with ibuprofen compared to acetaminophen. Compelling data on the benefit of fever warrant further research and review to determine when to treat or withhold ibuprofen for early stage fever for COVID-19 and other related viral illnesses .” 24
• EXAMPLE 2. Exploratory hypothesis (qualitative research)
• - Explores particular areas deeper to clarify subjective experience and develop a formal hypothesis potentially testable in a future quantitative approach
• “We hypothesized that when thinking about a past experience of help-seeking, a self distancing prompt would cause increased help-seeking intentions and more favorable help-seeking outcome expectations .” 25
• “Conclusion
• Although a priori hypotheses were not supported, further research is warranted as results indicate the potential for using self-distancing approaches to increasing help-seeking among some people with depressive symptomatology.” 25
• EXAMPLE 3. Hypothesis-generating research to establish a framework for hypothesis testing (qualitative research)
• “We hypothesize that compassionate care is beneficial for patients (better outcomes), healthcare systems and payers (lower costs), and healthcare providers (lower burnout). ” 26
• Compassionomics is the branch of knowledge and scientific study of the effects of compassionate healthcare. Our main hypotheses are that compassionate healthcare is beneficial for (1) patients, by improving clinical outcomes, (2) healthcare systems and payers, by supporting financial sustainability, and (3) HCPs, by lowering burnout and promoting resilience and well-being. The purpose of this paper is to establish a scientific framework for testing the hypotheses above . If these hypotheses are confirmed through rigorous research, compassionomics will belong in the science of evidence-based medicine, with major implications for all healthcare domains.” 26
• EXAMPLE 4. Statistical hypothesis (quantitative research)
• - An assumption is made about the relationship among several population characteristics ( gender differences in sociodemographic and clinical characteristics of adults with ADHD ). Validity is tested by statistical experiment or analysis ( chi-square test, Students t-test, and logistic regression analysis)
• “Our research investigated gender differences in sociodemographic and clinical characteristics of adults with ADHD in a Japanese clinical sample. Due to unique Japanese cultural ideals and expectations of women's behavior that are in opposition to ADHD symptoms, we hypothesized that women with ADHD experience more difficulties and present more dysfunctions than men . We tested the following hypotheses: first, women with ADHD have more comorbidities than men with ADHD; second, women with ADHD experience more social hardships than men, such as having less full-time employment and being more likely to be divorced.” 27
• “Statistical Analysis
• ( text omitted ) Between-gender comparisons were made using the chi-squared test for categorical variables and Students t-test for continuous variables…( text omitted ). A logistic regression analysis was performed for employment status, marital status, and comorbidity to evaluate the independent effects of gender on these dependent variables.” 27

EXAMPLES OF HYPOTHESIS AS WRITTEN IN PUBLISHED ARTICLES IN RELATION TO OTHER PARTS

• EXAMPLE 1. Background, hypotheses, and aims are provided
• “Pregnant women need skilled care during pregnancy and childbirth, but that skilled care is often delayed in some countries …( text omitted ). The focused antenatal care (FANC) model of WHO recommends that nurses provide information or counseling to all pregnant women …( text omitted ). Job aids are visual support materials that provide the right kind of information using graphics and words in a simple and yet effective manner. When nurses are not highly trained or have many work details to attend to, these job aids can serve as a content reminder for the nurses and can be used for educating their patients (Jennings, Yebadokpo, Affo, & Agbogbe, 2010) ( text omitted ). Importantly, additional evidence is needed to confirm how job aids can further improve the quality of ANC counseling by health workers in maternal care …( text omitted )” 28
• “ This has led us to hypothesize that the quality of ANC counseling would be better if supported by job aids. Consequently, a better quality of ANC counseling is expected to produce higher levels of awareness concerning the danger signs of pregnancy and a more favorable impression of the caring behavior of nurses .” 28
• “This study aimed to examine the differences in the responses of pregnant women to a job aid-supported intervention during ANC visit in terms of 1) their understanding of the danger signs of pregnancy and 2) their impression of the caring behaviors of nurses to pregnant women in rural Tanzania.” 28
• EXAMPLE 2. Background, hypotheses, and aims are provided
• “We conducted a two-arm randomized controlled trial (RCT) to evaluate and compare changes in salivary cortisol and oxytocin levels of first-time pregnant women between experimental and control groups. The women in the experimental group touched and held an infant for 30 min (experimental intervention protocol), whereas those in the control group watched a DVD movie of an infant (control intervention protocol). The primary outcome was salivary cortisol level and the secondary outcome was salivary oxytocin level.” 29
• “ We hypothesize that at 30 min after touching and holding an infant, the salivary cortisol level will significantly decrease and the salivary oxytocin level will increase in the experimental group compared with the control group .” 29
• EXAMPLE 3. Background, aim, and hypothesis are provided
• “In countries where the maternal mortality ratio remains high, antenatal education to increase Birth Preparedness and Complication Readiness (BPCR) is considered one of the top priorities [1]. BPCR includes birth plans during the antenatal period, such as the birthplace, birth attendant, transportation, health facility for complications, expenses, and birth materials, as well as family coordination to achieve such birth plans. In Tanzania, although increasing, only about half of all pregnant women attend an antenatal clinic more than four times [4]. Moreover, the information provided during antenatal care (ANC) is insufficient. In the resource-poor settings, antenatal group education is a potential approach because of the limited time for individual counseling at antenatal clinics.” 30
• “This study aimed to evaluate an antenatal group education program among pregnant women and their families with respect to birth-preparedness and maternal and infant outcomes in rural villages of Tanzania.” 30
• “ The study hypothesis was if Tanzanian pregnant women and their families received a family-oriented antenatal group education, they would (1) have a higher level of BPCR, (2) attend antenatal clinic four or more times, (3) give birth in a health facility, (4) have less complications of women at birth, and (5) have less complications and deaths of infants than those who did not receive the education .” 30

Research questions and hypotheses are crucial components to any type of research, whether quantitative or qualitative. These questions should be developed at the very beginning of the study. Excellent research questions lead to superior hypotheses, which, like a compass, set the direction of research, and can often determine the successful conduct of the study. Many research studies have floundered because the development of research questions and subsequent hypotheses was not given the thought and meticulous attention needed. The development of research questions and hypotheses is an iterative process based on extensive knowledge of the literature and insightful grasp of the knowledge gap. Focused, concise, and specific research questions provide a strong foundation for constructing hypotheses which serve as formal predictions about the research outcomes. Research questions and hypotheses are crucial elements of research that should not be overlooked. They should be carefully thought of and constructed when planning research. This avoids unethical studies and poor outcomes by defining well-founded objectives that determine the design, course, and outcome of the study.

Disclosure: The authors have no potential conflicts of interest to disclose.

Author Contributions:

• Conceptualization: Barroga E, Matanguihan GJ.
• Methodology: Barroga E, Matanguihan GJ.
• Writing - original draft: Barroga E, Matanguihan GJ.
• Writing - review & editing: Barroga E, Matanguihan GJ.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Data Descriptor
• Open access
• Published: 20 November 2018

A dataset of clinically generated visual questions and answers about radiology images

• Jason J. Lau 1 ,
• Soumya Gayen 1 ,
• Asma Ben Abacha 1 &
• Dina Demner-Fushman 1  

Scientific Data volume  5 , Article number:  180251 ( 2018 ) Cite this article

17k Accesses

104 Citations

90 Altmetric

Metrics details

• Data mining
• Radiography

Radiology images are an essential part of clinical decision making and population screening, e.g., for cancer. Automated systems could help clinicians cope with large amounts of images by answering questions about the image contents. An emerging area of artificial intelligence, Visual Question Answering (VQA) in the medical domain explores approaches to this form of clinical decision support. Success of such machine learning tools hinges on availability and design of collections composed of medical images augmented with question-answer pairs directed at the content of the image. We introduce VQA-RAD, the first manually constructed dataset where clinicians asked naturally occurring questions about radiology images and provided reference answers. Manual categorization of images and questions provides insight into clinically relevant tasks and the natural language to phrase them. Evaluating with well-known algorithms, we demonstrate the rich quality of this dataset over other automatically constructed ones. We propose VQA-RAD to encourage the community to design VQA tools with the goals of improving patient care.

Machine-accessible metadata file describing the reported data (ISA-Tab format)

Similar content being viewed by others

MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain

Computer-aided diagnosis through medical image retrieval in radiology

A generalist vision–language foundation model for diverse biomedical tasks

Background & summary.

Visual question answering (VQA) is a computer vision and artificial intelligence (AI) problem that aims to answer questions about images. As more of medicine is digitized and medical data continues to grow, there is enormous opportunity for multimodal tools such as VQA to benefit patients, clinicians, and researchers. Radiology, with its wealth of images and textual reports, is a prime area where VQA could assist radiologists in reporting findings for a complicated patient or benefit trainees who have questions about the size of a mass or presence of a fracture. Many different techniques are applied to build VQA systems including computer vision, natural language processing, and deep learning. These systems need to be trained for the task and evaluated on large data collections consisting of images and pairs of questions asked about the images with corresponding answers. Although there has been great progress in image recognition in radiology 1 , the datasets that allowed this are not quite generalizable to VQA because none of the datasets have question-answer pairs directed at the images 2 , 3 . Deep learning models require vast amounts of data to approach abilities of humans and as a result, the most popular training datasets are crowdsourced with general public knowledge 4 – 8 or synthetically generated 9 , 10 . The images in most of these datasets were taken from Microsoft Common Objects in Context (MSCOCO) 11 . While researchers have tried balancing these large datasets 5 , 12 , bias in selection of images and questions makes it difficult to transfer models trained on these datasets to specific applications. For example, the creators of VizWiz 13 , the first VQA dataset designed from images taken by blind users and visual questions from those use cases, demonstrated challenges in using the state-of-the-art models trained on the larger VQA datasets to predict answers from their VizWiz blind users generated data.

While visual questions are prevalent in medicine, most communications between clinicians are not usually documented in the patient’s record or available for researchers. Medical education is one area where collections of visual questions exist. Through their training and continued medical education, clinicians will take exams testing their knowledge of medical images. However, these test questions often go beyond the scope of the image and require extra knowledge about epidemiology or next step treatments. Most exam questions may not be suitable for teaching machines to answer descriptive questions about images, but they still have a role for more abstract machine assisted question answering in the future.

In 2018, ImageCLEF-Med released a radiology dataset 14 and coordinated the first community-wide VQA challenge in medicine. While the dataset is an excellent starting point for the medical domain, several design issues prevent useful clinical applications. To overcome the lack of readily available natural visual questions, questions and answers were automatically generated from corresponding captions. Unfortunately, this resulted in many artificial questions that do not always make sense, to the point where a human could not reason what the questions were trying to ask. Another issue with the dataset is that images were automatically captured from PubMed Central articles. This automation may not help VQA support clinical tasks as the dataset included composites of a series of images and 3D reconstructions that are rarely useful for direct patient care. If VQA tools are to assist in clinical processes, the datasets need to be designed with aligned goals.

We introduce VQA-RAD, a manually constructed VQA dataset in radiology where questions and answers about images are naturally created and validated by clinicians. Trading off quantity of automatic generation, VQA-RAD is a high-quality design with only 60 hours of specialist contributions. Structured like other existing VQA datasets, each question can be answered with a single image alone. The images are a balanced sample from MedPix®, an open-access radiology archive of case reports and teaching cases. These images were presented to clinicians who wrote unguided questions they would ask a colleague or radiologist. To further explore the natural clinical language, clinicians also paraphrased questions in both free-form and template structures. All question-answer pairs are manually validated and categorized, helping to characterize clinically important areas of focus. The flow of building the VQA-RAD dataset is shown in Fig. 1 . We demonstrate the value of VQA-RAD and use cases by applying several well-known algorithms.

Flow Diagram of VQA-RAD build.

Image Selection

We sampled images from teaching cases in MedPix, https://medpix.nlm.nih.gov/ , an open-access database of radiology images and teaching cases. Our sampling criteria were as follows: (1) Only one image for each teaching case, so that all images represented unique patients. (2) All images are sharp enough to identify individual structures. (3) Images are clean of radiology markings, such as arrows or circles. (4) Images have captions that correspond to the image and are detailed enough to describe at least one structure.

Today, diagnostic imaging often contains stacks of images, which is a challenge that VQA tools will need to incorporate to be useful for assisting radiology. For a dataset just emerging into medicine, we wanted to start with simple one image and one question pairs, enabling comparisons with current existing VQA dataset structures and algorithms. To reduce bias of having multiple images of the same pathology, we chose to limit one image per case, since each case is a unique patient. In addition, MedPix images are stored in JPG format, which needs to be taken into consideration when using the images for teaching and for training AI approaches.

Captions include plane, modality, and image findings that were generated and reviewed by expert radiologists. In total, we selected 104 head axial single-slice CTs or MRIs, 107 chest x-rays, and 104 abdominal axial CTs. The balanced distribution from head, chest, and abdomen should help determine if visual questions differ for each organ system and if the algorithms perform differently on different regions.

Question Type and Answer Type Generation

In addition to categorizing images, we define several question and answer types. Broad question types were initially developed from a combination of using Task Directed Image Understanding Challenge (TDIUC) categories 12 , annotating radiology reports from MedPix, and categorizing CME questions from open-access domains such as MedPix, AuntMinnie ( https://www.auntminnie.com/ ), and Radiopaedia.org ( https://radiopaedia.org/ ). Question types were further refined after analyzing an initial sample of visual questions from annotators. The final taxonomy of medical VQA in our dataset is shown in Table 1 .

The tasks of assisting clinicians with radiology are different and range from support for those just learning the basics of reading a plain film x-ray, to those who are senior radiologists investigating rare presentations of diseases. We selected medical students and fellows who had at least their core clinical rotations completed because they will better understand how radiology is incorporated into every day clinical decision making and patient care. These trainees’ questions and tasks are closer to those of attending trainees, compared to students just starting medical school. However, we do expect that there will be additional needs from resident trainees as their personal skill in reading radiology images improves and they have more nuanced questions. For example, a junior trainee may ask if there is presence of a pneumothorax, while a senior, who can already recognize air in the lungs, may ask more questions about the size of the pneumothorax since this has implications for invasive treatment.

Questions and answers were generated by 15 volunteer clinical trainees using a web-interface developed for collecting the questions and the answers. We asked for volunteers from the clinical fellows from the NIH and students from class of 2018/2019 University of Massachusetts, classmates of one of the authors, Jason Lau. All participants had completed the core rotations of medical school, which typically occurs during the 3 rd year of school and exposes students to major fields of medicine such as surgery, internal medicine, neurology, etc. This ensures that all participants have basic clinical radiology reading skills and were exposed to a variety of settings where radiology was vital to the management of patients. Our participants had training from different regions of the U.S. and there was one board certified pathologist, three applying into ophthalmology, four applying into family medicine, two applying into internal medicine, two applying into emergency medicine, one applying into radiology, one applying into orthopedics, and one dermatology.

Question and Answer Generation

Participants enerated questions and answers in a two-part evaluation (shown in Fig. 1 ) from December 2017 to April 2018. Each participant reviewed at least 40 randomized images. For the first 20 images, participants provided “free-form” questions and answers without any restrictions. We instructed participants to create “free-form” question about the images by phrasing them in a natural way as if they are asking a colleague or another physician. The image alone had to be sufficient to answer the question and there should only be a single correct answer. We asked that answers to the visual questions be based off their level of knowledge. Since many of the participants were still in medical training, we provided captions with some image findings, plane, and modality information to provide additional ground truth reassurance.

For the next 20 images, participants were randomly paired and given another participant’s images and questions. They were asked to generate “rephrased” and “framed” questions based off the given “free-form” questions with corresponding image and caption. We asked the participants to paraphrase the question in a natural way and generate an answer that agreed with both the original and the paraphrased questions.

Participants generated “framed” questions by finding the closest question structure from a list of templates and filling in the blank spaces to retain the answer to the original questions.

QuestionAnswer and Question Type Validation

After completion of the evaluations, we used several methods to validate questions answer pairs and question types. During the paraphrasing part of the evaluation, participants answered another person’s questions. The answers could have strict or loose agreement. We defined strict agreement when the question and answer format and topic were the same. In loose agreement, the topic of the questions is the same or similar even though the answers may differ. Three subcategories of loose agreement are defined: inversion, conversion, and subsuming.

Examples of each as follows:

Inversion: Q1: “Are there abnormal findings in the lower lung fields?” is a negation of Q2: “Are the lower lung fields normal?”

Conversion : Q1: “How would you describe the abnormalities?” is open-ended while Q2: “Are the lesions ring-enhancing?” is closed-ended

Subsumption: Q1: “Is the heart seen in the image?” subsumes Q2: “is the heart seen on the left?”

We calculate F1 scores of user agreement for questions considered ‘evaluated’, meaning they were reviewed by two annotators. Disagreements amongst question category types were reviewed at weekly team meetings to determine general rules for all content. Disagreements in objective findings of images were reviewed by using MedPix metadata and advice from expert radiologist to find ground truth. This in-depth review was only required for 280 question-answer pairs. Questions are labeled as ‘not evaluated’ if they are not reviewed by a second participant or the paraphrased question is not similar enough to be used as validation. Both the evaluated and not evaluated questions are used as part of the test and training set.

We validated question types assigned by the participants. Final categorization was determined through consensus with the research team to resolve disagreements.

We anticipate that a VQA-RAD pipeline is extensible to produce more test data and training data for traditional machine learning approaches. We believe that more training data sufficient for training deep learning approaches could be produced automatically, bootstrapping from the VQA-RAD collection and using recent approaches to image synthesis and data augmentation 15 .

Data Records

The full dataset is archived with Open Science Framework ( Data Citation 1 ). All question-answer pairs referencing images are stored in a single dataset that we provide in three common formats: JSON, XML, and Excel. The dataset contains 14 variables of unique identifiers and categorization with a total of 2,248 elements. The 315 corresponding radiological images are contained in a separate folder. Naming conventions and descriptions of all files can be found in a provided Readme file.

Technical Validation

Analysis of questions and answers.

The final VQA-RAD dataset contains 3,515 total visual questions. Of these, 1,515 (43.1%) are free-form. 733 questions are rephrased which means 48.3% of the free-form questions have corresponding paraphrasing. The remaining 1,267 questions are framed and have corresponding free-form and rephrased questions. On average, there are 10 questions per image.

Of the free-form and rephrased questions, 75% (1,691) are evaluated using seven pairs of annotators. User agreement F1-scores range from 0.78 to 0.95 with a mean of 0.85.

All questions are short, with medians, mean, and modes ranging from 5 to 7 words per question. After converting all words to lowercase and comparing sentences for unique structures, the free-form and rephrased questions have more distinct questions (87% and 93% questions are distinct) compared to framed questions which had 49% uniquely structured questions. Answers are also short, with median and modes of 1 word per question (mean of 1.6). Comparing all lowercase answers for unique structures, we observed 486 distinct answers. This represents 32% distinct answers of the total 1,515, which we expect because about half of all answers are either ‘yes’ or ‘no’.

Subgroup analysis of free-form questions demonstrates the relationship between question types and answer types. Of the 11 question types, Presence questions dominate, with 36% (483) of free-form questions. The least frequent question type is Counting, with only 1% (15) questions. The QA pairs are split into 42% (637) open-ended answer types and 58% (878) close-ended. Yes/no questions represent 92% of the close-ended QA pairs, and ‘yes’ (405) is as frequent as ‘no’ (406.)

Subgrouping question and answer types, we observe that people think about certain categories differently (See Fig. 2 ). Abnormalities, Presence, and Size questions are mostly closed-ended questions. The majority of Organ system and Positional questions are open-ended. While these imbalances are important for learning how to better balance future datasets to ensure that algorithms have enough data to learn to predict answers, these observations suggest that different approaches may be required for different question types. The favoring distribution of closed-ended size questions may suggest that in clinical practice, asking questions about whether something is “enlarged” or “atrophied” in reference to an acceptable relative size is more useful than asking an open-ended “what is the size of the organ?”.

Breakdown of different types of Closed vs Open-ended free form questions shows that certain question types are more likely to be open-ended: positional, counting questions and other.

Relationship of Closed and Open Questions

We next analyzed the types of words for each question type (The full analysis is provided with the dataset distribution). For each question category, we tokenized the words and listed the top 10 most common distinct words. Many of these words are unique to the question category and can be used as keywords to help guide question categorization. The frequencies of words give insight into the context in which certain questions are likely to be asked. For example, ‘heart’, ‘cardiac’, ‘cardiomegaly’, and ‘dilated’ are frequent words in the size questions, which suggests that participants thought about size of the heart more commonly than other organs.

Relationship of Image Type and Organ System

Examining the question categories in relation to image type, we observed several differences in what clinicians ask when they see images of the head, chest, or abdomen. Presence questions are asked evenly about abdomen and chest, with fewer head images, although remaining the most commonly asked question type. Positional and Color questions were more prevalent with images of the head. Contrastingly, positional questions were less frequent with abdominal images. Organs in the head, in particular structures of the brain, are much more fixed in location compared to organs in the abdomen. Therefore, positional questions are more relevant in the head where one can reference locations. In the abdomen, asking if something is seen in the image is more clinically relevant or easier to determine than knowing the location. Another trend is that size questions are asked most frequently in chest images, agreeing with the distinct words we described earlier. The relationship between question type and image type gives some insight into the patterns of clinical relevancy that clinicians may have. Not all questions are useful and maybe shouldn’t be asked for every image.

VQA-RAD Benchmarking

We validated the dataset by evaluating the performance of well-known VQA algorithms and their accuracy at answering visual questions from the VQA-RAD dataset.

We include two well-known VQA methods: Multimodal Compact Bilinear pooling (MCB) 16 and Stacked Attention Network (SAN) 17 . The MCB model, winner of the 2016 VQA challenge, uses a multimodal compact bilinear pooling method that first predicts attention visually and textually then combines these features with question representation. MCB includes three components: a CNN image model, an LSTM question model, and MCB pooling that first predicts the spatial attention and then combines the attention representation with the textual representation to predict the answers. For the image model, we used ResNet-152 pre-trained on imageNet. For the question model, a 2-layer LSTM model was used.

The SAN model is a stacked attention model that queries images multiple times to progressively narrow an attention. Similarly to MCB, SAN includes three components: the image model based on a CNN to extract high level image representations, the question model using an LSTM to extract a semantic vector of the question and the stacked attention model which locates the image regions that are relevant to answer the question. We used the last pooling layer of VGG-16 pre-trained on imageNet as image features, and the last LSTM layer as question features. The image features and the question vector were used to generate the attention distribution over the regions of the image.

The MCB and SAN were pre-trained on ResNet 18 and VGGNet 19 respectively to extract image features and then both trained on VQA V1.0 dataset training and validation 4 . We refer to these baseline models as MCB-VQA1.0 and SAN-VQA1.0, when trained on VQA V1.0. We then trained the models on the ImageCLEF-VQA-Med 14 training set to see how an existing medical VQA dataset influences the models. We refer to the ImageCLEF trained models as MCB-CLEF and SAN-CLEF. We then train the two networks with VQA-RAD training set creating MCB-RAD and SAN-RAD. Since VQA-RAD is a small dataset, we combined ImageCLEF-VQA-Med with VQA-RAD and trained the networks to explore any synergistic effects. As part of a baseline reference for closed-ended questions, we include predicted answer sets of all ‘yes’ and ‘no’.

From VQA-RAD we separated out a training set and test set. VQA-RAD test set is composed of 300 randomly chosen free-form questions and 151 corresponding paraphrased questions. The VQA-RAD training set is the remainder of the dataset. We conducted a manual review to calculate simple accuracies for each question type and use several accepted metrics to calculate overall performance for each model. For the manual review, we judge the predicted answers of the models to the gold standard VQA-RAD test. In addition to exact match, a full point is given for predicted answers that appropriately answer the original question. For example, “right lung” and “right upper lobe” are correct answers for the question “where is the lesion?” since the question does not indicate a degree of specification. Half a point is awarded for answers that incorporate part of the answer. For example, “right lung” would be given half a point for “which lobe is the lesion in?” where the correct answer is “right upper lobe”.

We employ multiple overall scoring methods to demonstrate limitations of current metrics and need for newer metrics to evaluate VQA in medicine. Simple accuracy is calculated as the number of correct answers over total answers. Mean accuracy is the average of accuracy of the question types, similar to TDIUC dataset 12 . The BLEU 20 method, a commonly used metric for evaluating machine translation, measures the difference based on number of different words used. Following the ImageCLEF-Med methods, we pre-process the human and machine answers by converting answers to lower-case, removing punctuation, tokenizing individual words, and removing stop words using an English list of 172 words.

Algorithmic validation

As might be expected, the models that best predicted VQA-RAD test set are trained on VQA-RAD training alone or partially. For closed-ended questions, MCB_RAD had slightly better simple accuracy of 60.6%, while SAN_RAD had a slightly better mean accuracy at 54.6%. This performance is comparable to the MCB and SAN models trained and tested on VQA1.0 dataset which had reported 64.2% 16 and 58.9% 17 accuracies for open and closed ended public domain questions. Predicting open-ended VQA-RAD questions, the models performed much lower with the MCB_RAD scoring the best at 25.4% simple accuracy and 19.3% mean accuracy. The contrast between open and closed-ended questions suggests that these models are currently still guessing and may require more data. Potential improvements can come from learning medical terminology and jargon, greater quantity and diversity of questions, and extracting image features from radiological images as opposed to public domain.

Manual evaluation of the accuracy of automatic answers to naturally occurring questions in the VQA-RAD test set is shown in Tables 2 and 3 .

In medicine, seemingly small changes in wording can result in very different questions and answers. The opposite also holds true, the same idea can be written in simple or complicated phrasing. The VQA-RAD test set contains 151 matched pairs of free-form and paraphrased questions. We analyzed the MCB_RAD model to better understand the variance in predicted answers for similarly written questions. We observed 48 pairs (31.8%) had differing behavior, Table 4 shows several select pairs. Medical terminology, grammatical variations, and clinical jargon present challenges for algorithms. Paraphrasing helps understand what VQA algorithms are learning and whether those rules are aligned or need some degree of supervision. In addition, paraphrasing gives insight into the natural language of clinicians and what they consider to be similar or different concepts.

Overall scores may be useful for picking a winner of a challenge, but they may also bias towards algorithms that can provide answers that are more common. In medicine, the best answer is the most clinically relevant, not necessarily the most common. Calculating accuracies for different question types can highlight unique features of algorithm designs. For example, the SAN_RAD model scored high for open and closed ended size questions suggesting for this type of task, the SAN design might have certain advantages over the MCB design. However, further refinement is needed to understand these differences, as it is not clear if the SAN model is able to better recognize image regions of interest or is better at answering questions.

VQA-RAD demonstrates the value of natural questions for the specialized medical domain. Models without VQA-RAD training scored the same or lower than the all ‘yes’ or ‘no’ baselines, suggesting they were no better than guessing yes/no as answers. While the models may have been able to recognize yes/no structured questions, they lacked image features and vocabulary. For example, we observed that the MCB model trained on VQA1.0 dataset predicted many open-ended questions with “face”, “clock”, or “banana” as answers.

Analysis using BLEU scoring shows the need for improvement to metrics measuring visual tasks in medicine. We observe an opposite pattern between BLEU and manually judged accuracies where all ‘no’ resulted in better scores. BLEU penalizes answers with varying lengths, which favors the short one-word VQA-RAD answers and does not consider semantic similarity. While BLEU may be appropriate for certain machine learning techniques, it is not useful for medical VQA where there are many ways to phrase an answer and semantics are more valuable than exact wording. Using VQA-RAD, improved metrics can be developed that are aligned with the task of clinical care.

Usage Notes

To ensure that all researchers understand how to use the data, we provided instructions in the Readme file in the Open Science Framework ( Data Citation 1 ).

Additional Information

How to cite this article : Lau, J. J. et al . A dataset of clinically generated visual questions and answers about radiology images. Sci. Data . 5:180251 doi: 10.1038/sdata.2018.251 (2018).

Publisher’s note : Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Litjens, G. et al. A Survey on Deep Learning in Medical Image Analysis. Med. Image Anal. 42 , 60–88 (2017).

Article   Google Scholar  

Menze, B. H. et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Trans Med Imaging 34 , 1993–2024 (2015).

Shin, H.-C. et al. Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation. in Proceedings of the IEEE conference on computer vision and pattern recognition 2497–2506, https://doi.org/10.1109/CVPR.2016.274CVPR (2016).

Antol, S. et al. Vqa: Visual question answering. Proc. IEEE Int. Conf. Comput. Vis. 2425–2433, https://doi.org/10.1007/s11263-016-0966-6 (2015).

Goyal, Y., Khot, T., Summers-Stay, D., Batra, D. & Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. in Proceedings of the IEEE 30 th conference on computer vision and pattern recognition 6325–6334 (CVPR, 2017).

Gao, H. et al. Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering. Adv. Neural Inf. Process. Syst 2296–2304, https://doi.org/10.1145/2733373.2807418 (2015).

Zhu, Y., Groth, O., Bernstein, M. & Fei-Fei, L. Visual7W: Grounded Question Answering in Images. In Proceedings of the IEEE conference on computer vision and pattern recognition 4995–5004, https://doi.org/10.1109/CVPR.2016.540CVPR (2016).

Krishna, R. et al. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Int. J. Comput. Vis. 123 , 32–73 (2017).

Article   MathSciNet   Google Scholar  

Andreas, J., Rohrbach, M., Darrell, T. & Klein, D. Deep Compositional Question Answering with Neural Module Networks , Preprint at https://arxiv.org/abs/1511.02799 (2015).

Johnson, J. et al. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE 30 th conference on computer vision and pattern recognition 1988–1997 CVPR (2017).

Lin, T. Y. et al. Microsoft COCO: Common objects in context. Lect. Notes Comput. Sci . (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 8693 LNCS , 740–755 (2014).

Google Scholar  

Kafle, K. & Kanan, C. An Analysis of Visual Question Answering Algorithms , Preprint at https://arxiv.org/abs/1703.09684 (2017).

Gurari, D. et al. VizWiz Grand Challenge: Answering Visual Questions from Blind People , Preprint at https://arxiv.org/abs/1802.08218 (2018).

Hasan, S. A. et al. Overview of ImageCLEF 2018 Medical Domain Visual Question Answering Task , Preprint at http://ceur-ws.org/Vol-2125/paper_212.pdf (2018).

Kafle, K., Yousefhussien, M. & Kanan, C. Data augmentation for visual question answering. In Proceedings of the 10th International Conference on Natural Language Generation 198–202 (2017).

Fukui, A. et al. Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 457–468, https://doi.org/10.18653/v1/D16-1044 (ACL, 2016).

Yang, Z., He, X., Gao, J., Deng, L. & Smola, A. Stacked Attention Networks for Image Question Answering. in Proceedings of the IEEE conference on computer vision and pattern recognition 21–29, https://doi.org/10.1109/CVPR.2016.10 CVPR (2016).

He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition 770–778, https://doi.org/10.1109/CVPR.2016.90 (2016).

Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition , Preprint at https://arxiv.org/abs/1409.1556 (2014).

Papineni, K., Roukos, S., Ward, T. & Zhu, W. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of Association for Computational Linguistics 311–318, https://doi.org/10.3115/1073083.1073135 (ACL 2002).

Data Citations

Lau, J., Gayen, S., & Demner-Fushman, D. Open Science Framework https://doi.org/10.17605/OSF.IO/89KPS (2018)

Download references

Acknowledgements

We thank Dr James G. Smirniotopoulos, Chief Editor, MedPix®, Former Professor and Chair of Radiology (retired), Uniformed Services University of the Health Sciences for help with MedPix and radiology questions. We thank Dr Yassine Mrabet for insightful conversations and would also like to thank the following people for their assistance on this effort: G. Thomas Brown, Katherine Chen, Tiffany Chen, Nisarg Chhaya, Shazia Dharssi, Eric Ding, Joseph Featherall, Jonathan Gammel, Eileen Hu-Wang, Fabricio Kury, Lucy Li, Kathee Liang, Liesl Matzka, David Stein, Alison Treichel. This work was partially supported by the Intramural Research Program of the National Library of Medicine, National Institutes of Health. This research was made possible through the National Institutes of Health (NIH) Medical Research Scholars Program, a public-private partnership supported jointly by the NIH and generous contributions to the Foundation for the NIH from the Doris Duke Charitable Foundation, Genentech, the American Association for Dental Research, the Colgate-Palmolive Company, Elsevier, alumni of student research programs, and other individual supporters via contributions to the Foundation for the National Institutes of Health. For a complete list, please visit the Foundation website at: http://fnih.org/what-we-do/current-education-and-training-programs/mrsp

Author information

Authors and affiliations.

Lister Hill National Center for Biomedical Communications, National Library of Medicine, Bethesda, MD, USA

Jason J. Lau, Soumya Gayen, Asma Ben Abacha & Dina Demner-Fushman

You can also search for this author in PubMed   Google Scholar

Contributions

J.L. conceptualized the study, selected images, designed annotator interface, manually reviewed data, analyzed data, and contributed to writing and editing of the manuscript. S.G. built interface and collection for annotator interface, performed data analysis with models and contributed to editing of the manuscript. A.B. analyzed and processed the datasets, performed deep learning experiments, participated to the evaluation process and contributed to the editing of the manuscript. D.D. oversaw study design and implementation and contributed to writing and editing of manuscript.

Corresponding author

Correspondence to Dina Demner-Fushman .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

ISA-Tab metadata

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ The Creative Commons Public Domain Dedication waiver http://creativecommons.org/publicdomain/zero/1.0/ applies to the metadata files made available in this article.

Reprints and permissions

About this article

Cite this article.

Lau, J., Gayen, S., Ben Abacha, A. et al. A dataset of clinically generated visual questions and answers about radiology images. Sci Data 5 , 180251 (2018). https://doi.org/10.1038/sdata.2018.251

Download citation

Received : 18 June 2018

Accepted : 16 September 2018

Published : 20 November 2018

DOI : https://doi.org/10.1038/sdata.2018.251

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Nature Medicine (2024)

ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset

• Johannes Rückert
• Louise Bloch
• Christoph M. Friedrich

Scientific Data (2024)

Visual Question Answer System for Skeletal Image Using Radiology Images in the Healthcare Domain Based on Visual and Textual Feature Extraction Techniques

• Jinesh Melvin Y.I.
• Mukesh Shrimali
• Sushopti Gawade

Annals of Data Science (2024)

Pre-trained multilevel fuse network based on vision-conditioned reasoning and bilinear attentions for medical image visual question answering

The Journal of Supercomputing (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

College & Research Libraries ( C&RL ) is the official, bi-monthly, online-only scholarly research journal of the Association of College & Research Libraries, a division of the American Library Association.

C&RL is now on Instragram! Follow us today.

C&RL News

ALA JobLIST

Advertising Information

• Research is an Activity and a Subject of Study: A Proposed Metaconcept and Its Practical Application (79325 views)
• Information Code-Switching: A Study of Language Preferences in Academic Libraries (40775 views)
• Three Perspectives on Information Literacy in Academia: Talking to Librarians, Faculty, and Students (29559 views)

Along for the Journey: Graduate Student Perceptions of Research

Alissa Droog, Kari D. Weaver, and Frances Brady *

Graduate students’ identities and personal lives are heavily tied to their experiences of research, and many struggle to find, understand, and use information for research purposes. Using a drawing exercise rooted in visual research methods combined with semi-structured interviews, a research team in the United States and Canada explored graduate student perceptions of research with nineteen participants. Thematic analysis identified six themes: research is abstract; research is an odyssey; social support makes or breaks the student experience; research is an emotional continuum; interplay between identity/values; information is problematic. The study has implications for how librarians support graduate student research.

Introduction

Graduate students are increasingly becoming a demographic of interest and focus for academic library services and programming. This burgeoning interest in the field originates from the desire to better support the scholars, and subsequent library champions, of tomorrow as well as the recognition that graduate students are interacting with in-depth, often interdisciplinary, research that needs holistic support. Despite these trends, graduate students’ information behaviors and research experiences have received significantly less notice in the literature than undergraduates or working professionals. This study fills that gap and helps those academic library individuals who work closely with graduate students to better understand their perceptions of, and experiences with, research to address their needs more fully.

Literature Review

Student information behaviors.

Studies in the Library and Information Sciences (LIS) field have focused on the student experience and self-conception of information seeking for the past thirty-five years. 1 Existing studies in the field have emphasized the literature review as paramount in considerations of graduate student research. 2 However, this study looks at the research process holistically during the entirety of graduate level study, connecting this work more strongly to the existing literature on student information behaviors and affect in research. Several theories that underpin information-seeking behavior have primarily focused on particular populations: Kuhlthau’s theory of information-seeking behaviors derived from work with primarily high school students; 3 sense-making theory has studied undergraduate students; 4 and cultural-historical activity theory has looked extensively at information use and conceptualization in the workplace. 5 Existing studies on graduate research behaviors have primarily examined how they used search strategies or perceived library resources, not the overall research process. 6 Furthermore, the majority of studies that do consider graduate students as a particular population have focused on those studying the humanities, 7 or education 8 rather than looking at students from diverse areas of study.

Largely missing from the existing literature are more holistic examinations of how graduate students—who are themselves being trained to participate in and conduct significant research—participate in and experience research as a phenomenon. This gap is partly due to the nature of training that graduate students receive while in their programs. There is a conception by faculty that graduate coursework adequately prepares students to engage in research. However, graduate students themselves report an overall lack of attention to their preparation as researchers. 9 Within graduate programs, the purpose of research is determined by the level of study, with an emphasis on direct application of concepts at the master’s level and the ability to advance the knowledge of a field at the doctoral level. 10 These factors in graduate student research training are further complicated by the increasing emphasis on interdisciplinarity in research, the requirement of interdisciplinary and interinstitutional research teams, and the need to develop graduate student identity across all areas of professional and research practice during the course of study.

How, then, do graduate students learn to do research and arrive at original research ideas? Jurisevic found that students needed to pull from their own lived and professional experiences to iterate within their research. 11 Golde suggests it is an emphasis on apprenticeship with experienced faculty researchers that develop research capacity. 12 Boyce et al. found that a clear understanding of the processes for funding, publication, and authorship developed these abilities. 13 Wessels et al. created a model to depict affective-motivational dispositions needed to attain research competence (RC) in moving through challenging situations in the research process. 14 Impacting these considerations of how students seek information is a fulsome understanding of the affective domain as it applies to research tasks.

Affective Dimension of Research

Kuhlthau asserts in her seminal information search process model that feelings correlate with thoughts and actions throughout the entire research process, from uncertainty at the beginning to relief after search closure. 15 Students often initially hold a negative bias against research, 16 which impedes successful research. When patrons lack motivation, they stop their search. 17 Conversely, positive attitudes—such as motivation to learn, personal investment in learning, enjoyment of learning, and sense of accomplishment—impact cognitive and sensorimotor success in research. 18 For example, phenomenological interviews demonstrate that passion for a topic motivates students to continue their search. 19 LIS studies divide the learning process into three domains: affective (feelings), cognitive (thoughts), and psychomotor (actions). 20 Some studies assert that a continuum characterizes the affective domain from primary neurological responses to foundational genetic characteristics. 21 Schroeder and Cahoy define the affective domain by its components: “attitudes, emotions, interests, motivation, self-efficacy, and values.” 22

In 2015, the Association of College and Research Libraries (ACRL) developed the Framework for Information Literacy for Higher Education (Framework) to aid librarians and faculty in teaching information literacy. In addition to knowledge practices, the Framework addresses dispositions, “which describe ways in which to address the affective, attitudinal, or valuing dimension of learning.” 23 Several studies analyzed which frames connected to college students’ experiences of research. 24

Most information literacy research regarding the impact of the affective domain on the research process studied undergraduate populations, 25 with a few outliers reviewing other populations: high school students, 26 faculty, 27 and information science graduate students. 28 Most studies involved interviews. 29 Two studies watched participants in a search process. 30 While Nahl and Tenopir did involve non-library science graduate students, 31 the study involved users who were new to using electronic databases due to the study’s date in 1996. Previous studies have not used visualization or drawing as part of their interview process to learn about students’ affect regarding the research process.

Visual Research Methods

The use of visual research methods may provide new insights to graduate perceptions of research. Used throughout the social sciences in qualitative research since the 1960s, the use of visual research methods in LIS has been increasing for the last ten years. As the use of these methods grows, researchers rely on methods developed outside of LIS 32 as well as examples of visual research from inside the field. 33 The growth of visual methods can be attributed to several advantages, including new insights for existing research questions, more complete, comprehensive data, flexibility to work with diverse populations, and more participatory relationships between researcher and subject that can empower participants. 34 In particular, visual methods, such as the draw-and-write technique or graphic elicitation, have been found to help researchers better understand complex and abstract phenomena, such as conceptions of information, 35 student conceptions of group work, 36 and conceptions of research by librarian and archivists, 37 as well as by graduate students and staff from university business schools. 38 In these studies, participants draw complex ideas and describe their drawings in writings, interviews, or focus groups with researchers.

Although Doucette & Hoffman have used graphic elicitation to understand librarian and archivist conceptions of research, 39 the method has had limited use in understanding how students, particularly graduate students, conceive of this complex topic. Bryan and Mavins discuss the value of drawing research in a classroom setting through their work with doctoral students and staff in the business school department who drew research or researchers in a workshop setting. 40 However, they focus less on how students perceive research. Visual research methods can provide deeper insights into research questions. Therefore, using this method to understand graduate student perceptions of research could give a much-needed understanding of how students understand research and how to best help graduate students become proficient researchers.

To date, there have been limited studies examining the graduate student conception of research itself, with prior research focusing on the performative aspects of conducting research rather than the mental models of understanding research. While studies have addressed the impact of the affective domain on the research process, they have not focused on graduate students, nor used visual methods. This paper seeks to address this gap by applying graphic elicitation methods to a semi-structured interview protocol to examine graduate student self-conceptions of research.

The research questions for this study were:

• How do graduate students at U.S. and Canadian institutions conceptualize research, including how they see themselves as researchers, how they perceive research is conducted, and what they consider the point of research to be?
• What emotions or elements of the affective domain do graduate students associate with research?

Justification of Methods

In this study we used a combination of two methods of data collection: semi-structured interviews and graphic elicitation. We thematically analyzed the data to identify the major findings, using a grounded theory approach. The purpose of a grounded theory study is for a researcher to, “derive a general, abstract theory of a process, action, or interaction grounded in the view of the participants of the study.” 41 We selected these methods as our research questions focused on understanding graduate student perceptions of, and experiences—both lived and affective—with research. We chose a semi-structured interview approach to provide points of comparison between research participants while also allowing for follow-up questions that could be tailored to individual participants to better explore the phenomena under study. 42

While data collection for qualitative research is often rooted in analyzing written text, there has been a recent movement toward defining text as inhabiting a variety of visual information formats beyond written work, including drawings, videos, and artwork. 43 Graphic elicitation is a visual research method where the participant draws something and then discusses their drawing, which allows the researchers to, “see through the eyes of the participant” with a new lens. 44 Visual research methods: are beneficial for new insights into existing research questions; 45 provide more complete, comprehensive data; build more participatory relationships between researcher and subject, which can be empowering for participants; 46 and provide flexibility in working with diverse populations, representing abstract concepts like information, group work, and research. 47 Given these benefits, and following the best practice of combining qualitative methods for data validity purposes, 48 we chose to use a visual research method with semi-structured interviews to provide greater insight and participant explanation of the drawing and thoughts about research.

Recruitment, Ethics, and Informed Consent

Participants for the study were graduate students aged 18 or older enrolled in a program of study at a U.S. or Canadian institution who could communicate with the researchers in English; they were voluntarily recruited from the institutions at which the researchers worked, via email. We sent emails directly to students at the University of Waterloo from a list available to the member of the research team employed there, to enrolled students at Adler University, and to a randomly selected group of enrolled graduate students at Northern Illinois University. We scheduled participants who met the study inclusion criteria for a Zoom interview with two research team members (an interviewer who had never interacted with the participant before and a second researcher taking notes). We provided informed consent in advance of each interview and again at the beginning the interview.

During the interview, we asked participants to, “draw what research is to you” and gave them ten minutes to draw, after which we asked them to describe their drawing and any emotions associated with it. Participants provided their own drawing utensils and completed their drawings in a number of ways. Some drew digitally, others drew with a pencil and paper, and others added color. After the interviews, we asked participants to share their drawings and had the option to sign a second informed consent for using their copyrighted drawings in subsequent presentations and publications. Research participants were incentivized to participate in the study through inclusion in a drawing for one pair of Apple AirPods, which we provided at our own expense.

The study received ethics approval through the Institutional Review Board (IRB) at Northern Illinois University in the United States (Protocol # HS21-0349), through the Office of Research Ethics at the University of Waterloo in Canada (ORE # 43220), and a reciprocal agreement with Northern Illinois University by the IRB at Adler University. We recorded interviews and used these recordings to review and edit auto-transcriptions generated by Zoom for accuracy. Data were protected through a shared, encrypted folder on a Canadian-based server in compliance with the data privacy protections required by research ethics approval.

Participant Pool

The participant pool for this study consisted of nineteen self-selected graduate students who responded to recruitment emails. Nineteen participants is an appropriate number for this type of study as, “Qualitative researchers usually study a single setting or a small number of individuals or sites, using theoretical or purposeful rather than probability sampling.” 49 There were eleven master’s and eight doctoral participants from all campuses associated with the researchers’ institutions, including online programs; therefore, students resided across the United States and Canada. The students were enrolled in programs in various fields of study, including counseling, education, psychology, engineering, public health, biology, policy administration, anthropology, and kinesiology. While the researchers did not collect demographic data about our participants outside their study area, participants were openly diverse in their race, gender, age, life experience, and citizenship. The participants were likely diverse in other ways, but we did not gather these demographic data as it was not a direct focus of the study. This size of the participant pool and diversity of experiences at the graduate level of study is representative of the later graduate student population and allows the data gathered to provide in-depth elucidation of the participant experiences.

Data Analysis

After the research team edited transcripts and verified them for accuracy, we used a thematic coding approach. This method of analysis was selected as thematic coding is appropriate for “those exploring a participant’s psychological world of beliefs, constructs, identity development, and emotional experiences.” 50 The use of nVivo (version 12) software allowed researchers to use the same codes reliably across coding sessions. The researchers coded the first transcript in a collaborative, synchronous meeting to establish the initial code book and subsequently coded all other transcripts in pairs. Any issues of disagreement or uncertainty were then referred to the third member of the research team for a final determination and to ensure inter-coder reliability. Where applicable, we used the drawings to complement, illustrate and thicken quotations from transcripts. 51 Once the research team coded all transcripts, we met to group the codes into larger themes. We determined the final themes through iterative reflection, which is represented in the findings.

Researcher Positionality

Given the impact researcher identities can have on research, we need to share the context of our positionalities. 52 The members of the research team acknowledge that we present as White. Two of the researchers are cis-gendered women, and one identifies as non-binary. We are all employed professionally as librarians at the institutions from which our research participants originated. We attempted to mitigate the influence of social power structures by having a member of the research team from a different institution as the participant conduct the interview. Our identities and the potential for perceived power may have influenced what our research participants shared with us as well as what was of interest to us in probing further during follow-up questions in the interviews. Further, we acknowledge that our own experiences as graduate students in LIS and other fields influenced our understanding and presentation of the findings and helped us make meaning from our data.

Six themes emerged from a thematic analysis of the semi-structured interviews: 1. research is abstract; 2. research is an odyssey; 3. social support makes or breaks the student experience; 4. research is an emotional continuum; 5. interplay between identity and values; 6.(information is problematic.

Research is Abstract

The experience of research as an abstract and hard-to-grasp phenomenon was common throughout our interviews. Students often struggled to describe what research means to them, and turned to metaphors and similes for their descriptions. Among many other metaphors, participants described research as:

• A sphere of light that sends out waves to light up other bulbs
• A rolling ball
• A series of cabinets to open
• Circles within circles
• Digging in the desert with paintbrushes
• A game of baby steps
• A path that is not always smooth
• Planting a tree that you will never sit in its shade
• Panning for gold
• A big mess I have to clean up
• Jumping off a diving board into information
• A bobsled or roller coaster
• A treasure hunt

These metaphoric, or abstract descriptions also appeared in many of the drawings that participants completed. For example, one participant likened research to a funnel (figure 1), writing:

So, here we have the internet vortex of data. And then you do your first search, and you have general data. And then after looking that over and solidifying what you want, and then it goes through more filters and then you wind up with just the distilled data, but, when you get here, that doesn’t exactly mean that you are done because from here, it can also mean that you have to go back because you found more information in the data and you have to re-refine or gather more information from it. It’s cyclical, but it can go off into rabbit holes. But, it doesn’t, that’s not always a bad thing. 53

At least two participants described research in relation to sight. For one, they drew a pair of muddy goggles and described how: “I truly feel like research is starts out as a big mess that I have to clean up” (figure 2). For another, they drew a pair of eyes and described research as a way of seeing the world (figure 3), writing:

So, what research means, or is, to me is seeing. And so, eyes, and then these are people that are being seen. And then through the eyes of the researcher it’s being translated into all these many more, many more either people or ideas or data. But it’s sort of transmitting through the eyes. I think that’s the most important feature. Translating, transmitting.

Another participant described research as a wall that they had to get past (figure 4), writing:

there’s a person at the bottom, and they’re not entirely sure. They had this idea of where they want to go, but they’re not entirely sure how to get there, so he’s sitting there at this like wall kind of stuck…Umm, and so they have this idea of where they’re supposed to go, but they can’t even quite see all of what the final product could be anyway. Umm, because, despite everything you’re still chasing almost like a moving target…Um, a lot of that comes from like, my experience of just being like, I’m stuck and I don’t quite know where I’m going. But that’s, that’s like really what research is to me.

The use of metaphors to describe research suggests the largeness of the concept. For one student, the inability to truly know what research was likened to pregnancy: “It’s like being pregnant. There’s no point of reference. And everything is new. Right. But after you’ve had a child, better prepared about what to expect.” Due to the abundance of metaphors in the drawings and interviews, the researchers concluded that research is an abstract concept that is difficult to describe.

Research is an Odyssey

Within this study, participants were far from homogenous, but their experiences were simultaneously in tension between the universal and the unique. One participant shared their conception of research as follows:

Okay, so basically my idea of our research is that we face a mess, or any issues or whatever questions we are interest to do a little bit further investigations. So of course, there’s always questions but then we have a kind of like later moment to think about maybe there are some possible solutions.

Another participant was heavily invested in the same construction of research being messy and unknown, observing:

Because it’s research like: man we don’t know what we’re going to find, we don’t know what we’re gonna solve, you know, discover, but we’re going to get in the water. [tone becoming more excited here] And see, and that’s the research basis part, and where you make these discoveries where you figure out what’s real, right? You take those thoughts that you had in your mind, and you bring them down into a space where you’re going to, sort of, like, find things that are new so that was an interesting question.

This same participant carried the metaphor of jumping into the water further in their drawing of the research experience, sharing that research is a bridge between two big ideas (figure 5). The surrounding dark waters are characterized by the reality of doing the work and the potential for discovery as an eventual outcome.

Participants’ descriptions were unique, yet these overall conceptions of research were shared universally. Participants held this same tension between their unique experiences and the universal when considering how they performed research or engaged with the research process. One student noted: “So, in the beginning it was really just, you know, trial and error, I’m looking for these things out there.” A second participant expanded on this concept of trial and error describing their experience with research as, “Once you go about doing it and you start trying to do it and keep realizing, things aren’t working and you have to like continually go back to the drawing board and figure out something.”

These conceptualizations strongly mirrored classic stories, like the Odyssey, in which an individual goes on a long and harrowing journey and the expectations of the journey are not always met. Further, the experience of graduate-level research is both liminal, that is, constantly existing between two states or places of being, as well as a recursive process. One participant expressed this experience eloquently, stating:

So I think it increases the comfort that like it’s all going to work out. if you, if you think of it as this kind of like: these are the steps that I have to take. But I also feel like it doesn’t help anybody to ignore the realities of, you’re gonna have to go back, you know, it’s not going to work out in that linear process. I feel like the linear process is a place to start when talking about research.

Another participant described their approach to working within liminal spaces noting, “But the biggest thing that I’ve learned is to just set in it, set in the confusion for a minute and just, let it marinate and sit. And the more you’re able to look a things objectively, or, well, yeah, objectively, the clearer the path will become for me.” A third participant continued to grapple with the in-between experience of research stating, “You know, that’s a struggle with, with research in general is they don’t tell you what’s enough. You know like no one, no one tells you like, this is good enough, or like, this is where it should be.” This concept of the journey also appeared in participant drawings, including one where the participant had to take a pause for family reasons, exemplified by the tent, part way through their academic program (figure 6). The onslaught of research always threatens to create an avalanche that can impede or throw off the journey, but reaching the summit is a rewarding and compelling outcome.

Ultimately, many participants expressed value in the journey and the learning that resulted. Expressing this value, one student shared, “And, but by the time the end of it you have this incredible bond that you build you have done this thing you made these discoveries and you have a remarkable story to tell.”

Social Support Makes or Breaks the Student Experience

A common theme in our interviews and our participant’s drawings of research was the importance of social support to research. Participants discussed the importance of social support, often mentioning the value of supervisors, fellow graduate students, and others as instrumental in their progression and success.

Participants described how the support or lack of support from a supervisor could enhance a student’s experience or make it impossible for them to succeed. One participant shared:

Whenever I’m doing research and I have to send it back to the supervisor real quickly and be like: Hey, what do I add here? Is this full? Do you think this is, am I applying theory the right way? And then she would comment back and gives me great feedback, it might take time but it still gets back to me, which is great. Versus other students where their supervisor sometimes like, I don’t know, you work it out, you’re the student. It’s like, woah. But I’m learning from you, technically. Or even simple things like maybe she reminds us, “Apply for the scholarship. Do this. I’ll give you a reference dah, dah, dah.” I’ve heard of students… Um, that one supervisor told the other student, her own student, that she’s not going to give a reference to it because they’ve been working online and she doesn’t know the student well. But it’s like, one of the requirements of that scholarship is the supervisor gives you reference. Second, it’s the time of COVID and we are working online. So it’s just like that. How is that even acceptable? Um, so they could be examples of how can one succeed in this whole role versus how can one not succeed.

This quote exemplifies how the supervisor provided feedback and guided the research process, which supported the student in applying for scholarships. At the same time, the participant recognized the value of this relationship as they compared it to peers who were held back by supervisors who refused to support scholarships.

It was not just supervisors that participants mentioned. Many participants also discussed the importance of their fellow graduate students as supports throughout the research process. One participant shared:

And the research that I do wouldn’t be the research that I’m doing without my collaborators. I, you know, I spent a lot of time alone doing statistical analysis but even that isn’t completely solitary because I’m constantly asking for help and feedback and new ideas and, you know, even the research part—that’s like the part where you’re reading before you start a project, I don’t view is entirely solitary because you’re building on the ideas that other people had. And if you’re lucky some of those people are still alive and willing to talk to you. [laughs]

One participant included their colleagues in their drawing of research, describing this aspect of their drawing as “a couple of students sitting on the other side of things called friends who are going to edit this work” (figure 7).

Some participants also shared positive interactions with librarians, primarily in reference situations, as supportive to their process. However, it was rarely a focus of their discussions unless they were specifically referring to the literature review. One participant stated:

Even when I…read studies and things like that for papers I feel a little like I don’t know if I’m doing this right…. I had a meeting with [a librarian] and it just really, really helped kind of organize this is how you do it…. And then I did find all these articles and then even that again [the librarian] was like okay now don’t read all of them or don’t read two of them and forget the rest because you don’t have time, sort of that process of get this layer first get the abstracts, and understand what’s happening there…. It was helpful. It’s almost like the structure became visible to me and it never was visible, before.

One participant bemoaned the diminished opportunities to talk to other graduate students due to the pandemic, stating:

So, I haven’t had the chance to talk to participants, or you know collect data, and then also because of Covid, I haven’t had a chance to socialize with other graduate students, which I think, by the way, is the most, one of the most valuable things as a research graduate students, right, being around other graduate students who are also struggling to run their studies. You know, it just helps to know that you’re not alone.

Lastly, participants described the importance of other people in their life outside of academia supporting their research. For example, one participant described the value of family support during their research as follows:

And also it’s, I find that it’s important for my son to see me invested in my work. And, and when he’s older I guess explain to him like this is what I had to do during that time… like I have support, like I’m lucky I have support I wouldn’t say it’s like for everybody.

Overall, graduate students described many examples of how their social support networks either helped or hindered their progress in research.

Research is an Emotional Continuum

The first question the researchers asked participants was: “what kinds of emotions are associated with research in your drawing?” One student responded, “it’s just a huge [sigh] collection of emotions.” Many participants in this study repeated the sense that research involves different emotions.

Students described negative emotions at the beginning of the research process, which they often attributed to being overwhelmed by the number of sources to read and evaluate for their literature reviews. One student asserted, “it’s scary because it’s unknown.” Others felt overwhelmed by learning to balance uncertainty with a desire to read everything; as one student explained, “it’s not always easy to feel like you don’t know all of the literature and I have to remind myself that’s not possible.” Here, the student knew they were not expected to, nor could they, read all the literature, but they wrestled with recognizing saturation. Another student struggled with not having enough time:

You know, because you don’t have…a lot of time, you know, to kind of narrow your focus and, you know, I mean journal articles and research studies are not short, and they’re not necessarily easy to read, especially when you’re new, you know, to this type of research and writing.

Another student said that, beyond a lack of time, the graduate experience of research contains more uncertainty than undergraduate research, which increases anxiety. They explained:

But when you go to graduate school, you realize that there’s so many unknowns, and there’s no definite answer…and they [the committee] would ask you questions about research that…it’s not well established or not well known or it’s a knowledge gap and then you’re trying to address this, and you’re not quite sure, because you have not read all of the literature [sweeping gesture with hands out and around] that exists, right? So I would never feel like I’m as confident as I was an undergrad where it’s like you have those readings, you read them, and then they test you on those readings.

This quote indicates essential differences in the research experience between undergraduates and graduate students. Graduate students are expected to find current gaps and create new knowledge, rather than summarizing texts from others.

Even after completing the literature review, participants continued to lack confidence, leading to emotions of fear and frustration. One student, referring to their research experiment stated, “and so there’s I think that fear part of me is like, I’m not going to do this right.” Another participant explained:

I think that no one’s comfortable with the idea that their experiments aren’t going to work, or that they’re not going to find something that is worth disseminating. I think that’s like the scariest thing to a graduate researcher, thinking like, ‘Oh my gosh, what if I don’t find something.’

Several participants echoed this fear that even if the research is designed well, they still might not find relevant results.

These negative emotions were often intertwined with positive emotions for the participants, creating tension between perseverance and procrastination. One student articulated:

Oh, I thought this method will work or this model will work, but in reality, when we start really doing something then I need to change. So I guess it’s kind of like half of it is exciting and the other half is uncertainties.

Some participants found the beginning of the process more frustrating, whereas they later felt more excitement, particularly after a success. One participant said: “finishing things like this gives me, you know, that sense of accomplishment, you know, it’s one step, one thing. So it’s both good and bad feeling I suppose.” Here the student highlighted their sense of accomplishment, but also the frustration that this positive emotion was just for one small task within the larger scope of their research.

Several students said research connected them to something beyond themselves. One participant shared that their passion for research allowed them to become immersed in the flow, saying:

And I know with me myself and I, the three of us can become extremely—engaged doesn’t even cover…—what is the word I’m looking for, so ‘immensed’, so part of the process that hours can literally fly by. It’s almost as bad as Facebook. So I am so drawn in and captivated by what I’m doing that, everything else is out, it doesn’t even dawn on me [hands move in the air around their head, like ideas floating past the head]. I have to set an alarm to make dinner, because I get all wrapped up in what I’m doing. And when a portion is completed, whether that’s chapter, whether that’s survey item, whether that’s looking for a coefficient value, whatever it is, I just feel so delighted with me. [laughs] and I feel achieved, so there’s you know, a sense of ‘all right Me! You go Me!’

Other participants appreciated how research connected them to others. One drew lightbulbs of different shapes and sizes, depicting which resources impact others (figure 8). They explained that research gives them a “feeling of connectedness and understanding…I do feel like I’m actually connecting in some sense with the authors of the papers that I read.”

However, even amid their excitement, most participants still returned to feelings of overwhelm. One participant shared:

I often get overwhelmed by the idea of all these things that I have to do… on the one hand, it’s very, very overwhelming to think about all the, all the details, because there’s so many of them, you know, there’s research just requires so much little pieces of effort, and all those little pieces of effort, you know, have to be, you know, then eventually coalesced into your, your research project. But year, then on the other hand, I find research very fulfilling, and so, I think that’s the other side of it is, is the sort of accomplishment feeling.

In the quote above, the participant expressed the tension between emotions amid the overwhelm. Another student similarly explained the overwhelming and conflicting emotions of research, stating:

there’s a lot of this idea of like, continuously moving forward even if moving forward seems like a step backwards in some cases. So like yes, I did figure, I did figure out how to like track all these particles. But oh no, tracking all those particles in the way I was doing it wasn’t particularly useful. How do I go back and actually put this into a usable format? And it’s like, okay well now I’ve got all this. It’s like, wait a minute some numbers aren’t making sense. Why aren’t they making sense? Oh, that’s because actually didn’t want these things wrong and I need to go back and start looking at it again. And now that I have the output in the form that I want it, now I can watch them change as I’m making these very small changes to how I’m looking for certain angles and certain characteristics.

Feeling various emotions was a common theme throughout the interviews with graduate student participants. They expressed a mix of positive and negative emotions, as well as a sense of discord between their various emotions.

Interplay between Identity/Values

Participants pursued research for a variety of reasons, many firmly connected to their identity, such as inherent personality traits, an interest in knowledge itself, a desire to improve the world, and personal career goals. Many participants noted that it was their personality that drew them to research, or that enhanced their enjoyment. Several focused on their innate curiosity, such as one participant who stated, “I have this avid curiosity. So, I would describe myself, like my thoughts around research, as hopelessly in love with it.” Similarly, another participant believed their inquisitiveness has always made them a researcher, stating, “if I were to think about it now, I think I became a person who was always going to be a researcher, very young. I think that folks who have that inherent curiosity, they kind of have the spirit of the researcher… I think what… unites researchers is a drive to find things out.” Some specifically mentioned enjoying challenges and bringing this into their research, such as the participant who shared: “as I progressed, I was like, I felt I was slowly getting to my end goal, but it was definitely not a smooth journey and I didn’t expect it to be smooth because I did do my master’s, so I knew what research entails, and if anything, I enjoy challenges. And if it were to be smooth, I don’t think I would… be drawn to research.”

However, others felt that their personality or life experiences decreased their aptitude for research. One participant suffered the death of a loved one during their dissertation, which they described as impacting both their mental health and research, sharing, “my own mental illness and my own loss that made my ability greatly diminished, you know, I don’t think I’m the researcher I was two years ago. Instead of getting better with experience, I think… my own problems have made me kind of worse at it. I guess because of these, you know, fears and difficulties with it, right, so you’re like, you’ve got to put yourself out there and things like that, and that’s incredibly difficult.” Another student recounted their frustration with the lack of rewards or recognition for their work which they expected, stating, “I would work very long hours, and there’s nobody who’s like, ‘oh wow you’ve been doing so many hours, like that’s amazing.’ Like, your work ethic is kind of like expected, as opposed to if I was working in the industry and I was investing this much time and effort, I would be rewarded at a much greater rate.” This student decided they did not want to continue in academia but would instead go into industry after graduation.

Figure 9 depicts multiple values, including investing in a career. This participant shared, “I tried to shape my research more like that would be helpful for me, for future, like when I graduated. I can use my research, and the skills again from my research to find the job I want.” Another participant focused on obtaining a degree when asked why they research. These profoundly personal rationales impacted how students reacted to pitfalls or detours in the research process.

Several students discussed the importance of the impact of their research, which they linked to their values. They valued this knowledge for themselves or for the sake of the world. One participant said, “I think I’m coming to the conclusion that it ultimately, me doing this is as much to enlighten myself as it is to fix the world around me.” This participant had drawn partially muddy goggles (figure 2), which they explained showed the chipping away at ignorance. Another student stated:

and when I say learning, that’s also including the research, because when I’m doing the research, I’m also learning about. And it isn’t, you know, I may be making it sound glib. But it isn’t. It’s also the reason why I get jazzed about what I’m studying because information is good. Knowledge is good. Science is good.

This quote exemplifies the connection between the value of information and the interest in research. One student who had given birth during their dissertation process reflected on how their struggles with research would impact their son, sharing, “I find it’s important for my son to see me invested in my work.”

Some participants were motivated by broader aspirations of their research shaping the scientific community or the world, such as the participant who wrote: “one of the things that… was very powerful and meaningful to me was the idea of finally getting to be a part of like this, scientific community intent on bettering, like humanity as a whole, and not just being someone doing something on their own. It’s the idea that everything we do is for people.” This quote exemplifies what others also expressed: the desire to be part of something larger. Several students noted that their research might play only a minimal part: “if you can see circles within circles, then you’re… at least cognizant… that you’re… a speck, you’re a spec of reality.” Another also spoke about, and drew, research in terms of circles (figure 10), saying, “but if you take into account the millions upon millions of people doing research right now. Each tiny blip that they had increases the circumference of this circle, which is all the combined knowledge that we have up to this moment.” So, despite the small potential impact of any particular study, students still believed that research would “positively impact our society. And, you know, help us move forward.” For many graduate students, the connection between their identities and their research was closely tied.

Information is Problematic

Throughout the interviews, participants expressed an awareness of the idea that information itself is problematic. Participants expressed that finding the information you want or need is not a clear-cut experience and requires an openness to exploration.

Some participants expressed this difficulty with information seeking and use as a major aspect of their drawing. In both figures 11 and 12, the participants identify information gathering as a critical component of generating ideas, as well as exploring ways of examining and understanding the experiences of others within their research. Figure 11 identifies journal databases as a key component of research, and figure 12 shows a diagram of how library/resources help point to an idea. The participants also discussed this concept at some length during the interviews. Representing the feelings of many, one participant captured this idea well by sharing,

I feel like when you’re doing research that each step of the research along the way is connected to the previous things like if I have a research question. Then I go in with, with a certain formulation of it but then as I explore conflicts in the literature space. It’s like the question often changes and, and morphs into a related but slightly different question. It’s quite associative? Yeah, yeah so it ends up forming a network like, like a neural net or something like that.

These conflicts within the existing information could change the way an individual approached their work, exemplified by a participant who shared, “It expands and sometimes shifts. Yeah. So, as you kind of like, illuminate more of the research puzzle. You end up either broadening or narrowing your question, depending on what you find.”

While the problematic nature of information could be felt in how it shaped and reshaped the overall research process, it also led to significant questions about validity, trustworthiness, and saturation. Participants grappled with how to know if they had found enough information, the right information, or the best information. A participant succinctly captured these tensions observing, “But there’s also ones where you look at the article and you go, okay, I am not skilled enough to determine the validity of this yet, or, I don’t have a sense of the field and to know who was in the field and who’s looked at this and like how.” The participant further shared that they attempted to use citation count as a proxy for information quality, saying, “I guess I could say that I’ve started to kind of pick up on like, oh, they’ve been cited a lot in a paper and Google Scholar is good for that, like, you know, you see whose side of what, how many times it’s been cited, all that kind of stuff .” The participant went on to explain that they worked strategically over time to overcome this issue, saying, “I would say that I pretty consistently worked over the course of that stretch of months to get to that point where it’s like, oh okay I’m kind of getting this. But it still feels like I’m having trouble determining validity versus, you know, public acceptance.” These interconnected thoughts from one participant were indicative of observations and approaches shared by several others who attempted to develop or employ strategies to determine the validity or value of the research upon which they planned to base their original work upon.

Beyond approaches used to navigate and assess validity in published information, participants also grappled with how their approaches to original research could result in problematic information. One participant, working on survey-based research, opined,

I feel like to truly gain somebody’s trust in a way that they are going to even answer survey questions totally honestly and from a frame of mind where they’re trying to be unbiased with their own experiences, umm is really difficult. And, you know, like, you want to start asking people about their own trauma, like everyone finds their own way to process and deal with their trauma, and a lot of that is not completely true to what actually happened.

Another shared that they were fascinated by why certain things might be missing from the existing body of literature, saying,

I mean the temperature in your room effects to what your kids learn like and why isn’t that part of the conversation? Or, or the humidity in your room can affect what your kids are learning, or, you know, the exposure to light the amount of windows or the quantity, the air quality, like all that stuff is very, very pertinent but, somehow, like you, know early theorists skipped over it because they had bigger ideas or whatever else. Who, knows? You know?

For this participant, the missing information in a study was concerning. Whereas another participant felt that researchers were aware of the limitations of their work, but that the general public pushed for the information gleaned from studies to be less complex and more generalizable than it actually is, explaining:

And I feel like research scientists take that and we’ll publish. You know, this is what we found, this does not mean that this is applicable to anybody else in any other context. That we’re not saying that this is how humans work. This is just what we found under the specific sets of circumstances and there’s always, you know, that section: opportunities for future research. You know, like I think social scientist researchers [emphasis on researchers] are really good about understanding the limits, but human beings are not. And so you have things that people pick up from data sets without really looking at the limitations or understanding the data well.

For many graduate students in this study, dealing with the information in research created challenges for understanding, organizing, and working with that information.

Models of Research

Existing theories of information-seeking behavior do not fully apply to the experience of graduate students in this study. While there is a strong emotional component for graduate students and other populations studied, both Kuhlthau’s Information Search Process 54 and Dervin’s Sense-Making Theory 55 indicate that individuals feel a level of affective relief and increased certainty or motivation in their research tasks. For the participants in this study, heightened emotions and a sense of uncertainty or imposter syndrome persisted throughout their experiences. This distinction may be related to the graduate student experience particularly or it may be indicative of the difference between research activities aimed at creating new knowledge rather than those focused on synthesizing existing knowledge.

The cultural-historical activity theory, 56 which, to-date, has primarily been used to investigate workplace information use and human computer interaction, is potentially a better fit for explaining the research experiences of graduate students because it intentionally incorporates a social dimension to the experience of learning and communicating. Within the experiences articulated in this study of graduate students, participants routinely mentioned social supports, as well as opportunities to interact with supervisors and peers, as critical for their persistence or success. This theory fails, however, to address the affective dimension more fully realized in the Information Search Process or Sense-Making Theories. Taken as a whole, these findings indicate a single existing theory of information behavior is unlikely to explain the complex, varied, emotionally tenuous, and socially-informed research experiences of graduate students.

Research is Complex

Research at the graduate level is fraught with overwhelming complexity. Graduate students communicated this complexity via their metaphors and drawings of research, and in their stories about research. Conceptual Metaphor Theory suggests that the metaphors students used in their drawings and descriptions of research are not just linguistic devices, but a way of making and expressing meaning for complex phenomena. 57 In choosing to use metaphors to describe research, graduate students struggled with the amorphous nature of research. The sheer size of research at the graduate level often resulted in expressions of overwhelm. Existing studies discuss the emotional components of research, 58 but what was interesting for the graduate students in our interviews was the simultaneous emotional tension between positive and negative emotions throughout the research process.

The stories that graduate students shared about their research were also complex and filled with challenges that often-mirrored great classical journeys like the Odyssey. Despite interviewing graduate students from different disciplines, and at different stages throughout their research process, the similarity in their journeys suggests there is a strong commonality to conducting research at the graduate level. This finding stands in contrast to existing LIS research focused on the literature review aspects of the research process in which disciplinary conventions and differences were distinct. 59 Considering the graduate research experience as a broader journey necessitates the examination of the many barriers that graduate students experience. One of these challenges involves information: finding, making sense of, organizing, and using information. Other research has looked at some of the challenges that graduate students experience related to information; 60 however, what is interesting for librarians from this study is that few students discussed issues with using a database or knowing where to click. Instead, graduate students discussed challenges like knowing when to stop gathering information, how to know which information is essential, or how to know when a topic is too narrow or too broad. Existing research has explored some of these challenges. Moore and Singley highlight that students follow threads of information, no matter how far away it takes them from their original topic of study. 61 Barrett identifies the constant searching for information as a continuous “digging” cycle. 62 This study suggests that surfacing and explicitly addressing the emotional challenges of the research process, as well as information discovery and use, is a critical element for student success that supersedes disciplinary considerations.

Research is Personal

The interviews in this study suggest that research is both intra- and inter-personal for graduate students. Previous literature supports the connection of identity to research. 63 The participants in this study also indicated that their personal identities guide and motivate their research interests. Some linked their values to the impact they hoped their research would have. Those values and impacts included personal enlightenment, improvement of society, and access to specific careers. Students indicated that their dispositions moderate their resiliency to the inevitable complexities encountered during their research process. For example, some mentioned relishing challenges or being innately curious as being protective factors against wavering/failure. However, one participant described how their grief and mental illness undermined their resiliency.

Our findings also show that interpersonal social support influences graduate student research experiences, corroborating past literature. 64 For example, Frick and Pyhältö reported that, in their quantitative study of doctoral students in Finland and South Africa, some of the most prominent positive and negative experiences revolved around supervisor encouragement and, conversely, lack of supervisor support. 65 Participants in this study also indicated that supervisor encouragement and commitment to the relationship created positive experiences for the students. One interviewee mentioned that their chair suggested scholarship opportunities and provided helpful feedback on their work. Their experience suggests a discrepancy in how effective different chairs are at mentoring graduate students, which raises some interesting questions for faculty working with graduate students.

Additionally, students mentioned the importance of connecting with peers to normalize the emotional journey of research and improve their skills. Could more opportunities for graduate student socialization be provided? Is there a way to create more collaboration between peers on research? The physical isolation of the Covid-19 pandemic may have mitigated the social support students in this study received, as interviews were conducted in the summer and fall of 2021. Future studies should review the impact of the current work-from-home culture on graduate student sense of support.

Implications for Practice

The findings of this study show a need for librarians working with graduate student research to decenter the database, to join students on their journey using an affective lens, and to emphasize pedagogy that increases connection and social support across disciplines. Although participants knew they were talking with librarians, they did not discuss database struggles. Instead, they discussed challenges around biases in information, knowing when they had enough, how to deal with too many relevant results, and refining topics. Graduate student research experiences show the need to decenter database demonstrations in information literacy instruction. By this, we mean focusing classroom time on the examination of deeper information contexts and problems, while intentionally moving away from a primary focus on demonstrating database interfaces and features. This pedagogical shift provides more time for deeper grappling with information in instruction and reference appointments while better aligning with the ACRL Framework for Information Literacy. 66

This study reveals that students experience many emotions, including continuous overwhelm, throughout the research process. This sense of overwhelm can impede their ability to progress. The more reference and instruction librarians are involved in research themselves, the better they will be able to personally connect to the student’s emotional journey while also helping students understand how the literature review relates to the rest of the research process. In instruction sessions and reference appointments, librarians can normalize emotions and illuminate a path through the research process so students can move forward despite being overwhelmed.

Participant discussions indicate that graduate student research journeys transcend disciplinary boundaries when considering the research process holistically. Therefore, librarians can consider various models beyond the classroom or reference-based interventions, including co-curricular workshops, asynchronous learning objects, and interdisciplinary orientation sessions. These approaches may be most student-centered when designed in collaboration with others who can bolster social support at other points of the research process beyond the literature review.

Limitations

This study was subject to several limitations. First, as with studies of this nature generally, 67 the participants self-selected, which likely resulted in a participant pool of students who are more interested in, and engaged with, research than might be typical of those in graduate school as a whole. Because of the self-selection, the participants were also not representative of all potential areas of study. This limitation, however, is somewhat balanced by the finding that research experiences had consistent common elements across all participants. Second, the participants were drawn from only three institutions, all of which are situated within the North American higher education context. These limitations are consistent with those routinely found in qualitative research, which often includes participants from only a single data collection site and a specific geographic region. 68

Finally, validating interpretations of subjective data, such as drawings, presents challenges when using a visual research method like graphic elicitation. 69 A few participants commented throughout the interviews that they might change their drawing, which suggests that drawings cannot represent a complete understanding of a large concept like research, and echoes existing findings that interviewees refine and clarify their thinking during the interview process. 70 These limitations are partly why we used visual methods in combination with the second method of semi-structured interviews, so that the images could be connected to the participants’ discussion to guide interpretation.

This paper explores graduate students’ holistic conception of research. Grounding the study in graphic elicitation provided a key to accessing participant affect, allowing the interviewees to delve beyond their immediate reactions, 71 particularly given the complexity and amorphous nature of research. Although the findings of the emotional aspects of research are not entirely unfamiliar within the field of library science, the contribution here exists in how the graduate experience of research differs in the magnitude of uncertainty throughout the research process.

The interviews show that graduate students’ internal values and motivations were protective factors, providing perseverance to complete the research. Similar to prior research, 72 the qualitative data point to the prominence that constructive relationships (supervisors, classmates, community/family, and librarians) empowered graduate students. When asked about research, graduate students in this study, unlike other populations, 73 were less concerned with search experiences but rather with the complexities of information itself, as well as the challenges inherent in navigating unchartered, amorphous processes. Additionally, as seen in this study and supported by Frick and Pyhältö, 74 graduate student research experiences are not bounded by a particular field of study.

These findings imply that graduate students have an experience of research distinct from other populations. This difference suggests that librarians who help graduate students with their research should focus less on teaching database navigation. Instead, librarians need to acknowledge the complexity of the entire research process, take time to help the student process where they are on the journey, and validate the inevitable emotions involved with that experience. Librarians should also provide structure to graduate students so they have at least a sense of how to build a map of their research journey. Additionally, if librarians bring themselves fully into interactions with graduate students, they may serve as empowering social support. The help provided by librarians will, therefore, in part, need to transcend disciplinary boundaries.

Future studies will benefit from incorporating a similar methodology as different ways of thinking about information may help the participants to think more holistically and deeply. Longitudinal studies are needed to study graduate students’ conceptions of research as they move through their programs; this would be particularly salient during the dissertation process. Further, while this study did not collect data on participant identity, participants still mentioned how their identities shaped their research. Future studies could consider the connection of participant positionality to graduate student research.

Acknowledgements

The authors extend our deep appreciation for our research participants, who were generous in sharing their time and experience with us. This work would not exist without their participation, and the impact of the findings would have been significantly muted without their open and candid conversation.

Finally, the authors note that the researchers in this study contributed to the project equally at all points during the process. Authorship order was determined by a collaborative conversation within the team and reflects the order in which each author joined the study.

Appendix A: Interview Questions

• Please draw what research is to you. You will have up to ten minutes to draw and then we’ll ask to see the drawing and to have you tell us about it. If you’re ready to discuss sooner, please let us know.
• Please share your drawing.
• What kinds of emotions are associated with research in your drawing?
• How do you see yourself as a researcher?
• Tell me about your most recent experience with research.
• Why do you do research? OR What do you hope to achieve by engaging in research?
• Last question: Is there anything else you would like to share today?

1. Joy H. McGregor, “Information Seeking and Use: Students’ Thinking and Their Mental Models,” Journal of Youth Services in Libraries 8, no. 1 (1994): 69–76; Glenda M. Insua, Catherine Lantz, and Annie Armstrong, “Navigating Roadblocks: First-Year Writing Challenges through the Lens of the ACRL Framework,” Communications in Information Literacy 12, no. 2 (2018): 86-106; Carol Collier Kuhlthau, “Developing a Model of the Library Search Process: Cognitive and Affective Aspects,” RQ 28, no. 2 (1988): 86-106.

2. D. N. Boote and P. Beile, “Scholars before researchers: On the centrality of the dissertation literature review in research preparation,” Educational Researcher 34, no. 6 (2005): 3-15, https://doi.org/10.3102/0013189X034006003 ; C. S. Harris, “The case for partnering doctoral students with librarians: A synthesis of the literatures,” Library Review 60, no. 7 (2011): 599-620, https://doi.org/10.1108/00242531111153614 ; H. G. Rempel, “A longitudinal assessment of graduate student research behavior and the impact of attending a library literature review workshop,” College & Research Libraries , 71, no. 6 (2010): 532-547, https://doi.org/10.5860/crl-79 ; H. G. Rempel and J. Davidson, “Providing information literacy instruction to graduate students through literature review workshops,” Issues in Science and Technology Librarianship 53, no. 2 (2008): https://doi.org/10.5860/crl-79 .

3. Carol Collier Kuhlthau, Seeking Meaning: A Process Approach to Library and Information Services , 2nd ed. (Westport, CT: Libraries Unlimited, 2004), 53-69.

4. Brenda Dervin, “Sense‐making Theory and Practice: An Overview of User Interests in Knowledge Seeking and Use,” Journal of Knowledge Management 2, no. 2 (1998): 39–43, https://doi.org/10.1108/13673279810249369 .

5. Kirsten A. Foot, “Cultural-Historical Activity Theory: Exploring a Theory to Inform Practice and Research,” Journal of Human Behavior in the Social Environment 24, no. 3 (2014): 338-342, https://doi.org/10.1080/10911359.2013.831011 ; David Allen, Stan Karanasios, and Mira Slavova, “Working with Activity Theory: Context, Technology, and Information Behavior,” Journal of the American Society for Information Science and Technology 62, no. 4 (2011): 776–788, https://onlinelibrary.wiley.com/doi/10.1002/asi.21441 .

6. Amy Catalano, “Patterns of Graduate Students’ Information Seeking Behavior: A Meta-Synthesis of the Literature,” Journal of Documentation 69, no. 2 (2013): 24-27, https://doi.org/10.1108/00220411311300066 ; Geraldine Delaney and Jessica Bates, “How Can the University Library Better Meet the Information Needs of Research Students? Experiences From Ulster University,” New Review of Academic Librarianship 24, no. 1 (2018): 73-76, https://doi.org/10.1080/13614533.2017.1384267 .

7. Andy Barrett, “The Information-Seeking Habits of Graduate Student Researchers in the Humanities,” The Journal of Academic Librarianship 31, no. 4 (2005): 325, https://doi.org/10.1016/j.acalib.2005.04.005 ; Monica Moore and Emily Singley, “Understanding the Information Behaviors of Doctoral Students: An Exploratory Study,” Portal: Libraries and the Academy 19, no. 2 (2019): 282-284, https://doi.org/10.1353/pla.2019.0016 .

8. Corinne Bishop, “An Investigation of the Information Practices of Education Doctoral Students.” PhD, University of Central Florida, 2015, 42-51, https://stars.library.ucf.edu/etd/56/ ; Amy Jo Catalano, “Using ACRL Standards to Assess the Information Literacy of Graduate Students in an Education Program,” Evidence Based Library and Information Practice 5, no. 4 (2010): 23-24, https://doi.org/10.18438/B8V62B ; Boote and Beile, “Scholars before researchers, 3-15.

9. B. Ann Boyce et al., “Doctoral Students’ Perspectives on Their Training as Researchers in Higher Education,” Quest, 71, no. 3 (July 3, 2019): 277–288, https://doi.org/10.1080/00336297.2019.1618065 .

10. Roberta E. Rikli, “The Role of Master’s Institutions in Developing Researchers: Rethinking the Master Plan,” Quest 61, no. 1 (2009): 59–73, https://doi.org/10.1080/00336297.2009.10483601 .

11. Mojca Jurisevic, “Postgraduate Students’ Perception of Creativity in the Research Process,” Center for Educational Policy Studies Journal 1, no. 1 (2011): 169-190.

12. Chris M. Golde, “Applying Lessons from Professional Education to the Preparation of the Professoriate,” New Directions for Teaching and Learning Revitalizing Ourselves and Our Institutions 113 (2008): 17-25.

13. Boyce et al., “Doctoral Students’ Perspectives,” 277–288.

14. Insa Wessels et al., “Beyond Cognition: Experts’ Views on Affective-Motivational Research Dispositions in the Social Sciences,” Frontiers in Psychology 9 (2018): 1-10, https://doi.org/10.3389/fpsyg.2018.01300 .

15. Kuhlthau, “Developing a Model,” 232-242.

16. Leon A. Jakobovits and Diane Nahl-Jakobovits, “Learning the Library: Taxonomy of Skills and Errors,” College and Research Libraries 48, no. 3 (1987): 203-213.

17. Diane Nahl and Carol Tenopir, “Affective and Cognitive Searching Behavior of Novice End-Users of a Full-Text Database,” Journal of the American Society for Information Science 47, no. 4 (1996): 276-286; Ellysa Cahoy and Robert Schroeder, “Embedding Affective Learning Outcomes in Library Instruction,” Communications in Information Literacy 6, no. 1 (2012): 73-90, https://doi.org/10.15760/comminfolit.2012.6.1.119 .

18. Diane Nahl-Jakobovits and Leon A. Jakobovits, “Bibliographic Instructional Design for Information Literacy: Integrating Affective and Cognitive Objectives,” Research Strategies 11, no. 12 (1993): 73-88.

19. Sarah Rose Fitzgerald, “The Role of Affect in the Information Seeking of Productive Scholars,” The Journal of Academic Librarianship 44, no. 2 (2018): 263–68.

20. Jakobovits and Nahl-Jakobovits, “Learning the Library,” 203-213; Kuhlthau, “Developing a Model,” 232-242.

21. Richard J. Davidson, Klaus R. Scherer, and H. Hill Goldsmith, eds., Handbook of Affective Sciences, Series in Affective Science (Oxford: Oxford University Press, 2003); Reijo Savolainen, “Emotions as Motivators for Information Seeking: A Conceptual Analysis,” Library & Information Science Research 36, no. 1 (2014): 59–65.

22. Robert Schroeder and Ellysa Stern Cahoy, “Valuing Information Literacy: Affective Learning and the ACRL Standards,” Portal: Libraries and the Academy 10, no. 2 (2010): 75; Cahoy and Schroeder, “Embedding Affective Learning,” 73-90.

23. Association of College and Research Libraries Board, “Framework for Information Literacy for Higher Education,” Association of College & Research Libraries (2015): 1-36, https://www.ala.org/acrl/standards/ilframework .

24. Paula R. Dempsey and Heather Jagman, “‘I Felt Like Such a Freshman’: First-Year Students Crossing the Library Threshold,” Portal: Libraries and the Academy 16, no. 1 (2016): 89–107, https://doi.org/10.1353/pla.2016.0011 ; Insua, Lantz, and Armstrong, “Navigating Roadblocks,” 86-106.

25. Diane Nahl, “Affective Elaborations in Boolean Search Instructions for Novices: Effects on Comprehension, Self-Confidence, and Error Type,” Proceedings of the ASIS Annual Meeting 32 (1995): 69-76; Rebekah Willson and Lisa M. Given, “Student Search Behaviour in an Online Public Access Catalogue: An Examination of ‘Searching Mental Models’ and ‘Searcher Self-Concept,’” Information Research: An International Electronic Journal 19, no. 3 (2014); Dempsey and Jagman, “‘I Felt Like,’” 89–107; Rui Wang, “Assessment for One-Shot Library Instruction: A Conceptual Approach,” portal: Libraries and the Academy 16, no. 3 (2016): 619–48; Insua, Lantz, and Armstrong, “Navigating Roadblocks,” 86-106.

26. Kuhlthau, “Developing a Model,” 232-242.

27. Fitzgerald, “The Role of Affect,” 263–268.

28. Jette Hyldegaard, “Collaborative Information Behaviour—Exploring Kuhlthau’s Information Search Process Model in a Group-Based Educational Setting,” Information Processing & Management 42, no. 1 (2006): 276–298.

29. Kuhlthau, “Developing a Model,” 232-242; Willson and Given, “Student Search Behaviour”; Fitzgerald, “The Role of Affect,” 263–268; Insua, Lantz, and Armstrong, “Navigating Roadblocks,” 86-106.

30. Diane Nahl, “Affective Elaborations,” 69-76; Nahl and Crol Tenopir, “Affective and Cognitive Searching,” 276-286.

31. Nahl and Crol Tenopir, “Affective and Cognitive Searching,” 276-286.

32. Gillian Rose, Visual Methodologies: An Introduction to Researching with Visual Materials , 3rd ed. (Los Angeles: SAGE, 2012).

33. Shailoo Bedi and Jenaya Webb, Visual Research Methods: An Introduction for Library and Information Studies (London: Facet Publishing, 2020).

34. Angela Pollak, “Visual Research in LIS: Complementary and Alternative Methods,” Library & Information Science Research 39, no. 2 (2017): 102-104.

35. Jenna Hartel, “Adventures in Visual Analysis,” The Visual Methodologies Journal 5, no. 1 (2017): 82-83.

36. Pamela McKinney and Chloe Cook, “Student Conceptions of Group Work: Visual Research into LIS Student Group Work Using the Draw-and-Write Technique,” Journal of Education for Library & Information Science 59, no. 4 (2018): 221-224.

37. Lise Doucette and Kristin Hoffmann, “Conceptions of Research Among Academic Librarians and Archivists,” Canadian Journal of Academic Librarianship 5 (2019): 9, https://doi.org/10.33137/cjal-rcbu.v5.30417 .

38. Patricia Bryans and Sharon Mavin, “Visual Images: A Technique to Surface Conceptions of Research and Researchers,” Qualitative Research in Organizations and Management 1, no. 2 (2006): 122-126, https://doi.org/10.1108/17465640610686370 .

39. Doucette and Hoffmann, “Conceptions of Research,” 1–25.

40. Bryans and Mavin, “Visual Images,” 117-118.

41. John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 2nd ed. (Thousand Oaks, CA: SAGE Publications, 2003), 14.

42. Corrine Glesne, Becoming Qualitative Researchers: An Introduction , 5th ed. (Boston: Pearson, 2016), 96-123; Joseph A. Maxwell, Qualitative Research Design: An Interactive Approach , 3rd ed. (Los Angeles: SAGE Publications, 2013), 88-90.

43. Glesne, Becoming Qualitative Researchers , 88-90.

44. Pollak, “Visual Research in LIS,” 102.

45. Julia Ellis et al., “Draw Me a Picture, Tell Me a Story: Evoking Memory and Supporting Analysis through Pre-Interview Drawing Activities,” Alberta Journal of Educational Research 58, no. 4 (2012): 504.

46. Pollak, “Visual Research in LIS,” 102-104; Alison Hicks and Annemaree Lloyd, “Seeing Information: Visual Methods as Entry Points to Information Practices,” Journal of Librarianship & Information Science 50, no. 3 (2018): 232.

47. Hartel, “Adventures in Visual Analysis,” 82-83; McKinney and Cook, “Student Conceptions,” 221; Doucette and Hoffmann, “Conceptions of Research,” 1–25.

48. Maxwell, Qualitative Research Design , 102; Pollak, “Visual Research in LIS,” 104.

49. Maxwell, Qualitative Research Design , 136-137.

50. Johnny Saldana, The Coding Manual for Qualitative Researchers , 3rd ed. (Los Angeles: SAGE Publications, 2016), 298.

51. Glesne, Becoming Qualitative Researchers , 67-68.

52. Maxwell, Qualitative Research Design , 88-100.

53. All illustrations in this publication are photographs of original drawings created by participants in this study. Participants own the copyright to their drawings and the research team has consent from research participants to use the photos of drawings displayed in this publication.

54. Kuhlthau, Seeking Meaning , 89-106.

55. Dervin, “Sense‐making Theory and Practice,” 39–43.

56. Allen, Karanasios, and Slavova, “Working with Activity Theory,” 776–788; Foot, “Cultural-Historical Activity Theory,” 338-342.

57. Raymond W. Gibbs, “Evaluating Conceptual Metaphor Theory,” Discourse Processes 48, no. 8 (2011): 529–562, https://doi.org/10.1080/0163853X.2011.606103 .

58. Jakobovits and Nahl-Jakobovits, “Learning the Library,” 203-213; Kuhlthau, “Developing a Model,” 232-242.

59. Rempel, “A longitudinal assessment,” 532-547.

60. Catalano, “Patterns of Graduate Students’,” 267-268; Catalano, “Using ACRL Standards,” 25-27.

61. Moore and Singley, “Understanding the Information,” 288-289.

62. Barrett, “The Information-Seeking Habits,” 327.

63. Montserrat Castelló et al., “What perspectives underlie ‘researcher identity’? A review of two decades of empirical studies”, Higher Education 81, no. 3 (2021): 573-86, https://doi.org/10.1007/s10734-020-00557-8 ; Frances Giampapa, “The Politics of ‘Being and Becoming’ a Researcher: Identity, Power, and Negotiating the Field,” Journal of Language, Identity & Education 10, no. 3 (2011): 133-37, https://doi.org/10.1080/15348458.2011.585304 .

64. Kelley A. Tompkins, Kierra Brecht, Brock Tucker, Lucia L. Neander, and Joshua K. Swift, “Who matters most? The contribution of faculty, student-peers, and outside support in predicting graduate student satisfaction,” Training and Education in Professional Psychology, 10, no. 2 (2016): 102–108, https://doi.org/10.1037/tep0000115 ; Julaine Rigg, Jonathon Day, and Howard Adler, “Emotional exhaustion in graduate students: The role of engagement, self-efficacy and social support,” Journal of Educational and Developmental Psychology 3, no. 2 (2013): 138.

65. B. Liezel Frick and Kirsi Pyhältö, “Experiences of the Doctoral Journey: A Cross-National Perspective,” Innovations in Education and Teaching International 59, no. 1 (January 2, 2022): 70–81, https://doi.org/10.1080/14703297.2020.1811132 .

66. Association of College and Research Libraries Board, “Framework for Information Literacy for Higher Education,” Association of College & Research Libraries, 1-36.

67. Catherine Marshall and Gretchen B. Rossman, Designing Qualitative Research , 3rd ed. (Thousand Oaks, CA: SAGE Publications, 1999), 51-93.

68. Creswell, Research Design , 179-183.

69. Pollak, “Visual Research in LIS,” 104.

70. Glesne, Becoming Qualitative Researchers, 111-116.

71. Ellis et al., “Draw Me a Picture, Tell Me a Story,” 504.

72. Frick and Pyhältö, “Experiences of the Doctoral Journey,” 70–81.

73. Kuhlthau, Seeking Meaning , 127-144.

74. Frick and Pyhältö, “Experiences of the Doctoral Journey,” 70–81.

* Alissa Droog is Assistant Professor and Education and Social Sciences Librarian at Northern Illinois University, email: [email protected] ; Kari D. Weaver is Learning, Teaching, and Instructional Design Librarian at the University of Waterloo, email: [email protected] ; and Frances Brady is Reference and Instruction Librarian at Adler University, email: [email protected] . ©2024 Alissa Droog, Kari D. Weaver, and Frances Brady, Attribution-NonCommercial ( https://creativecommons.org/licenses/by-nc/4.0/ ) CC BY-NC.

Article Views (Last 12 Months)

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

© 2024 Association of College and Research Libraries , a division of the American Library Association

Print ISSN: 0010-0870 | Online ISSN: 2150-6701

ALA Privacy Policy

ISSN: 2150-6701

Cognitive and Psychological Sciences

Focus areas.

We share a belief that transformational advances in understanding mind, brain, and behavior will occur at the boundaries of disciplines, across levels of analysis, and through a diversity of approaches, paradigms, and perspectives. Our interdisciplinary character is captured by this image, which crosses traditional areas with our cross-cutting research themes.

The heat map shows how our faculty self-affiliate.

Our faculty conduct cutting-edge, award-winning research in these areas using a range of approaches and methods, and we are highly collaborative both inside and outside of Brown. Explore this page to learn more about CoPsy's research interests!

Behavioral Neuroscience/Comparative

Our Behavioral Neuroscience research delves into the neural foundations and computational models that drive critical processes, such as interval timing, emotional development, and auditory perception. We explore the complexities of memory, and higher cognitive functions. Our work also extends to understanding the intricacies of canine communication and social cognition, offering insights that bridge human and animal behavior.

Recent Example Publication : Felsche, E., Völter, C. J., Herrmann, E., Seed, A. M., & Buchsbaum, D. (2024). How can I find what I want? Can children, chimpanzees and capuchin monkeys form abstract representations to guide their behavior in a sampling task?.   Cognition ,  245 , 105721.

Image taken from original article and available for use under a Creative Commons License  (CC BY-NC-ND 4.0); no changes were made.

Daphna Buchsbaum

Ruth colwill, andrea megela simmons, cognitive neuroscience.

Our Cognitive Neuroscience research uncovers the neural mechanisms underlying essential cognitive functions like attention, perception, and learning. We investigate how the brain manages memory, regulates emotions, and exercises executive control, all of which are crucial for decision-making. This work provides a deeper understanding of the intricate processes that shape our thoughts and behaviors.

Recent Example Publication : Kikumoto, A., Bhandari, A., Shibata, K., & Badre, D. (in press at Nature Communications ). A transient high-dimensional geometry affords stable conjunctive subspaces for efficient action election .

Image taken from original article and available for use under a Creative Commons License  (CC BY-NC-ND 4.0); the orientation of the figure was adjusted to be horizontal rather then vertical.

David Badre

Serra favila, oriel feldmanhall, elena festa, michael frank, william heindel, takeo watanabe, development.

Our Development research explores the foundations of cognition in both human and animal models. We delve into how visual attention, learning, and memory evolve, alongside the development of causal reasoning, pretend play, social behavior, language, and perception. This research provides valuable insights into the processes that shape cognitive development across species.

Recent Example Publication : Brody, G., & Feiman, R. (2024). Mapping words to the world: Adults, but not children, understand how mismatching descriptions refer .  Journal of Experimental Psychology: General .

Copyright belongs to Gabor Brody, first-author of the publication.

Roman Feiman

Julia marshall, james morgan, higher-level cognition.

Our Higher-Level Cognition research delves into the complexities of human memory, learning, and cognitive control. We explore how people make inductive inferences, reason causally, and navigate decision-making. Our work also examines the development of moral reasoning, social cognition, and theory of mind, shedding light on the intricate processes that underpin human thought and social interactions.

Recent Example Publication : Light, N., Fernbach, P. M., Rabb, N., Geana, M. V., & Sloman, S. A. (2022). Knowledge overconfidence is associated with anti-consensus views on controversial scientific issues .  Science Advances ,  8 (29), eabo0038.

Joachim Krueger

David levari, bertram f. malle, steven sloman, neural/computational models of mind, brain, and behavior.

Our research in Computational Models focuses on creating neural and computational frameworks to understand key processes like motor control, vision, categorization, learning, reasoning, and language. These models provide powerful insights into the mechanisms driving human cognition, enabling us to simulate and predict complex mental functions with precision.

Recent Example Publication : Jaskir, A., & Frank, M. J. (2023). On the normative advantages of dopamine and striatal opponency for learning and choice .  Elife ,  12 , e85107.

Thomas Serre

Perception and action.

Our Perception and Action research combines computational, psychophysical, and ecological approaches to unravel how we perceive shape and motion, recognize objects and scenes, and process auditory events. We also investigate the mechanisms behind attention, perceptual learning, and the control of action, offering comprehensive insights into how we interact with and interpret the world around us.

Recent Example Publication : Fel, T., Boutin, V., Béthune, L., Cadène, R., Moayeri, M., Andéol, L., ... & Serre, T. (2024). A holistic approach to unifying automatic concept extraction and concept importance estimation .  Advances in Neural Information Processing Systems ,  36 .

Image lawfully reproduced from NeurIPS.

Fulvio Domini

Joo-hyun song, william warren, leslie welch, social psychology.

Our Social Psychology research delves into how we understand and navigate the social world. We explore social cognition, theory of mind, and moral judgment, as well as how we perceive personality and interact with different situations. Our work also examines self-image, social projection, intergroup perception, and strategic behavior, providing deep insights into the complexities of human social behavior.

Recent Example Publication : Son, J. Y., Bhandari, A., & FeldmanHall, O. (2023). Abstract cognitive maps of social network structure aid adaptive inference .  Proceedings of the National Academy of Sciences ,  120 (47), e2310801120.

Malik Boykin

Jason okonofua.

Quantitative analysis of momentum in tennis matches summary

• Zhao, Chenxi
• Lei, Yiming

The tennis match is full of variables, and the outcome is affected by many factors, such as serving skills, physical strength and mental state. Good psychological quality is conducive to the stable performance of athletes. It is important to study these factors to improve the value of tennis matches. Firstly, this problem first selects indicators from the data set to establish a comprehensive evaluation system, including multiple indicators such as average serve speed and ACE number. Then the principal component analysis method is used to determine the number of retained principal components and calculate the weight. Then, according to the weight and index data of principal components, the comprehensive evaluation score of players in each round is calculated. Finally, through the calculation of the model, the performance of the players in each round can be quantitatively assessed and presented in a visual way. Secondly, we tested the scoring sequence of the two players in the first 10 matches through the Ljung-Box test, and the results showed that the score sequence of player 1 was autocorrelated in each match, while the score sequence of player 2 was random in only one match. Therefore, the performance of supporting players is affected by a variety of factors and is not random. Further analysis shows that the first two principal components of the players have a high correlation with the total score, while the third principal component has a low correlation. Thirdly, this question analyzes the performance of two tennis players in the match through the principal component comprehensive evaluation model, divides their performance into strengths and weaknesses, and adds corresponding labels to the data set. Then, a variety of machine learning classification models are used to dig and distinguish the player status. In the model evaluation, the random forest model performed the best, which could accurately identify the "momentum" change of players. Finally, by analyzing the characteristic importance of the model, some suggestions are put forward, such as improving ball speed, increasing service score and reducing unforced errors, so as to improve players' match performance. Fourthly, this question is tested in three randomly selected matches based on the trained random forest model. The results show that the prediction results of the model are highly correlated with the actual competition situation, which proves its accuracy. In addition, based on the research results, the model promotion process suitable for different competition types is provided. Based on the results of the analysis, this study provides suggestions to help coaches take advantage of the role of players' momentum in the game. At the same time, it puts forward the measures to deal with various events that affect the course of tennis match.