review article quantitative research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Recent quantitative research on determinants of health in high income countries: A scoping review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Centre for Health Economics Research and Modelling Infectious Diseases, Vaccine and Infectious Disease Institute, University of Antwerp, Antwerp, Belgium

Roles Conceptualization, Data curation, Funding acquisition, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

• Vladimira Varbanova, 
• Philippe Beutels

• Published: September 17, 2020
• https://doi.org/10.1371/journal.pone.0239031
• Peer Review
• Reader Comments

Identifying determinants of health and understanding their role in health production constitutes an important research theme. We aimed to document the state of recent multi-country research on this theme in the literature.

We followed the PRISMA-ScR guidelines to systematically identify, triage and review literature (January 2013—July 2019). We searched for studies that performed cross-national statistical analyses aiming to evaluate the impact of one or more aggregate level determinants on one or more general population health outcomes in high-income countries. To assess in which combinations and to what extent individual (or thematically linked) determinants had been studied together, we performed multidimensional scaling and cluster analysis.

Sixty studies were selected, out of an original yield of 3686. Life-expectancy and overall mortality were the most widely used population health indicators, while determinants came from the areas of healthcare, culture, politics, socio-economics, environment, labor, fertility, demographics, life-style, and psychology. The family of regression models was the predominant statistical approach. Results from our multidimensional scaling showed that a relatively tight core of determinants have received much attention, as main covariates of interest or controls, whereas the majority of other determinants were studied in very limited contexts. We consider findings from these studies regarding the importance of any given health determinant inconclusive at present. Across a multitude of model specifications, different country samples, and varying time periods, effects fluctuated between statistically significant and not significant, and between beneficial and detrimental to health.

Conclusions

We conclude that efforts to understand the underlying mechanisms of population health are far from settled, and the present state of research on the topic leaves much to be desired. It is essential that future research considers multiple factors simultaneously and takes advantage of more sophisticated methodology with regards to quantifying health as well as analyzing determinants’ influence.

Citation: Varbanova V, Beutels P (2020) Recent quantitative research on determinants of health in high income countries: A scoping review. PLoS ONE 15(9): e0239031. https://doi.org/10.1371/journal.pone.0239031

Editor: Amir Radfar, University of Central Florida, UNITED STATES

Received: November 14, 2019; Accepted: August 28, 2020; Published: September 17, 2020

Copyright: © 2020 Varbanova, Beutels. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting Information files.

Funding: This study (and VV) is funded by the Research Foundation Flanders ( https://www.fwo.be/en/ ), FWO project number G0D5917N, award obtained by PB. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Identifying the key drivers of population health is a core subject in public health and health economics research. Between-country comparative research on the topic is challenging. In order to be relevant for policy, it requires disentangling different interrelated drivers of “good health”, each having different degrees of importance in different contexts.

“Good health”–physical and psychological, subjective and objective–can be defined and measured using a variety of approaches, depending on which aspect of health is the focus. A major distinction can be made between health measurements at the individual level or some aggregate level, such as a neighborhood, a region or a country. In view of this, a great diversity of specific research topics exists on the drivers of what constitutes individual or aggregate “good health”, including those focusing on health inequalities, the gender gap in longevity, and regional mortality and longevity differences.

The current scoping review focuses on determinants of population health. Stated as such, this topic is quite broad. Indeed, we are interested in the very general question of what methods have been used to make the most of increasingly available region or country-specific databases to understand the drivers of population health through inter-country comparisons. Existing reviews indicate that researchers thus far tend to adopt a narrower focus. Usually, attention is given to only one health outcome at a time, with further geographical and/or population [ 1 , 2 ] restrictions. In some cases, the impact of one or more interventions is at the core of the review [ 3 – 7 ], while in others it is the relationship between health and just one particular predictor, e.g., income inequality, access to healthcare, government mechanisms [ 8 – 13 ]. Some relatively recent reviews on the subject of social determinants of health [ 4 – 6 , 14 – 17 ] have considered a number of indicators potentially influencing health as opposed to a single one. One review defines “social determinants” as “the social, economic, and political conditions that influence the health of individuals and populations” [ 17 ] while another refers even more broadly to “the factors apart from medical care” [ 15 ].

In the present work, we aimed to be more inclusive, setting no limitations on the nature of possible health correlates, as well as making use of a multitude of commonly accepted measures of general population health. The goal of this scoping review was to document the state of the art in the recent published literature on determinants of population health, with a particular focus on the types of determinants selected and the methodology used. In doing so, we also report the main characteristics of the results these studies found. The materials collected in this review are intended to inform our (and potentially other researchers’) future analyses on this topic. Since the production of health is subject to the law of diminishing marginal returns, we focused our review on those studies that included countries where a high standard of wealth has been achieved for some time, i.e., high-income countries belonging to the Organisation for Economic Co-operation and Development (OECD) or Europe. Adding similar reviews for other country income groups is of limited interest to the research we plan to do in this area.

In view of its focus on data and methods, rather than results, a formal protocol was not registered prior to undertaking this review, but the procedure followed the guidelines of the PRISMA statement for scoping reviews [ 18 ].

We focused on multi-country studies investigating the potential associations between any aggregate level (region/city/country) determinant and general measures of population health (e.g., life expectancy, mortality rate).

Within the query itself, we listed well-established population health indicators as well as the six world regions, as defined by the World Health Organization (WHO). We searched only in the publications’ titles in order to keep the number of hits manageable, and the ratio of broadly relevant abstracts over all abstracts in the order of magnitude of 10% (based on a series of time-focused trial runs). The search strategy was developed iteratively between the two authors and is presented in S1 Appendix . The search was performed by VV in PubMed and Web of Science on the 16 th of July, 2019, without any language restrictions, and with a start date set to the 1 st of January, 2013, as we were interested in the latest developments in this area of research.

Eligibility criteria

Records obtained via the search methods described above were screened independently by the two authors. Consistency between inclusion/exclusion decisions was approximately 90% and the 43 instances where uncertainty existed were judged through discussion. Articles were included subject to meeting the following requirements: (a) the paper was a full published report of an original empirical study investigating the impact of at least one aggregate level (city/region/country) factor on at least one health indicator (or self-reported health) of the general population (the only admissible “sub-populations” were those based on gender and/or age); (b) the study employed statistical techniques (calculating correlations, at the very least) and was not purely descriptive or theoretical in nature; (c) the analysis involved at least two countries or at least two regions or cities (or another aggregate level) in at least two different countries; (d) the health outcome was not differentiated according to some socio-economic factor and thus studied in terms of inequality (with the exception of gender and age differentiations); (e) mortality, in case it was one of the health indicators under investigation, was strictly “total” or “all-cause” (no cause-specific or determinant-attributable mortality).

Data extraction

The following pieces of information were extracted in an Excel table from the full text of each eligible study (primarily by VV, consulting with PB in case of doubt): health outcome(s), determinants, statistical methodology, level of analysis, results, type of data, data sources, time period, countries. The evidence is synthesized according to these extracted data (often directly reflected in the section headings), using a narrative form accompanied by a “summary-of-findings” table and a graph.

Search and selection

The initial yield contained 4583 records, reduced to 3686 after removal of duplicates ( Fig 1 ). Based on title and abstract screening, 3271 records were excluded because they focused on specific medical condition(s) or specific populations (based on morbidity or some other factor), dealt with intervention effectiveness, with theoretical or non-health related issues, or with animals or plants. Of the remaining 415 papers, roughly half were disqualified upon full-text consideration, mostly due to using an outcome not of interest to us (e.g., health inequality), measuring and analyzing determinants and outcomes exclusively at the individual level, performing analyses one country at a time, employing indices that are a mixture of both health indicators and health determinants, or not utilizing potential health determinants at all. After this second stage of the screening process, 202 papers were deemed eligible for inclusion. This group was further dichotomized according to level of economic development of the countries or regions under study, using membership of the OECD or Europe as a reference “cut-off” point. Sixty papers were judged to include high-income countries, and the remaining 142 included either low- or middle-income countries or a mix of both these levels of development. The rest of this report outlines findings in relation to high-income countries only, reflecting our own primary research interests. Nonetheless, we chose to report our search yield for the other income groups for two reasons. First, to gauge the relative interest in applied published research for these different income levels; and second, to enable other researchers with a focus on determinants of health in other countries to use the extraction we made here.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0239031.g001

Health outcomes

The most frequent population health indicator, life expectancy (LE), was present in 24 of the 60 studies. Apart from “life expectancy at birth” (representing the average life-span a newborn is expected to have if current mortality rates remain constant), also called “period LE” by some [ 19 , 20 ], we encountered as well LE at 40 years of age [ 21 ], at 60 [ 22 ], and at 65 [ 21 , 23 , 24 ]. In two papers, the age-specificity of life expectancy (be it at birth or another age) was not stated [ 25 , 26 ].

Some studies considered male and female LE separately [ 21 , 24 , 25 , 27 – 33 ]. This consideration was also often observed with the second most commonly used health index [ 28 – 30 , 34 – 38 ]–termed “total”, or “overall”, or “all-cause”, mortality rate (MR)–included in 22 of the 60 studies. In addition to gender, this index was also sometimes broken down according to age group [ 30 , 39 , 40 ], as well as gender-age group [ 38 ].

While the majority of studies under review here focused on a single health indicator, 23 out of the 60 studies made use of multiple outcomes, although these outcomes were always considered one at a time, and sometimes not all of them fell within the scope of our review. An easily discernable group of indices that typically went together [ 25 , 37 , 41 ] was that of neonatal (deaths occurring within 28 days postpartum), perinatal (fetal or early neonatal / first-7-days deaths), and post-neonatal (deaths between the 29 th day and completion of one year of life) mortality. More often than not, these indices were also accompanied by “stand-alone” indicators, such as infant mortality (deaths within the first year of life; our third most common index found in 16 of the 60 studies), maternal mortality (deaths during pregnancy or within 42 days of termination of pregnancy), and child mortality rates. Child mortality has conventionally been defined as mortality within the first 5 years of life, thus often also called “under-5 mortality”. Nonetheless, Pritchard & Wallace used the term “child mortality” to denote deaths of children younger than 14 years [ 42 ].

As previously stated, inclusion criteria did allow for self-reported health status to be used as a general measure of population health. Within our final selection of studies, seven utilized some form of subjective health as an outcome variable [ 25 , 43 – 48 ]. Additionally, the Health Human Development Index [ 49 ], healthy life expectancy [ 50 ], old-age survival [ 51 ], potential years of life lost [ 52 ], and disability-adjusted life expectancy [ 25 ] were also used.

We note that while in most cases the indicators mentioned above (and/or the covariates considered, see below) were taken in their absolute or logarithmic form, as a—typically annual—number, sometimes they were used in the form of differences, change rates, averages over a given time period, or even z-scores of rankings [ 19 , 22 , 40 , 42 , 44 , 53 – 57 ].

Regions, countries, and populations

Despite our decision to confine this review to high-income countries, some variation in the countries and regions studied was still present. Selection seemed to be most often conditioned on the European Union, or the European continent more generally, and the Organisation of Economic Co-operation and Development (OECD), though, typically, not all member nations–based on the instances where these were also explicitly listed—were included in a given study. Some of the stated reasons for omitting certain nations included data unavailability [ 30 , 45 , 54 ] or inconsistency [ 20 , 58 ], Gross Domestic Product (GDP) too low [ 40 ], differences in economic development and political stability with the rest of the sampled countries [ 59 ], and national population too small [ 24 , 40 ]. On the other hand, the rationales for selecting a group of countries included having similar above-average infant mortality [ 60 ], similar healthcare systems [ 23 ], and being randomly drawn from a social spending category [ 61 ]. Some researchers were interested explicitly in a specific geographical region, such as Eastern Europe [ 50 ], Central and Eastern Europe [ 48 , 60 ], the Visegrad (V4) group [ 62 ], or the Asia/Pacific area [ 32 ]. In certain instances, national regions or cities, rather than countries, constituted the units of investigation instead [ 31 , 51 , 56 , 62 – 66 ]. In two particular cases, a mix of countries and cities was used [ 35 , 57 ]. In another two [ 28 , 29 ], due to the long time periods under study, some of the included countries no longer exist. Finally, besides “European” and “OECD”, the terms “developed”, “Western”, and “industrialized” were also used to describe the group of selected nations [ 30 , 42 , 52 , 53 , 67 ].

As stated above, it was the health status of the general population that we were interested in, and during screening we made a concerted effort to exclude research using data based on a more narrowly defined group of individuals. All studies included in this review adhere to this general rule, albeit with two caveats. First, as cities (even neighborhoods) were the unit of analysis in three of the studies that made the selection [ 56 , 64 , 65 ], the populations under investigation there can be more accurately described as general urban , instead of just general. Second, oftentimes health indicators were stratified based on gender and/or age, therefore we also admitted one study that, due to its specific research question, focused on men and women of early retirement age [ 35 ] and another that considered adult males only [ 68 ].

Data types and sources

A great diversity of sources was utilized for data collection purposes. The accessible reference databases of the OECD ( https://www.oecd.org/ ), WHO ( https://www.who.int/ ), World Bank ( https://www.worldbank.org/ ), United Nations ( https://www.un.org/en/ ), and Eurostat ( https://ec.europa.eu/eurostat ) were among the top choices. The other international databases included Human Mortality [ 30 , 39 , 50 ], Transparency International [ 40 , 48 , 50 ], Quality of Government [ 28 , 69 ], World Income Inequality [ 30 ], International Labor Organization [ 41 ], International Monetary Fund [ 70 ]. A number of national databases were referred to as well, for example the US Bureau of Statistics [ 42 , 53 ], Korean Statistical Information Services [ 67 ], Statistics Canada [ 67 ], Australian Bureau of Statistics [ 67 ], and Health New Zealand Tobacco control and Health New Zealand Food and Nutrition [ 19 ]. Well-known surveys, such as the World Values Survey [ 25 , 55 ], the European Social Survey [ 25 , 39 , 44 ], the Eurobarometer [ 46 , 56 ], the European Value Survey [ 25 ], and the European Statistics of Income and Living Condition Survey [ 43 , 47 , 70 ] were used as data sources, too. Finally, in some cases [ 25 , 28 , 29 , 35 , 36 , 41 , 69 ], built-for-purpose datasets from previous studies were re-used.

In most of the studies, the level of the data (and analysis) was national. The exceptions were six papers that dealt with Nomenclature of Territorial Units of Statistics (NUTS2) regions [ 31 , 62 , 63 , 66 ], otherwise defined areas [ 51 ] or cities [ 56 ], and seven others that were multilevel designs and utilized both country- and region-level data [ 57 ], individual- and city- or country-level [ 35 ], individual- and country-level [ 44 , 45 , 48 ], individual- and neighborhood-level [ 64 ], and city-region- (NUTS3) and country-level data [ 65 ]. Parallel to that, the data type was predominantly longitudinal, with only a few studies using purely cross-sectional data [ 25 , 33 , 43 , 45 – 48 , 50 , 62 , 67 , 68 , 71 , 72 ], albeit in four of those [ 43 , 48 , 68 , 72 ] two separate points in time were taken (thus resulting in a kind of “double cross-section”), while in another the averages across survey waves were used [ 56 ].

In studies using longitudinal data, the length of the covered time periods varied greatly. Although this was almost always less than 40 years, in one study it covered the entire 20 th century [ 29 ]. Longitudinal data, typically in the form of annual records, was sometimes transformed before usage. For example, some researchers considered data points at 5- [ 34 , 36 , 49 ] or 10-year [ 27 , 29 , 35 ] intervals instead of the traditional 1, or took averages over 3-year periods [ 42 , 53 , 73 ]. In one study concerned with the effect of the Great Recession all data were in a “recession minus expansion change in trends”-form [ 57 ]. Furthermore, there were a few instances where two different time periods were compared to each other [ 42 , 53 ] or when data was divided into 2 to 4 (possibly overlapping) periods which were then analyzed separately [ 24 , 26 , 28 , 29 , 31 , 65 ]. Lastly, owing to data availability issues, discrepancies between the time points or periods of data on the different variables were occasionally observed [ 22 , 35 , 42 , 53 – 55 , 63 ].

Health determinants

Together with other essential details, Table 1 lists the health correlates considered in the selected studies. Several general categories for these correlates can be discerned, including health care, political stability, socio-economics, demographics, psychology, environment, fertility, life-style, culture, labor. All of these, directly or implicitly, have been recognized as holding importance for population health by existing theoretical models of (social) determinants of health [ 74 – 77 ].

https://doi.org/10.1371/journal.pone.0239031.t001

It is worth noting that in a few studies there was just a single aggregate-level covariate investigated in relation to a health outcome of interest to us. In one instance, this was life satisfaction [ 44 ], in another–welfare system typology [ 45 ], but also gender inequality [ 33 ], austerity level [ 70 , 78 ], and deprivation [ 51 ]. Most often though, attention went exclusively to GDP [ 27 , 29 , 46 , 57 , 65 , 71 ]. It was often the case that research had a more particular focus. Among others, minimum wages [ 79 ], hospital payment schemes [ 23 ], cigarette prices [ 63 ], social expenditure [ 20 ], residents’ dissatisfaction [ 56 ], income inequality [ 30 , 69 ], and work leave [ 41 , 58 ] took center stage. Whenever variables outside of these specific areas were also included, they were usually identified as confounders or controls, moderators or mediators.

We visualized the combinations in which the different determinants have been studied in Fig 2 , which was obtained via multidimensional scaling and a subsequent cluster analysis (details outlined in S2 Appendix ). It depicts the spatial positioning of each determinant relative to all others, based on the number of times the effects of each pair of determinants have been studied simultaneously. When interpreting Fig 2 , one should keep in mind that determinants marked with an asterisk represent, in fact, collectives of variables.

Groups of determinants are marked by asterisks (see S1 Table in S1 Appendix ). Diminishing color intensity reflects a decrease in the total number of “connections” for a given determinant. Noteworthy pairwise “connections” are emphasized via lines (solid-dashed-dotted indicates decreasing frequency). Grey contour lines encircle groups of variables that were identified via cluster analysis. Abbreviations: age = population age distribution, associations = membership in associations, AT-index = atherogenic-thrombogenic index, BR = birth rate, CAPB = Cyclically Adjusted Primary Balance, civilian-labor = civilian labor force, C-section = Cesarean delivery rate, credit-info = depth of credit information, dissatisf = residents’ dissatisfaction, distrib.orient = distributional orientation, EDU = education, eHealth = eHealth index at GP-level, exch.rate = exchange rate, fat = fat consumption, GDP = gross domestic product, GFCF = Gross Fixed Capital Formation/Creation, GH-gas = greenhouse gas, GII = gender inequality index, gov = governance index, gov.revenue = government revenues, HC-coverage = healthcare coverage, HE = health(care) expenditure, HHconsump = household consumption, hosp.beds = hospital beds, hosp.payment = hospital payment scheme, hosp.stay = length of hospital stay, IDI = ICT development index, inc.ineq = income inequality, industry-labor = industrial labor force, infant-sex = infant sex ratio, labor-product = labor production, LBW = low birth weight, leave = work leave, life-satisf = life satisfaction, M-age = maternal age, marginal-tax = marginal tax rate, MDs = physicians, mult.preg = multiple pregnancy, NHS = Nation Health System, NO = nitrous oxide emissions, PM10 = particulate matter (PM10) emissions, pop = population size, pop.density = population density, pre-term = pre-term birth rate, prison = prison population, researchE = research&development expenditure, school.ref = compulsory schooling reform, smoke-free = smoke-free places, SO = sulfur oxide emissions, soc.E = social expenditure, soc.workers = social workers, sugar = sugar consumption, terror = terrorism, union = union density, UR = unemployment rate, urban = urbanization, veg-fr = vegetable-and-fruit consumption, welfare = welfare regime, Wwater = wastewater treatment.

https://doi.org/10.1371/journal.pone.0239031.g002

Distances between determinants in Fig 2 are indicative of determinants’ “connectedness” with each other. While the statistical procedure called for higher dimensionality of the model, for demonstration purposes we show here a two-dimensional solution. This simplification unfortunately comes with a caveat. To use the factor smoking as an example, it would appear it stands at a much greater distance from GDP than it does from alcohol. In reality however, smoking was considered together with alcohol consumption [ 21 , 25 , 26 , 52 , 68 ] in just as many studies as it was with GDP [ 21 , 25 , 26 , 52 , 59 ], five. To aid with respect to this apparent shortcoming, we have emphasized the strongest pairwise links. Solid lines connect GDP with health expenditure (HE), unemployment rate (UR), and education (EDU), indicating that the effect of GDP on health, taking into account the effects of the other three determinants as well, was evaluated in between 12 to 16 studies of the 60 included in this review. Tracing the dashed lines, we can also tell that GDP appeared jointly with income inequality, and HE together with either EDU or UR, in anywhere between 8 to 10 of our selected studies. Finally, some weaker but still worth-mentioning “connections” between variables are displayed as well via the dotted lines.

The fact that all notable pairwise “connections” are concentrated within a relatively small region of the plot may be interpreted as low overall “connectedness” among the health indicators studied. GDP is the most widely investigated determinant in relation to general population health. Its total number of “connections” is disproportionately high (159) compared to its runner-up–HE (with 113 “connections”), and then subsequently EDU (with 90) and UR (with 86). In fact, all of these determinants could be thought of as outliers, given that none of the remaining factors have a total count of pairings above 52. This decrease in individual determinants’ overall “connectedness” can be tracked on the graph via the change of color intensity as we move outwards from the symbolic center of GDP and its closest “co-determinants”, to finally reach the other extreme of the ten indicators (welfare regime, household consumption, compulsory school reform, life satisfaction, government revenues, literacy, research expenditure, multiple pregnancy, Cyclically Adjusted Primary Balance, and residents’ dissatisfaction; in white) the effects on health of which were only studied in isolation.

Lastly, we point to the few small but stable clusters of covariates encircled by the grey bubbles on Fig 2 . These groups of determinants were identified as “close” by both statistical procedures used for the production of the graph (see details in S2 Appendix ).

Statistical methodology

There was great variation in the level of statistical detail reported. Some authors provided too vague a description of their analytical approach, necessitating some inference in this section.

The issue of missing data is a challenging reality in this field of research, but few of the studies under review (12/60) explain how they dealt with it. Among the ones that do, three general approaches to handling missingness can be identified, listed in increasing level of sophistication: case-wise deletion, i.e., removal of countries from the sample [ 20 , 45 , 48 , 58 , 59 ], (linear) interpolation [ 28 , 30 , 34 , 58 , 59 , 63 ], and multiple imputation [ 26 , 41 , 52 ].

Correlations, Pearson, Spearman, or unspecified, were the only technique applied with respect to the health outcomes of interest in eight analyses [ 33 , 42 – 44 , 46 , 53 , 57 , 61 ]. Among the more advanced statistical methods, the family of regression models proved to be, by and large, predominant. Before examining this closer, we note the techniques that were, in a way, “unique” within this selection of studies: meta-analyses were performed (random and fixed effects, respectively) on the reduced form and 2-sample two stage least squares (2SLS) estimations done within countries [ 39 ]; difference-in-difference (DiD) analysis was applied in one case [ 23 ]; dynamic time-series methods, among which co-integration, impulse-response function (IRF), and panel vector autoregressive (VAR) modeling, were utilized in one study [ 80 ]; longitudinal generalized estimating equation (GEE) models were developed on two occasions [ 70 , 78 ]; hierarchical Bayesian spatial models [ 51 ] and special autoregressive regression [ 62 ] were also implemented.

Purely cross-sectional data analyses were performed in eight studies [ 25 , 45 , 47 , 50 , 55 , 56 , 67 , 71 ]. These consisted of linear regression (assumed ordinary least squares (OLS)), generalized least squares (GLS) regression, and multilevel analyses. However, six other studies that used longitudinal data in fact had a cross-sectional design, through which they applied regression at multiple time-points separately [ 27 , 29 , 36 , 48 , 68 , 72 ].

Apart from these “multi-point cross-sectional studies”, some other simplistic approaches to longitudinal data analysis were found, involving calculating and regressing 3-year averages of both the response and the predictor variables [ 54 ], taking the average of a few data-points (i.e., survey waves) [ 56 ] or using difference scores over 10-year [ 19 , 29 ] or unspecified time intervals [ 40 , 55 ].

Moving further in the direction of more sensible longitudinal data usage, we turn to the methods widely known among (health) economists as “panel data analysis” or “panel regression”. Most often seen were models with fixed effects for country/region and sometimes also time-point (occasionally including a country-specific trend as well), with robust standard errors for the parameter estimates to take into account correlations among clustered observations [ 20 , 21 , 24 , 28 , 30 , 32 , 34 , 37 , 38 , 41 , 52 , 59 , 60 , 63 , 66 , 69 , 73 , 79 , 81 , 82 ]. The Hausman test [ 83 ] was sometimes mentioned as the tool used to decide between fixed and random effects [ 26 , 49 , 63 , 66 , 73 , 82 ]. A few studies considered the latter more appropriate for their particular analyses, with some further specifying that (feasible) GLS estimation was employed [ 26 , 34 , 49 , 58 , 60 , 73 ]. Apart from these two types of models, the first differences method was encountered once as well [ 31 ]. Across all, the error terms were sometimes assumed to come from a first-order autoregressive process (AR(1)), i.e., they were allowed to be serially correlated [ 20 , 30 , 38 , 58 – 60 , 73 ], and lags of (typically) predictor variables were included in the model specification, too [ 20 , 21 , 37 , 38 , 48 , 69 , 81 ]. Lastly, a somewhat different approach to longitudinal data analysis was undertaken in four studies [ 22 , 35 , 48 , 65 ] in which multilevel–linear or Poisson–models were developed.

Regardless of the exact techniques used, most studies included in this review presented multiple model applications within their main analysis. None attempted to formally compare models in order to identify the “best”, even if goodness-of-fit statistics were occasionally reported. As indicated above, many studies investigated women’s and men’s health separately [ 19 , 21 , 22 , 27 – 29 , 31 , 33 , 35 , 36 , 38 , 39 , 45 , 50 , 51 , 64 , 65 , 69 , 82 ], and covariates were often tested one at a time, including other covariates only incrementally [ 20 , 25 , 28 , 36 , 40 , 50 , 55 , 67 , 73 ]. Furthermore, there were a few instances where analyses within countries were performed as well [ 32 , 39 , 51 ] or where the full time period of interest was divided into a few sub-periods [ 24 , 26 , 28 , 31 ]. There were also cases where different statistical techniques were applied in parallel [ 29 , 55 , 60 , 66 , 69 , 73 , 82 ], sometimes as a form of sensitivity analysis [ 24 , 26 , 30 , 58 , 73 ]. However, the most common approach to sensitivity analysis was to re-run models with somewhat different samples [ 39 , 50 , 59 , 67 , 69 , 80 , 82 ]. Other strategies included different categorization of variables or adding (more/other) controls [ 21 , 23 , 25 , 28 , 37 , 50 , 63 , 69 ], using an alternative main covariate measure [ 59 , 82 ], including lags for predictors or outcomes [ 28 , 30 , 58 , 63 , 65 , 79 ], using weights [ 24 , 67 ] or alternative data sources [ 37 , 69 ], or using non-imputed data [ 41 ].

As the methods and not the findings are the main focus of the current review, and because generic checklists cannot discern the underlying quality in this application field (see also below), we opted to pool all reported findings together, regardless of individual study characteristics or particular outcome(s) used, and speak generally of positive and negative effects on health. For this summary we have adopted the 0.05-significance level and only considered results from multivariate analyses. Strictly birth-related factors are omitted since these potentially only relate to the group of infant mortality indicators and not to any of the other general population health measures.

Starting with the determinants most often studied, higher GDP levels [ 21 , 26 , 27 , 29 , 30 , 32 , 43 , 48 , 52 , 58 , 60 , 66 , 67 , 73 , 79 , 81 , 82 ], higher health [ 21 , 37 , 47 , 49 , 52 , 58 , 59 , 68 , 72 , 82 ] and social [ 20 , 21 , 26 , 38 , 79 ] expenditures, higher education [ 26 , 39 , 52 , 62 , 72 , 73 ], lower unemployment [ 60 , 61 , 66 ], and lower income inequality [ 30 , 42 , 53 , 55 , 73 ] were found to be significantly associated with better population health on a number of occasions. In addition to that, there was also some evidence that democracy [ 36 ] and freedom [ 50 ], higher work compensation [ 43 , 79 ], distributional orientation [ 54 ], cigarette prices [ 63 ], gross national income [ 22 , 72 ], labor productivity [ 26 ], exchange rates [ 32 ], marginal tax rates [ 79 ], vaccination rates [ 52 ], total fertility [ 59 , 66 ], fruit and vegetable [ 68 ], fat [ 52 ] and sugar consumption [ 52 ], as well as bigger depth of credit information [ 22 ] and percentage of civilian labor force [ 79 ], longer work leaves [ 41 , 58 ], more physicians [ 37 , 52 , 72 ], nurses [ 72 ], and hospital beds [ 79 , 82 ], and also membership in associations, perceived corruption and societal trust [ 48 ] were beneficial to health. Higher nitrous oxide (NO) levels [ 52 ], longer average hospital stay [ 48 ], deprivation [ 51 ], dissatisfaction with healthcare and the social environment [ 56 ], corruption [ 40 , 50 ], smoking [ 19 , 26 , 52 , 68 ], alcohol consumption [ 26 , 52 , 68 ] and illegal drug use [ 68 ], poverty [ 64 ], higher percentage of industrial workers [ 26 ], Gross Fixed Capital creation [ 66 ] and older population [ 38 , 66 , 79 ], gender inequality [ 22 ], and fertility [ 26 , 66 ] were detrimental.

It is important to point out that the above-mentioned effects could not be considered stable either across or within studies. Very often, statistical significance of a given covariate fluctuated between the different model specifications tried out within the same study [ 20 , 49 , 59 , 66 , 68 , 69 , 73 , 80 , 82 ], testifying to the importance of control variables and multivariate research (i.e., analyzing multiple independent variables simultaneously) in general. Furthermore, conflicting results were observed even with regards to the “core” determinants given special attention, so to speak, throughout this text. Thus, some studies reported negative effects of health expenditure [ 32 , 82 ], social expenditure [ 58 ], GDP [ 49 , 66 ], and education [ 82 ], and positive effects of income inequality [ 82 ] and unemployment [ 24 , 31 , 32 , 52 , 66 , 68 ]. Interestingly, one study [ 34 ] differentiated between temporary and long-term effects of GDP and unemployment, alluding to possibly much greater complexity of the association with health. It is also worth noting that some gender differences were found, with determinants being more influential for males than for females, or only having statistically significant effects for male health [ 19 , 21 , 28 , 34 , 36 , 37 , 39 , 64 , 65 , 69 ].

The purpose of this scoping review was to examine recent quantitative work on the topic of multi-country analyses of determinants of population health in high-income countries.

Measuring population health via relatively simple mortality-based indicators still seems to be the state of the art. What is more, these indicators are routinely considered one at a time, instead of, for example, employing existing statistical procedures to devise a more general, composite, index of population health, or using some of the established indices, such as disability-adjusted life expectancy (DALE) or quality-adjusted life expectancy (QALE). Although strong arguments for their wider use were already voiced decades ago [ 84 ], such summary measures surface only rarely in this research field.

On a related note, the greater data availability and accessibility that we enjoy today does not automatically equate to data quality. Nonetheless, this is routinely assumed in aggregate level studies. We almost never encountered a discussion on the topic. The non-mundane issue of data missingness, too, goes largely underappreciated. With all recent methodological advancements in this area [ 85 – 88 ], there is no excuse for ignorance; and still, too few of the reviewed studies tackled the matter in any adequate fashion.

Much optimism can be gained considering the abundance of different determinants that have attracted researchers’ attention in relation to population health. We took on a visual approach with regards to these determinants and presented a graph that links spatial distances between determinants with frequencies of being studies together. To facilitate interpretation, we grouped some variables, which resulted in some loss of finer detail. Nevertheless, the graph is helpful in exemplifying how many effects continue to be studied in a very limited context, if any. Since in reality no factor acts in isolation, this oversimplification practice threatens to render the whole exercise meaningless from the outset. The importance of multivariate analysis cannot be stressed enough. While there is no “best method” to be recommended and appropriate techniques vary according to the specifics of the research question and the characteristics of the data at hand [ 89 – 93 ], in the future, in addition to abandoning simplistic univariate approaches, we hope to see a shift from the currently dominating fixed effects to the more flexible random/mixed effects models [ 94 ], as well as wider application of more sophisticated methods, such as principle component regression, partial least squares, covariance structure models (e.g., structural equations), canonical correlations, time-series, and generalized estimating equations.

Finally, there are some limitations of the current scoping review. We searched the two main databases for published research in medical and non-medical sciences (PubMed and Web of Science) since 2013, thus potentially excluding publications and reports that are not indexed in these databases, as well as older indexed publications. These choices were guided by our interest in the most recent (i.e., the current state-of-the-art) and arguably the highest-quality research (i.e., peer-reviewed articles, primarily in indexed non-predatory journals). Furthermore, despite holding a critical stance with regards to some aspects of how determinants-of-health research is currently conducted, we opted out of formally assessing the quality of the individual studies included. The reason for that is two-fold. On the one hand, we are unaware of the existence of a formal and standard tool for quality assessment of ecological designs. And on the other, we consider trying to score the quality of these diverse studies (in terms of regional setting, specific topic, outcome indices, and methodology) undesirable and misleading, particularly since we would sometimes have been rating the quality of only a (small) part of the original studies—the part that was relevant to our review’s goal.

Our aim was to investigate the current state of research on the very broad and general topic of population health, specifically, the way it has been examined in a multi-country context. We learned that data treatment and analytical approach were, in the majority of these recent studies, ill-equipped or insufficiently transparent to provide clarity regarding the underlying mechanisms of population health in high-income countries. Whether due to methodological shortcomings or the inherent complexity of the topic, research so far fails to provide any definitive answers. It is our sincere belief that with the application of more advanced analytical techniques this continuous quest could come to fruition sooner.

Supporting information

S1 checklist. preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (prisma-scr) checklist..

https://doi.org/10.1371/journal.pone.0239031.s001

S1 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s002

S2 Appendix.

https://doi.org/10.1371/journal.pone.0239031.s003

• View Article
• Google Scholar
• PubMed/NCBI
• 75. Dahlgren G, Whitehead M. Policies and Strategies to Promote Equity in Health. Stockholm, Sweden: Institute for Future Studies; 1991.
• 76. Brunner E, Marmot M. Social Organization, Stress, and Health. In: Marmot M, Wilkinson RG, editors. Social Determinants of Health. Oxford, England: Oxford University Press; 1999.
• 77. Najman JM. A General Model of the Social Origins of Health and Well-being. In: Eckersley R, Dixon J, Douglas B, editors. The Social Origins of Health and Well-being. Cambridge, England: Cambridge University Press; 2001.
• 85. Carpenter JR, Kenward MG. Multiple Imputation and its Application. New York: John Wiley & Sons; 2013.
• 86. Molenberghs G, Fitzmaurice G, Kenward MG, Verbeke G, Tsiatis AA. Handbook of Missing Data Methodology. Boca Raton: Chapman & Hall/CRC; 2014.
• 87. van Buuren S. Flexible Imputation of Missing Data. 2nd ed. Boca Raton: Chapman & Hall/CRC; 2018.
• 88. Enders CK. Applied Missing Data Analysis. New York: Guilford; 2010.
• 89. Shayle R. Searle GC, Charles E. McCulloch. Variance Components: John Wiley & Sons, Inc.; 1992.
• 90. Agresti A. Foundations of Linear and Generalized Linear Models. Hoboken, New Jersey: John Wiley & Sons Inc.; 2015.
• 91. Leyland A. H. (Editor) HGE. Multilevel Modelling of Health Statistics: John Wiley & Sons Inc; 2001.
• 92. Garrett Fitzmaurice MD, Geert Verbeke, Geert Molenberghs. Longitudinal Data Analysis. New York: Chapman and Hall/CRC; 2008.
• 93. Wolfgang Karl Härdle LS. Applied Multivariate Statistical Analysis. Berlin, Heidelberg: Springer; 2015.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Quantitative Research? | Definition, Uses & Methods

What Is Quantitative Research? | Definition, Uses & Methods

Published on June 12, 2020 by Pritha Bhandari . Revised on June 22, 2023.

Quantitative research is the process of collecting and analyzing numerical data. It can be used to find patterns and averages, make predictions, test causal relationships, and generalize results to wider populations.

Quantitative research is the opposite of qualitative research , which involves collecting and analyzing non-numerical data (e.g., text, video, or audio).

Quantitative research is widely used in the natural and social sciences: biology, chemistry, psychology, economics, sociology, marketing, etc.

• What is the demographic makeup of Singapore in 2020?
• How has the average temperature changed globally over the last century?
• Does environmental pollution affect the prevalence of honey bees?
• Does working from home increase productivity for people with long commutes?

Table of contents

Quantitative research methods, quantitative data analysis, advantages of quantitative research, disadvantages of quantitative research, other interesting articles, frequently asked questions about quantitative research.

You can use quantitative research methods for descriptive, correlational or experimental research.

• In descriptive research , you simply seek an overall summary of your study variables.
• In correlational research , you investigate relationships between your study variables.
• In experimental research , you systematically examine whether there is a cause-and-effect relationship between variables.

Correlational and experimental research can both be used to formally test hypotheses , or predictions, using statistics. The results may be generalized to broader populations based on the sampling method used.

To collect quantitative data, you will often need to use operational definitions that translate abstract concepts (e.g., mood) into observable and quantifiable measures (e.g., self-ratings of feelings and energy levels).

Note that quantitative research is at risk for certain research biases , including information bias , omitted variable bias , sampling bias , or selection bias . Be sure that you’re aware of potential biases as you collect and analyze your data to prevent them from impacting your work too much.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Once data is collected, you may need to process it before it can be analyzed. For example, survey and test data may need to be transformed from words to numbers. Then, you can use statistical analysis to answer your research questions .

Descriptive statistics will give you a summary of your data and include measures of averages and variability. You can also use graphs, scatter plots and frequency tables to visualize your data and check for any trends or outliers.

Using inferential statistics , you can make predictions or generalizations based on your data. You can test your hypothesis or use your sample data to estimate the population parameter .

First, you use descriptive statistics to get a summary of the data. You find the mean (average) and the mode (most frequent rating) of procrastination of the two groups, and plot the data to see if there are any outliers.

You can also assess the reliability and validity of your data collection methods to indicate how consistently and accurately your methods actually measured what you wanted them to.

Quantitative research is often used to standardize data collection and generalize findings . Strengths of this approach include:

• Replication

Repeating the study is possible because of standardized data collection protocols and tangible definitions of abstract concepts.

• Direct comparisons of results

The study can be reproduced in other cultural settings, times or with different groups of participants. Results can be compared statistically.

• Large samples

Data from large samples can be processed and analyzed using reliable and consistent procedures through quantitative data analysis.

• Hypothesis testing

Using formalized and established hypothesis testing procedures means that you have to carefully consider and report your research variables, predictions, data collection and testing methods before coming to a conclusion.

Despite the benefits of quantitative research, it is sometimes inadequate in explaining complex research topics. Its limitations include:

• Superficiality

Using precise and restrictive operational definitions may inadequately represent complex concepts. For example, the concept of mood may be represented with just a number in quantitative research, but explained with elaboration in qualitative research.

• Narrow focus

Predetermined variables and measurement procedures can mean that you ignore other relevant observations.

• Structural bias

Despite standardized procedures, structural biases can still affect quantitative research. Missing data , imprecise measurements or inappropriate sampling methods are biases that can lead to the wrong conclusions.

• Lack of context

Quantitative research often uses unnatural settings like laboratories or fails to consider historical and cultural contexts that may affect data collection and results.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Operationalization means turning abstract conceptual ideas into measurable observations.

For example, the concept of social anxiety isn’t directly observable, but it can be operationally defined in terms of self-rating scores, behavioral avoidance of crowded places, or physical anxiety symptoms in social situations.

Before collecting data , it’s important to consider how you will operationalize the variables that you want to measure.

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics. It is used by scientists to test specific predictions, called hypotheses , by calculating how likely it is that a pattern or relationship between variables could have arisen by chance.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). What Is Quantitative Research? | Definition, Uses & Methods. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/methodology/quantitative-research/

Is this article helpful?

Pritha Bhandari

Other students also liked, descriptive statistics | definitions, types, examples, inferential statistics | an easy introduction & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Archer Library

Quantitative research: literature review .

• Archer Library This link opens in a new window
• Research Resources handout This link opens in a new window
• Locating Books
• Library eBook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Statistics
• Literature Review Resources
• Citations & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

Archer Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: Scholarly & Practitioner Journals

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

The Chronicle of Higher Education has the nation’s largest newsroom dedicated to covering colleges and universities.  Source of news, information, and jobs for college and university faculty members and administrators

The Chronicle features complete contents of the latest print issue; daily news and advice columns; current job listings; archive of previously published content; discussion forums; and career-building tools such as online CV management and salary databases. Dates covered: 1970-present.

Offers in-depth coverage of national and international business and finance as well as first-rate coverage of hard news--all from America's premier financial newspaper. Covers complete bibliographic information and also subjects, companies, people, products, and geographic areas. 

Comprehensive coverage back to 1984 is available from the world's leading financial newspaper through the ProQuest database. 

Newspaper Source provides cover-to-cover full text for hundreds of national (U.S.), international and regional newspapers. In addition, it offers television and radio news transcripts from major networks.

Provides complete television and radio news transcripts from CBS News, CNN, CNN International, FOX News, and more.

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Basic Searching with EBSCO
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Statistics
• Next: Literature Review Resources >>
• Last Updated: Aug 29, 2024 11:19 AM
• URL: https://libguides.ashland.edu/quantitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

Emotion Regulation and Academic Burnout Among Youth: a Quantitative Meta-analysis

• META-ANALYSIS
• Open access
• Published: 10 September 2024
• Volume 36 , article number  106 , ( 2024 )

Cite this article

You have full access to this open access article

• Ioana Alexandra Iuga   ORCID: orcid.org/0000-0001-9152-2004 1 , 2 &
• Oana Alexandra David   ORCID: orcid.org/0000-0001-8706-1778 2 , 3  

Emotion regulation (ER) represents an important factor in youth’s academic wellbeing even in contexts that are not characterized by outstanding levels of academic stress. Effective ER not only enhances learning and, consequentially, improves youths’ academic achievement, but can also serve as a protective factor against academic burnout. The relationship between ER and academic burnout is complex and varies across studies. This meta-analysis examines the connection between ER strategies and student burnout, considering a series of influencing factors. Data analysis involved a random effects meta-analytic approach, assessing heterogeneity and employing multiple methods to address publication bias, along with meta-regression for continuous moderating variables (quality, female percentage and mean age) and subgroup analyses for categorical moderating variables (sample grade level). According to our findings, adaptive ER strategies are negatively associated with overall burnout scores, whereas ER difficulties are positively associated with burnout and its dimensions, comprising emotional exhaustion, cynicism, and lack of efficacy. These results suggest the nuanced role of ER in psychopathology and well-being. We also identified moderating factors such as mean age, grade level and gender composition of the sample in shaping these associations. This study highlights the need for the expansion of the body of literature concerning ER and academic burnout, that would allow for particularized analyses, along with context-specific ER research and consistent measurement approaches in understanding academic burnout. Despite methodological limitations, our findings contribute to a deeper understanding of ER's intricate relationship with student burnout, guiding future research in this field.

Avoid common mistakes on your manuscript.

Introduction

The transitional stages of late adolescence and early adulthood are characterized by significant physiological and psychological changes, including increased stress (Matud et al., 2020 ). Academic stress among students has long been studied in various samples, most of them focusing on university students (Bedewy & Gabriel, 2015 ; Córdova Olivera et al., 2023 ; Hystad et al., 2009 ) and, more recently, high school (Deb et al., 2015 ) and middle school students (Luo et al., 2020 ). Further, studies report an exacerbation of academic stress and mental health difficulties in response to the COVID-19 pandemic (Guessoum et al., 2020 ), with children facing additional challenges that affect their academic well-being, such as increasing workloads, influences from the family, and the issue of decreasing financial income (Ibda et al., 2023 ; Yang et al., 2021 ). For youth to maintain their well-being in stressful academic settings, emotion regulation (ER) has been identified as an important factor (Santos Alves Peixoto et al., 2022 ; Yildiz, 2017 ; Zahniser & Conley, 2018 ).

Emotion regulation, referring to”the process by which individuals influence which emotions they have, when they have them, and how they experience and express their emotions” (Gross, 1998b ), represents an important factor in youth’s academic well-being even in contexts that are not characterized by outstanding levels of stress. Emotion regulation strategies promote more efficient learning and, consequentially, improve youth’s academic achievement and motivation (Asareh et al., 2022 ; Davis & Levine, 2013 ), discourage academic procrastination (Mohammadi Bytamar et al., 2020 ), and decrease the chances of developing emotional problems such as burnout (Narimanj et al., 2021 ) and anxiety (Shahidi et al., 2017 ).

Approaches to Emotion Regulation

Numerous theories have been proposed to elucidate the process underlying the emergence and progression of emotional regulation (Gross, 1998a , 1998b ; Koole, 2009 ; Larsen, 2000 ; Parkinson & Totterdell, 1999 ). One prominent approach, developed by Gross ( 2015 ), refers to the process model of emotion regulation, which lays out the sequential actions people take to regulate their emotions during the emotion-generative process. These steps involve situation selection, situation modification, attentional deployment, cognitive change, and response modulation. The kind and timing of the emotion regulation strategies people use, according to this paradigm, influence the specific emotions people experience and express.

Recent theories of emotion regulation propose two separate, yet interconnected approaches: ER abilities and ER strategies. ER abilities are considered a higher-order process that guides the type of ER strategy an individual uses in the context of an emotion-generative circumstance. Further, ER strategies are considered factors that can also influence ER abilities, forming a bidirectional relationship (Tull & Aldao, 2015 ). Researchers use many definitions and classifications of emotion regulation, however, upon closer inspection, it becomes clear that there are notable similarities across these concepts. While there are many models of emotion regulation, it's important to keep from seeing them as competing or incompatible since each one represents a unique and important aspect of the multifaceted concept of emotion regulation.

Emotion Regulation and Emotional Problems

The connection between ER strategies and psychopathology is intricate and multifaceted. While some researchers propose that ER’s effectiveness is context-dependent (Kobylińska & Kusev, 2019 ; Troy et al., 2013 ), several ER strategies have long been attested as adaptive or maladaptive. This body of work suggests that certain emotion regulation strategies (such as avoidance and expressive suppression) demonstrate, based on findings from experimental studies, inefficacy in altering affect and appear to be linked to higher levels of psychological symptoms. These strategies have been categorized as ER difficulties. In contrast, alternative emotion regulation strategies (such as reappraisal and acceptance) have demonstrated effectiveness in modifying affect within controlled laboratory environments, exhibiting a negative association with clinical symptoms. As a result, these strategies have been characterized as potentially adaptive (Aldao & Nolen-Hoeksema, 2012a , 2012b ; Aldao et al., 2010 ; Gross, 2013 ; Webb et al., 2012 ).

A long line of research highlights the divergent impact of putatively maladaptive and adaptive ER strategies on psychopathology and overall well-being (Gross & Levenson, 1993 ; Gross, 1998a ). Increased negative affect, increased physiological reactivity, memory problems (Richards et al., 2003 ), a decline in functional behavior (Dixon-Gordon et al., 2011 ), and a decline in social support (Séguin & MacDonald, 2018 ) are just a few of the negative effects that have consistently been linked to emotional regulation difficulties, which include but are not limited to the use of avoidance, suppression, rumination, and self-blame strategies. Additionally, a wide range of mental problems, such as depression (Nolen-Hoeksema et al., 2008 ), anxiety disorders (Campbell-Sills et al., 2006a , 2006b ; Mennin et al., 2007 ), eating disorders (Prefit et al., 2019 ), and borderline personality disorder (Lynch et al., 2007 ; Neacsiu et al., 2010 ) are connected to self-reports of using these strategies.

Conversely, putatively adaptive strategies, including acceptance, problem-solving, and cognitive reappraisal, have consistently yielded beneficial outcomes in experimental studies. These outcomes encompass reductions in negative emotional responses, enhancements in interpersonal relationships, increased pain tolerance, reductions in physiological reactivity, and lower levels of psychopathological symptoms (Aldao et al., 2010 ; Goldin et al., 2008 ; Hayes et al., 1999 ; Richards & Gross, 2000 ).

Notably, despite the fact that therapeutic techniquest for enhancing the use of adaptive ER strategies are core elements of many therapeutic approaches, from traditional Cognitive Behavioral Therapy (CBT) to more recent third-wave interventions (Beck, 1976 ; Hofmann & Asmundson, 2008 ; Linehan, 1993 ; Roemer et al., 2008 ; Segal et al., 2002 ), the association between ER difficulties and psychopathology frequently show a stronger positive correlation compared to the inverse negative association with adaptive ER strategies, as highlighted by Aldao and Nolen-Hoeksema ( 2012a ).

Pines & Aronson ( 1988 ) characterize burnout that arises in the workplace context as a state wherein individuals encounter emotional challenges, such as experiencing fatigue and physical exhaustion due to heightened task demands. Recently, driven by the rationale that schools are the environments where students engage in significant work, the concept of burnout has been extended to educational contexts (Salmela-Aro, 2017 ; Salmela-Aro & Tynkkynen, 2012 ; Walburg, 2014 ). Academic burnout is defined as a syndrome comprising three dimensions: exhaustion stemming from school demands, a cynical and detached attitude toward one's academic environment, and feelings of inadequacy as a student (Salmela-Aro et al., 2004 ; Schaufeli et al., 2002 ).

School burnout has quickly garnered international attention, despite its relatively recent emergence, underscoring its relevance across multiple nations (Herrmann et al., 2019 ; May et al., 2015 ; Meylan et al., 2015 ; Yang & Chen, 2016 ). Similar to other emotional difficulties, it has been observed among students from various educational systems and academic policies, suggesting that this phenomenon transcends cultural and geographical boundaries (Walburg, 2014 ).

The link between ER and school burnout can be understood through Gross's ( 1998a ) process model of emotion regulation. This model suggests that an individual's emotional responses are influenced by their ER strategies, which are adaptive or maladaptive reactions to stressors like academic pressure. Given that academic stress greatly influences school burnout (Jiang et al., 2021 ; Nikdel et al., 2019 ), the ER strategies students use to manage this stress may impact their likelihood of experiencing burnout. In essence, whether a student employs efficient ER strategies or encounters ER difficulties could influence their susceptibility to school burnout.

The exploration of ER in relation to student burnout has garnered attention through various studies. However, the existing body of research is not yet robust enough, and its outcomes are not universally congruent. Suppression, defined as efforts to inhibit ongoing emotional expression (Balzarotti et al., 2010 ), has demonstrated a positive and significant correlation with both general and specific burnout dimensions (Chacón-Cuberos et al., 2019 ; Seibert et al., 2017 ), with the exception of the study conducted by Yu et al., ( 2022 ), where there is a negative, but not significant association between suppression and reduced accomplishment. Notably, research by Muchacka-Cymerman and Tomaszek ( 2018 ) indicates that ER strategies, encompassing both dispositional and situational approaches, exhibit a negative relationship with overall burnout. Situational ER, however, displays a negative impact on dimensions like inadequacy and declining interest, particularly concerning the school environment.

Cognitive ER strategies such as reappraisal, positive refocusing, and planning are, generally, negatively associated with burnout, while self-blame, other-blame, rumination, and catastrophizing present a positive association with burnout (Dominguez-Lara, 2018 ; Vinter et al., 2021 ). It's important to note that these relationships have not been consistently replicated across all investigations. Inconsistencies in the findings highlight the complexity of the interactions and the potential influence of various contextual factors. Consequently, there remains a critical need for further research to thoroughly examine these associations and identify the factors contributing to the variability in results.

Existing Research

Although we were unable to identify any reviews or meta-analyses that synthesize the literature concerning emotion regulation strategies and student burnout, recent meta-analyses have identified the role of emotion regulation across pathologies. A recent network meta-analysis identified the role of rumination and non-acceptance of emotions to be closely related to eating disorders (Leppanen et al., 2022 ). Further, compared to healthy controls, people presenting bipolar disorder symptoms reported significantly higher difficulties in emotion regulation (Miola et al., 2022 ). Weiss et al. ( 2022 ) identified a small to medium association between emotion regulation and substance use, and a subsequent meta-analysis conducted by Stellern et al. ( 2023 ) confirmed that individuals with substance use disorders have significantly higher emotion regulation difficulties compared to controls. The study of Dawel et al. ( 2021 ) represents the many research papers asking the question”Cause or symptom” in the context of emotion regulation. The longitudinal study brings forward the bidirectional relationship between ER and depression and anxiety, particularly in the case of suppression, suggesting that suppressing emotions is indicative of and can predict psychological distress.

Despite the increasing research attention to academic burnout in recent years, the current body of literature primarily concentrates on specific groups such as medical students (Almutairi et al., 2022 ; Frajerman et al., 2019 ), educators (Aloe et al., 2014 ; Park & Shin, 2020 ), and students at the secondary and tertiary education levels (Madigan & Curran, 2021 ) in the context of meta-analyses and reviews. A limited number of recent reviews have expanded their focus to include a more diverse range of participants, encompassing middle school, graduate, and university students (Kim et al., 2018 , 2021 ), with a particular emphasis on investigating social support and exploring the demand-control-support model in relation to student burnout.

The significance of managing burnout in educational settings is becoming more widely acknowledged, as seen by the rise in interventions designed to reduce the symptoms of burnout in students. Specific interventions for alleviating burnout symptoms among students continue to proliferate (Madigan et al., 2023 ), with a focus on stress reduction through mindfulness-based strategies (Lo et al., 2021 ; Modrego-Alarcón et al., 2021 ) and rational-emotive behavioral techniques (Ogbuanya et al., 2019 ) to enhance emotion-regulation skills (Charbonnier et al., 2022 ) and foster rational thinking (Bresó et al., 2011 ; Ezeudu et al., 2020 ). This underscores the significance of emotion regulation in addressing burnout.

Despite several randomized clinical trials addressing student burnout and an emerging body of research relating emotion regulation and academic burnout, there's a lack of a systematic examination of how emotion regulation strategies relate to various dimensions of student burnout. This highlights the necessity for a systematic review of existing evidence. The current meta-analysis addresses the association between emotion regulation strategies and student burnout.

A secondary objective is to test the moderating effect of school level and female percentage in the sample, as well as study quality, in order to identify possible sources of heterogeneity among effect sizes. By analyzing the moderating effect of school level and gender, we may determine if the strength of the association between student burnout and emotion regulation is contingent upon the educational setting and participant characteristics. This offers information on the findings' generalizability to all included student demographics, including those in elementary, middle, and secondary education and of different genders. Additionally, the reliability and validity of meta-analytic results rely on the evaluation of research quality, and the inclusion of study quality rating allows us to determine if the observed association between emotion regulation and student burnout differs based on the methodological rigor of the included studies.

Materials and Methods

Study protocol.

The present meta-analysis has been carried out following the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) statement (Moher et al., 2009 ). The protocol for the meta-analysis was pre-registered in PROSPERO (PROSPERO, 2022 CRD42022325570).

Selection of Studies

A systematic search was performed using relevant databases (PubMed, Web of Science, PsychINFO, and Scopus). The search was carried out on 25 May of 2023 using 25 key terms related to the variables of interest, such as: (a) academic burnout, (b) school burnout, (c) student burnout (d) education burnout, (d) exhaustion, (e) cynicism, (f) inadequacy, (g) emotion regulation, (h) coping, (i) self-blame, (j) acceptance, and (h) problem solving.

Studies of any design published in peer-reviewed journals were eligible for inclusion, provided they used empirical data to assess the relationship between student burnout and emotion regulation strategies. Only studies that employed samples of children, adolescents, and youth were eligible for inclusion. For the purpose of the current paper, we define youth as people aged 18 to 25, based on how it is typically defined in the literature (Westhues & Cohen, 1997 ).

Studies were excluded from the meta-analysis if they: (a) were not a quantitative study, (b) did not explore the relationship between academic burnout and emotion regulation strategies, (c) did not have a sample that can be defined as consisting of children and youth (Scales et al., 2016 ), (e) did not utilize Pearson’s correlation or measures that could be converted to a Pearson’s correlation, (f) include medical school or associated disciplines samples.

Statistical Analysis

For the data analysis, we employed Comprehensive Meta-Analysis 4 software. Anticipating significant heterogeneity in the included studies, we opted for a random effects meta-analytic approach instead of a fixed-effects model, a choice that acknowledges and accounts for potential variations in effect sizes across studies, contributing to a more robust and generalizable synthesis of the results. Heterogeneity among the studies was assessed using the I 2 and Q statistics, adhering to the interpretation thresholds outlined in the Cochrane Handbook (Deeks et al., 2023 ).

Publication bias was assessed through a multi-faceted approach. We first examined the funnel plot for the primary outcome measures, a graphical representation revealing potential asymmetry that might indicate publication bias. Furthermore, we utilized Duval and Tweedie's trim and fill procedure (Duval & Tweedie, 2000 ), as implemented in CMA, to estimate the effect size after accounting for potential publication bias. Additionally, Egger's test of the intercept was conducted to quantify the bias detected by the funnel plot and to determine its statistical significance.

When dealing with continuous moderating variables, we employed meta-regression to evaluate the significance of their effects. For categorical moderating variables, we conducted subgroup analyses to test for significance. To ensure the validity of these analyses, it was essential that there was a minimum of three effect sizes within each subgroup under the same moderating variable, following the guidelines outlined by Junyan and Minqiang ( 2020 ). In accordance with the guidance provided in the Cochrane Handbook (Schmid et al., 2020 ), our application of meta-regression analyses was limited to cases where a minimum of 10 studies were available for each examined covariate. This approach ensures that there is a sufficient number of studies to support meaningful statistical analysis and reliable conclusions when exploring the influence of various covariates on the observed relationships.

Data Extraction and Quality Assessment

In addition to the identification information (i.e., authors, publication year), we extracted data required for the effect size calculation for the variables relevant to burnout and emotion regulation strategies. Where data was unavailable, the authors were contacted via email in order to provide the necessary information. Potential moderator variables were coded in order to examine the sources of variation in study findings. The potential moderators included: (a) participants’ gender, (b), grade level (c) study quality, and (d) mean age.

The full-text articles were independently assessed using the Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields tool (Kmet et al., 2004 ) by a pair of coders (II and SM), to ensure the reliability of the data, resulting in a substantial level of agreement (Cohen’s k  = 0.89). The disagreements and discrepancies between the two coders were resolved through discussion and consensus. If consensus could not be reached, a third researcher (OD) was consulted to resolve the disagreement.

The checklist items focused on evaluating the alignment of the study's design with its stated objectives, the methodology employed, the level of precision in presenting the results, and the accuracy of the drawn conclusions. The assessment criteria were composed of 14 items, which were evaluated using a 3-point Likert scale (with responses of 2 for "yes," 1 for "partly," and 0 for "no"). A cumulative score was computed for each study based on these items. For studies where certain checklist items were not relevant due to their design, those items were marked as "n/a" and were excluded from the cumulative score calculation.

Study Selection

The combined search terms yielded a total of 15,179 results. The duplicate studies were removed using Zotero, and a total of 8,022 studies remained. The initial screening focused on the titles and abstracts of all remaining studies, removing all documents that target irrelevant predictors or outcomes, as well as qualitative studies and reviews. Two assessors (II and SA) independently screened the papers against the inclusion and exclusion criteria. A number of 7,934 records were removed, while the remaining 88 were sought for retrieval. Out of the 88 articles, we were unable to find one, while another has been retracted by the journal. Finally, 86 articles were assessed for eligibility. A total of 20 articles met the inclusion criteria (see Fig.  1 ). Although a specific cutoff criterion for reliability coefficients was not imposed during study selection, the majority of the included studies had alpha Cronbach values for the instruments assessing emotion regulation and school burnout greater than 0.70.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of the study selection process

Data Overview

Among the included studies, four focused on middle school students, two encompassed high school student samples, and the remaining 14 articles involved samples of university students. The majority of the included studies had cross-sectional designs (17), while the rest consisted of 2 longitudinal studies and one non-randomized controlled pilot study. The percentage of females within the samples ranged from 46% to 88.3%, averaging 65%, while the mean age of participants ranged from 10.39 to 25. The investigated emotional regulation strategies within the included studies exhibit variation, encompassing other-blame, self-blame, acceptance, rumination, catastrophizing, putting into perspective, reappraisal, planning, behavioral and mental disengagement, expressive suppression, and others (see Table  1 for a detailed study presentation).

Study Quality

Every study surpasses a quality threshold of 0.60, and 75% of the studies achieve a score above the more conservative threshold indicated by Kmet et al. ( 2004 ). This indicates a minimal risk of bias in these studies. Moreover, 80% of the studies adequately describe their objectives, while the appropriateness of the study design is recognized in 50% of the cases, mostly utilizing cross-sectional designs. While 95% of the studies provide sufficient descriptions of their samples, only 10% employ appropriate sampling methods, with the majority relying on convenience sampling. Notably, there is just one interventional study that lacks random allocation and blinding of investigators or subjects.

In terms of measurement, 85% of the studies employ validated and reliable tools. Adequacy in sample size and well-justified and appropriate analytic methods are observed across all included studies. While approximately 50% of the studies present estimates of variance, a mere 30% of them acknowledge the control of confounding variables. Lastly, 95% of the studies provide results in comprehensive detail, with 60% effectively grounding their discussions in the obtained results. The quality assessment criteria and results can be consulted in Supplementary Material 4 .

Pooled Effects

A sensitivity analysis using standardized residuals was conducted. Provided that the residuals are normally distributed, 95% of them would fall within the range of -2 to 2. Residuals outside this range were considered unusual. We applied this cutoff in our meta-analysis to identify any outliers. The results of the analysis revealed that several relationships had standardized residuals falling outside the specified range. Re-analysis excluding these outliers demonstrated that our initial results were robust and did not significantly change in magnitude or significance. As a result, we have moved on with the analysis for the entire sample.

The calculated overall effects can be consulted in Table  2 . The correlation between ER difficulties and student burnout is a significant one, with significant positive associations between ER difficulties and overall burnout (k = 13), r  = 0.25 (95% CI = 0.182; 0.311), p  < 0.001, as well as individual burnout dimensions: cynicism (k = 9), r  = 0.28 (95% CI = 0.195; 0.353) p  < 0.001, lack of efficacy (k = 8), r  = 0.17 (95% CI = 0.023; 0.303), p  < 0.05 and emotional exhaustion (k = 11), r  = 0.27 (95% CI = 0.207; 0.335) p  < 0.001. Regarding the relationship between adaptive ER strategies and student burnout, a statistically significant result is observed solely between overall student burnout and adaptive ER (k = 17), r  = -14 (95% CI = -0.239; 0.046) p  < 0.005. The forest plots can be consulted in Supplementary Material 1 .

Heterogeneity and Publication Bias

Table 3 shows that all Q tests were significant, indicating that there is significant variation among the effect sizes of the individual studies included in the meta-analysis. Further, all I 2 indices are over 75%, ranging from 83.67% to 99.32%, which also indicates high heterogeneity (Borenstein et al., 2017 ). This consistently high level of heterogeneity indicates substantial variation in effect sizes, pointing to influential factors that significantly shape the outcomes of the included studies. Consequentially, subgroup and meta-regression analyses are to be carried out in order to unravel the underlying factors driving this pronounced heterogeneity. The results of the publication bias analysis are presented individually below and, additionally, you can consult the funnel plots included in Supplementary Material 2 .

Adaptive ER and School Burnout

Upon visual examination of the funnel plot, asymmetry to the right of the mean was observed. To validate this observation, a trim-and-fill analysis using Duval and Tweedie’s method was conducted, revealing the absence of three studies on the left side of the mean. The adjusted effect size ( r  = -0.17, 95% CI [0.27; 0.68]) resulting from this analysis was found to be higher than the initially observed effect size. Nevertheless, the application of Egger’s test did not yield a significant indication of publication bias ( B  = -5.34, 95% CI [-11.85; 1.16], p  = 0.10).

Adaptive ER and Cynicism

Following a visual examination of the funnel plot, a symmetrical arrangement of effect sizes around the mean was apparent. This finding was contradicted by the application of Duval and Tweedie's trim-and-fill method, which revealed two missing studies to the right of the mean. The adjusted effect size ( r  = 0.04, 95% CI [-0.21; 0.13]) is smaller than the initially observed effect size. The application of Egger’s test did not yield a significant indication of publication bias ( B  = -2.187, 95% CI [-8.57; 4.19], p  = 0.43).

ER difficulties and Lack of Efficacy

The visual examination of the funnel plot revealed asymmetry to the right of the mean. This finding was validated by the application of Duval and Tweedie's trim-and-fill method, which revealed two missing studies to the left of the mean and a lower adjusted effect size ( r  = 0.08, 95% CI [-0.07; 0.23]), the effect becoming statistically non-significant. The application of Egger’s test did not yield a significant indication of publication bias ( B  = 7.76, 95% CI [-16.53; 32.05], p  = 0.46).

Adaptive ER and Emotional Exhaustion

The visual examination of the funnel plot revealed asymmetry to the left of the mean. The trim-and-fill method also revealed one missing study to the right of the mean and a lower adjusted effect size ( r  = 0.00, 95% CI [-0.13; 0.12]). The application of Egger’s test did not yield a significant indication of publication bias ( B  = 7.02, 95% CI [-23.05; 9.02], p  = 0.46).

Adaptive ER and Lack of Efficacy; ER difficulties and School Burnout, Cynicism, and Exhaustion

Upon visually assessing the funnel plot, a balanced distribution of effect sizes centered around the mean was observed. This observation is corroborated by the application of Duval and Tweedie's trim-and-fill method, which also revealed no indication of missing studies. The adjusted effect size remained consistent, and the intercept signifying publication bias was found to be statistically insignificant.

Moderator Analysis

We performed moderator analyses for the categorical variables, in the case of significant relationships that were uncovered in the initial analysis. These analyses were carried out specifically for cases where there were more than three effect sizes available within each subgroup that fell under the same moderating variable.

Students’ grade level was used as a categorical moderator. Pre-university students included students enrolled in primary and secondary education, while the university student category included tertiary education students. The results, presented in Table  4 , show that the moderating effect of grade level is not significant for the relationship between adaptive ER and overall school burnout Q (1) = 0.20, p  = 0.66. At a specific level, the moderating effect is significant for the relationship between ER difficulties and overall burnout Q (1) = 9.81, p  = 0.002, cynicism Q (1) = 16.27, p  < 0.001, lack of efficacy Q (1) = 15.47 ( p  < 0.001), and emotional exhaustion Q (1) = 13.85, p  < 0.001. A particularity of the moderator analysis in the relationship between ER difficulties and lack of efficacy is that, once the effect of the moderator is accounted for, the relationship is not statistically significant anymore for the university level, r  = -0.01 (95% CI = -0.132; 0.138), but significant for the pre-university level, r  = 0.33 (95% CI = 0.217; 0.439).

Meta-regressions

Meta-regression analyses were employed to examine how the effect size or relationship between variables changes based on continuous moderator variables. We included as moderators the female percentage (the proportion of female participants in each study’s sample) and the study quality assessed based on the Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields tool (Kmet et al., 2004 ).

Results, presented in Table  5 , show that study quality does not significantly influence the relationship between ER and school burnout. The proportion of female participants in the study sample significantly influences the relationship between ER difficulties and overall burnout ( β , -0.0055, SE = 0.001, p  < 0.001), as well as the emotional exhaustion dimension ( β , -0.0049, SE = 0.002, p  < 0.01). Mean age significantly influences the relationship between ER difficulties and overall burnout ( β , -0.0184, SE = 0.006, p  < 0.01). Meta-regression plots can be consulted in detail in Supplementary Material 3 .

A post hoc power analysis was conducted using the metapower package in R. For the pooled effects analysis of the relationship between ER difficulties and overall school burnout, as well as with cynicism and emotional exhaustion, the statistical power was adequate, surpassing the recommended 0.80 cutoff. The analysis of the association between ER difficulties and lack of efficacy, along with the relationship between adaptive ER and school burnout, cynicism, lack of efficacy, and emotional exhaustion were greatly underpowered. In the case of the moderator analysis, the post-hoc power analysis indicates insufficient power. Please consult the coefficients in Table  6 .

The central goal of this meta-analysis was to examine the relationship between emotion-regulation strategies and student burnout dimensions. Additionally, we focused on the possible effects of sample distribution, in particular on participants’ age, education levels they are enrolled in, and the percentage of female participants included in the sample. The study also aimed to determine how research quality influences the overall findings. Taking into consideration the possible moderating effects of sample characteristics and research quality, the study aimed to offer a thorough assessment of the literature concerning the association between emotion regulation strategies and student burnout dimensions. A correlation approach was used as the current literature predominantly consists of cross-sectional studies, with insufficient longitudinal studies or other designs that would allow for causal interpretation of the results.

The study’s main findings indicate that adaptive ER strategies are associated with overall burnout, whereas ER difficulties are associated with both overall burnout and all its dimensions encompassing emotional exhaustion, cynicism, and lack of efficacy.

Prior meta-analyses have similarly observed that adaptive ER strategies tend to exhibit modest negative associations with psychopathology, while ER difficulties generally presented more robust positive associations with psychopathology (Aldao et al., 2010 ; Miu et al., 2022 ). These findings could suggest that the observed variation in the effect of ER strategies on psychopathology, as previously indicated in the literature, can also be considered in the context of academic burnout.

However, it would be an oversimplification to conclude that adaptive ER strategies are less effective in preventing psychopathology than ER difficulties are in creating vulnerability to it. Alternatively, as previously underlined, researchers should consider the frequency, flexibility, and variability in the way ER strategies are applied and how they relate to well-being and psychopathology. Further, it’s important to also address the possible directionality of the relationship. While the few studies that assume a prediction model for academic burnout and ER treat ER as a predictor for burnout and its dimensions (see Seibert et al., 2017 ; Vizoso et al., 2019 ), we were unable to identify studies that assume the role of burnout in the development of ER difficulties. Additionally, the studies identified that relate to academic burnout have a cross-sectional design that makes it even more difficult to pinpoint the ecological directionality of the relationship.

While the focus on the causal role of ER strategies in psychopathology and psychological difficulties is of great importance for psychological interventions, addressing a factor that merely reflects an effect or consequence of psychopathology will not lead to an effective solution. According to Gross ( 2015 ), emotion regulation strategies are employed when there is a discrepancy between a person's current emotional state and their desired emotional state. Consequently, individuals could be likely to also utilize emotion regulation strategies in response to academic burnout. Additionally, studies that have utilized a longitudinal approach have demonstrated that, in the case of spontaneous ER, people with a history of psychopathology attempt to regulate their emotions more when presented with negative stimuli (Campbell-Sills et al., 2006a , 2006b ; Ehring et al., 2010 ; Gruber et al., 2012 ). The results of Dawel et al. ( 2021 ) further solidify a bidirectional model that could and should be also applied to academic burnout research.

Following the moderator analysis, the results indicate that the moderating effect of grade level did not have a substantial impact on the relationship between adaptive ER and school burnout. In the context of this discussion, it is important to note that regarding the relationship between adaptive ER and overall burnout, there is an imbalance in the distribution of studies between the university and pre-university levels, which could potentially present a source of bias or error.

When it comes to the relationship between ER difficulties and burnout, the inclusion of the moderator exhibited notable significance, overall and at the dimensions’ level. Particularly noteworthy is the finding that, within the relationship involving ER difficulties and lack of efficacy, the inclusion of the moderator rendered the association statistically insignificant for university-level students, while maintaining significance for pre-university-level students. The outcomes consistently demonstrate larger effect sizes for the relationship between ER difficulties and burnout at the pre-university level in comparison to the university level. Additionally, the mean age significantly influences the relationship between ER difficulties and overall burnout.

These findings may imply the presence of additional variables that exert a varying influence at the two educational levels and as a function of age. There are several contextual factors that could be framing the current findings, such as parental education anxiety (Wu et al., 2022 ), parenting behaviors, classroom atmosphere (Lin & Yang, 2021 ), and self-efficacy (Naderi et al., 2018 ). As the level of independence drastically increases from pre-university to university, the influence of negative parental behaviors and attitudes can become limited. Furthermore, the university-level learning environment often provides a satisfying and challenging educational experience, with greater opportunities for students to engage in decision-making and take an active role in their learning (Belaineh, 2017 ), which can serve as a protective factor against student’s academic burnout (Grech, 2021 ). At an individual level, many years of experience in navigating the educational environment can increase youths’ self-efficacy in the educational context and offer proper learning tools and techniques, which can further influence various aspects of self-regulated learning, such as monitoring of working time and task persistence (Bouffard-Bouchard et al., 1991 ; Cattelino et al., 2019 ).

The findings of the meta-regression analysis suggest that the association between ER and school burnout is not significantly impacted by study quality. It’s important to interpret these findings in the context of rather homogenous study quality ratings that can limit the detection of significant impacts.

The current results underline that the correlation between ER difficulties and both overall burnout and the emotional exhaustion dimension is significantly influenced by the percentage of female participants in the study sample. Previous research has shown that girls experience higher levels of stress, as well as higher expectations concerning their school performance, which can originate not only intrinsically, but also from external sources such as parents, peers, and educators (Östberg et al., 2015 ). These heightened expectations and stress levels may contribute to the gender differences in how emotion regulation difficulties are associated with school burnout.

The results of this meta-analysis suggest that most of the included studies present an increased level of methodological quality, reaching or surpassing the quality thresholds previously established. These encouraging results indicate a minimal risk of bias in the selected research. Moreover, it’s notable that a sizable proportion of the included studies clearly articulate their research objectives and employ well-established measurement tools, that would accurately capture the constructs of interest. There are still several areas of improvement, especially with regard to variable conceptualization and sampling methods, highlighting the importance of maintaining methodological rigor in this area of research.

Significant Q tests and I 2 identified in the case of several analyses indicate a strong heterogeneity among the effect sizes of individual studies in the meta-analysis's findings. This variability suggests that there is a significant level of diversity and variation among the effects observed in the studies, and it is improbable that this diversity is solely attributable to random chance. Even with as few as 10 studies, with 30 participants in the primary studies, the Q test has been demonstrated to have good power for identifying heterogeneity (Maeda & Harwell, 2016 ). Recent research (Mickenautsch et al., 2024 ), suggests that the I 2 statistic is not influenced by the number of studies and sample sizes included in a meta-analysis. While the relationships between Adaptive ER—cynicism, ER difficulties—cynicism, Adaptive ER—lack of efficacy, and ER difficulties—lack of efficacy are based on a limited number of studies (8–9 studies), it's noteworthy that the primary study sample sizes for these relationships are relatively large, averaging above 300. This suggests that despite the small number of studies, the robustness of the findings may be supported by the substantial sample sizes, which can contribute to the statistical power of the analysis.

However, it's essential to consider potential limitations such as range restriction or measurement error, which could impact the validity of the findings. Despite these considerations, the combination of substantial primary study sample sizes and the robustness of the Q test provides a basis for confidence in the results.

The results obtained when publication bias was examined using funnel plots, trim-and-fill analyses, and Egger's tests were varying across different outcomes. In the case of adaptive emotion regulation (ER) and school burnout, no evidence of publication bias was found, suggesting that the observed effects are likely robust. The trim-and-fill analysis, however, indicated the existence of missing studies for adaptive ER and cynicism, potentially influencing the initial effect size estimate. For ER difficulties and lack of efficacy, the adjustment for missing studies in the trim-and-fill analysis led to a non-significant effect. Additionally, adaptive ER and emotional exhaustion displayed a similar pattern with the trim-and-fill method leading to a lower, non-significant effect size. This indicates the need for additional studies to be included in future meta-analyses. According to the Cochrane Handbook (Higgins et al., 2011 ), the results of Egger’s test and funnel plot asymmetry should be interpreted with caution, when conducted on fewer than 10 studies.

The results of the post-hoc power analysis reveal that the relationship between ER difficulties and cynicism, as well as emotional exhaustion, meets the threshold of 0.80 for statistical power, as suggested by Harrer et al. ( 2022 ). This implies that our study had a high likelihood of detecting significant associations between ER difficulties and these specific outcomes, providing robust evidence for the observed relationships. However, for the relationship between ER difficulties and overall burnout, the power coefficient falls just below the indicated threshold. While our study still demonstrated considerable power to detect effects, the slightly lower coefficient suggests a marginally reduced probability of detecting significant associations between ER difficulties and overall burnout.

The power coefficients for the remaining post-hoc analyses are fairly small, which suggests that there is not enough statistical power to find meaningful relationships. This shows that there might not have been enough power in our investigation to find significant correlations between the variables we sought to investigate. Even if these analyses' power coefficients are lower than ideal, it's important to consider the study's limitations and implications when interpreting the results.

Limitations and Future Directions

One important limitation of our meta-analysis is represented by the small number of studies included in the analysis. Smaller meta-analyses could result in less reliable findings, with estimates that could be significantly influenced by outliers and inclusion of studies with extreme results. The small number of studies also interferes with the interpretation of both Q and I 2 heterogeneity indices (von Hippel, 2015 ). In small sample sizes, it may be challenging to detect true heterogeneity, and the I 2 value may be imprecise or underestimate the actual heterogeneity.

The studies included in the current meta-analysis focused on investigating how individuals generally respond to stressors. However, it's crucial to remember that people commonly use various ER strategies based on particular contexts, or they could even combine ER strategies within a single context. This adaptability in ER strategies reflects the dynamic and context-dependent nature of emotional regulation, where people draw upon various tools and approaches to effectively manage their emotions in different circumstances.

Given the heterogeneity of studies that investigate ER as a context-dependent phenomenon in the context of academic burnout, as well as the diverse nature of these existing studies, it becomes imperative for future research to consider a number of key aspects. First and foremost, future studies should aim to expand the body of literature on this topic by conducting more research specifically focusing on the context-dependent and flexible nature of ER in the context of academic burnout and other psychopathologies. Taking into account the diversity of educational environments, curricula, and student demographics, these research initiatives should also include a wide range of academic contexts.

Furthermore, it is advisable for researchers to implement a uniform methodology for assessing and documenting ER strategies. This consistency in measurement will simplify the process of comparing results among different studies, bolster the reliability of the data, and pave the way for more extensive and comprehensive meta-analyses.

Insufficient research that delves into the connection between burnout and particular emotional regulation (ER) strategies, such as reappraisal or suppression, has made it unfeasible to conduct a meaningful analysis within the scope of the current meta-analysis, that could further bring specificity as to which ER strategies could influence or be affected in the context of academic burnout. Consequentially, the expansion of the inclusion criteria for future meta-analyses should be considered, along with the replication of the current meta-analysis in the context of future publications on this topic.

Future interventions aimed at addressing academic burnout should adopt a tailored approach that takes into consideration age or school-level influences, as well as gender differences. Implementing prevention programs in pre-university educational settings can play a pivotal role in equipping children and adolescents with vital emotion regulation skills and stress management strategies. Additionally, it is essential to provide additional support to girls, recognizing their unique stressors and increased academic expectations.

Implications

Our meta-analysis has several implications, both theoretical and practical. Firstly, the meta-analysis extends the understanding of the relationship between emotion regulation (ER) strategies and student burnout dimensions. Although the correlational and cross-sectional nature of the included studies does not allow for drawing causal implications, the results represent a great stepping stone for future research. Secondly, the results highlight the intricacy of ER strategies and their applicability in educational contexts. Along with the identified differences between preuniversity and university students, this emphasizes the importance of developmental and contextual factors in ER research and the necessity of having an elaborate understanding of the ways in which these strategies are used in various situations and according to individual particularities. The significant impact of the percentage of female participants on the relationship between ER strategies and academic burnout points to the need for gender-sensitive approaches in ER research. On a practical level, our results suggest the need for targeted interventions aimed at the specific needs of different educational levels and age groups, as well as gender-specific strategies to address ER difficulties.

In conclusion, the findings of the current meta-analysis reveal that adaptive ER strategies are associated with overall burnout, while ER difficulties are linked to both overall burnout and its constituent dimensions, including emotional exhaustion, cynicism, and lack of efficacy. These results align with prior research in the domain of psychopathology, suggesting that adaptive ER strategies may be more efficient in preventing psychopathology than ER difficulties are in creating vulnerability to it, or that academic burnout negatively influences the use of adaptive ER strategies in the youth population. As an alternative explanation, it might also be that the association between ER strategies, well-being, and burnout can vary based on the context, frequency, flexibility, and variability of their application. Furthermore, our study identified the moderating role of grade level and the sample’s gender composition in shaping these associations. The academic environment, parental influences, and self-efficacy may contribute to the observed differences between pre-university and university levels and age differences.

Despite some methodological limitations, the current meta-analysis underscores the need for context-dependent ER research and consistent measurement approaches in future investigations of academic burnout and psychopathology. The heterogeneity among studies may suggest variability in the relationship between emotion regulation and student burnout across different contexts. This variability could be explained through methodological differences, assessment methods, and other contextual factors that were not uniformly accounted for in the included studies. The included studies do not provide insights into changes over time as most studies were cross-sectional. Future research should aim to better understand the underlying reasons for the observed differences and to reach more conclusive insights through longitudinal research designs.

Overall, this meta-analysis contributes to a deeper understanding of the intricate relationship between ER strategies and student burnout and serves as a good reference point for future research within the academic burnout field.

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Alarcon, G. M., Edwards, J. M., & Menke, L. E. (2011). Student Burnout and Engagement: A Test of the Conservation of Resources Theory. The Journal of Psychology, 145 (3), 211–227. https://doi.org/10.1080/00223980.2011.555432

Article   Google Scholar  

Aldao, A., & Nolen-Hoeksema, S. (2012a). The influence of context on the implementation of adaptive emotion regulation strategies. Behaviour Research and Therapy, 50 (7), 493–501. https://doi.org/10.1016/j.brat.2012.04.004

Aldao, A., & Nolen-Hoeksema, S. (2012b). When are adaptive strategies most predictive of psychopathology? Journal of Abnormal Psychology, 121 (1), 276–281. https://doi.org/10.1037/a0023598

Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clinical Psychology Review, 30 (2), 217–237. https://doi.org/10.1016/j.cpr.2009.11.004

Almutairi, H., Alsubaiei, A., Abduljawad, S., Alshatti, A., Fekih-Romdhane, F., Husni, M., & Jahrami, H. (2022). Prevalence of burnout in medical students: A systematic review and meta-analysis. International Journal of Social Psychiatry, 68 (6), 1157–1170. https://doi.org/10.1177/00207640221106691

Aloe, A. M., Amo, L. C., & Shanahan, M. E. (2014). Classroom Management Self-Efficacy and Burnout: A Multivariate Meta-analysis. Educational Psychology Review, 26 (1), 101–126. https://doi.org/10.1007/s10648-013-9244-0

Arias-Gundín, O. (Olga), & Vizoso-Gómez, C. (Carmen). (2018). Relación entre estrategias activas de afrontamiento, burnout y engagement en futuros educadores . https://doi.org/10.15581/004.35.409-427

Asareh, N., Pirani, Z., & Zanganeh, F. (2022). Evaluating the effectiveness of self-help cognitive and emotion regulation training On the psychological capital and academic motivation of female students with anxiety. Journal of School Psychology, 11 (2), 96–110. https://doi.org/10.22098/jsp.2022.1702

Balzarotti, S., John, O. P., & Gross, J. J. (2010). An Italian Adaptation of the Emotion Regulation Questionnaire. European Journal of Psychological Assessment, 26 (1), 61–67. https://doi.org/10.1027/1015-5759/a000009

Beck, A. T. (1976). Cognitive therapy and the emotional disorders. International Universities Press.

Bedewy, D., & Gabriel, A. (2015). Examining perceptions of academic stress and its sources among university students: The Perception of Academic Stress Scale. Health Psychology Open, 2 (2), 205510291559671. https://doi.org/10.1177/2055102915596714

Belaineh, M. S. (2017). Students’ Conception of Learning Environment and Their Approach to Learning and Its Implication on Quality Education. Educational Research and Reviews, 12 (14), 695–703.

Boada-Grau, J., Merino-Tejedor, E., Sánchez-García, J.-C., Prizmic-Kuzmica, A.-J., & Vigil-Colet, A. (2015). Adaptation and psychometric properties of the SBI-U scale for Academic Burnout in university students. Anales de Psicología / Annals of Psychology, 31 (1). https://doi.org/10.6018/analesps.31.1.168581

Borenstein, M., Higgins, J., Hedges, L., & Rothstein, H. (2017). Basics of meta-analysis: I(2) is not an absolute measure of heterogeneity. Research synthesis methods, 8. https://doi.org/10.1002/jrsm.1230

Bouffard-Bouchard, T., Parent, S., & Larivee, S. (1991). Influence of Self-Efficacy on Self-Regulation and Performance among Junior and Senior High-School Age Students. International Journal of Behavioral Development, 14 (2), 153–164. https://doi.org/10.1177/016502549101400203

Bresó, E., Schaufeli, W. B., & Salanova, M. (2011). Can a self-efficacy-based intervention decrease burnout, increase engagement, and enhance performance? A Quasi-Experimental Study. Higher Education, 61 (4), 339–355. https://doi.org/10.1007/s10734-010-9334-6

Burić, I., Sorić, I., & Penezić, Z. (2016). Emotion regulation in academic domain: Development and validation of the academic emotion regulation questionnaire (AERQ). Personality and Individual Differences, 96 , 138–147. https://doi.org/10.1016/j.paid.2016.02.074

Campbell-Sills, L., Barlow, D. H., Brown, T. A., & Hofmann, S. G. (2006a). Effects of suppression and acceptance on emotional responses of individuals with anxiety and mood disorders. Behaviour Research and Therapy, 44 (9), 1251–1263. https://doi.org/10.1016/j.brat.2005.10.001

Campbell-Sills, L., Barlow, D. H., Brown, T. A., & Hofmann, S. G. (2006b). Acceptability and suppression of negative emotion in anxiety and mood disorders. Emotion, 6 (4), 587–595. https://doi.org/10.1037/1528-3542.6.4.587

Carver, C. S. (1997). You want to measure coping but your protocol’ too long: Consider the brief cope. International Journal of Behavioral Medicine, 4 (1), 92–100. https://doi.org/10.1207/s15327558ijbm0401_6

Carver, C. S., Scheier, M. F., & Weintraub, J. K. (1989). Assessing coping strategies: A theoretically based approach. Journal of Personality and Social Psychology, 56 (2), 267–283. https://doi.org/10.1037/0022-3514.56.2.267

Cattelino, E., Morelli, M., Baiocco, R., & Chirumbolo, A. (2019). From external regulation to school achievement: The mediation of self-efficacy at school. Journal of Applied Developmental Psychology, 60 , 127–133. https://doi.org/10.1016/j.appdev.2018.09.007

Chacón-Cuberos, R., Martínez-Martínez, A., García-Garnica, M., Pistón-Rodríguez, M. D., & Expósito-López, J. (2019). The Relationship between Emotional Regulation and School Burnout: Structural Equation Model According to Dedication to Tutoring. International Journal of Environmental Research and Public Health, 16 (23), 4703. https://doi.org/10.3390/ijerph16234703

Charbonnier, E., Trémolière, B., Baussard, L., Goncalves, A., Lespiau, F., Philippe, A. G., & Le Vigouroux, S. (2022). Effects of an online self-help intervention on university students’ mental health during COVID-19: A non-randomized controlled pilot study. Computers in Human Behavior Reports, 5 , 100175. https://doi.org/10.1016/j.chbr.2022.100175

Chen, S., Zheng, Q., Pan, J., & Zheng, S. (2000). Preliminary development of the Coping Style Scale for Middle School Students. Chinese Journal of Clinical Psychology, 8 , 211–214, 237.

Córdova Olivera, P., Gasser Gordillo, P., Naranjo Mejía, H., La Fuente Taborga, I., Grajeda Chacón, A., & Sanjinés Unzueta, A. (2023). Academic stress as a predictor of mental health in university students. Cogent Education, 10 (2), 2232686. https://doi.org/10.1080/2331186X.2023.2232686

Davis, E. L., & Levine, L. J. (2013). Emotion Regulation Strategies That Promote Learning: Reappraisal Enhances Children’s Memory for Educational Information: Reappraisal and Memory in Children. Child Development, 84 (1), 361–374. https://doi.org/10.1111/j.1467-8624.2012.01836.x

Dawel, A., Shou, Y., Gulliver, A., Cherbuin, N., Banfield, M., Murray, K., Calear, A. L., Morse, A. R., Farrer, L. M., & Smithson, M. (2021). Cause or symptom? A longitudinal test of bidirectional relationships between emotion regulation strategies and mental health symptoms. Emotion, 21 (7), 1511–1521. https://doi.org/10.1037/emo0001018

Deb, S., Strodl, E., & Sun, H. (2015). Academic stress, parental pressure, anxiety and mental health among Indian high school students. International Journal of Psychology and Behavioral Science, 5 (1), 1.

Google Scholar  

Deeks, J. J., Bossuyt, P. M., Leeflang, M. M., & Takwoingi, Y. (2023). Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy . John Wiley & Sons.

Book   Google Scholar  

Dixon-Gordon, K. L., Chapman, A. L., Lovasz, N., & Walters, K. (2011). Too upset to think: The interplay of borderline personality features, negative emotions, and social problem solving in the laboratory. Personality Disorders: Theory, Research, and Treatment, 2 (4), 243–260. https://doi.org/10.1037/a0021799

Dominguez-Lara, S. A. (2018). Agotamiento emocional académico en estudiantes universitarios: ¿cuánto influyen las estrategias cognitivas de regulación emocional? Educación Médica, 19 (2), 96–103. https://doi.org/10.1016/j.edumed.2016.11.010

Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56 (2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x

Ehring, T., Tuschen-Caffier, B., Schnülle, J., Fischer, S., & Gross, J. J. (2010). Emotion regulation and vulnerability to depression: Spontaneous versus instructed use of emotion suppression and reappraisal. Emotion, 10 (4), 563–572. https://doi.org/10.1037/a0019010

Ezeudu, F. O., Attah, F. O., Onah, A. E., Nwangwu, T. L., & Nnadi, E. M. (2020). Intervention for burnout among postgraduate chemistry education students. Journal of International Medical Research, 48 (1), 0300060519866279. https://doi.org/10.1177/0300060519866279

Fong, M., & Loi, N. M. (2016). The Mediating Role of Self-compassion in Student Psychological Health. Australian Psychologist, 51 (6), 431–441. https://doi.org/10.1111/ap.12185

Frajerman, A., Morvan, Y., Krebs, M.-O., Gorwood, P., & Chaumette, B. (2019). Burnout in medical students before residency: A systematic review and meta-analysis. European Psychiatry: The Journal of the Association of European Psychiatrists, 55 , 36–42. https://doi.org/10.1016/j.eurpsy.2018.08.006

Garnefski, N., Kraaij, V., & Spinhoven, P. (2001). Negative life events, cognitive emotion regulation and emotional problems. Personality and Individual Differences, 30 (8), 1311–1327. https://doi.org/10.1016/S0191-8869(00)00113-6

Goldin, P. R., McRae, K., Ramel, W., & Gross, J. J. (2008). The neural bases of emotion regulation: Reappraisal and suppression of negative emotion. Biological Psychiatry, 63 (6), 577–586. https://doi.org/10.1016/j.biopsych.2007.05.031

Grech, M. (2021). The Effect of the Educational Environment on the rate of Burnout among Postgraduate Medical Trainees – A Narrative Literature Review. Journal of Medical Education and Curricular Development, 8 , 23821205211018700. https://doi.org/10.1177/23821205211018700

Gross, J. J. (1998a). The emerging field of emotion regulation: An integrative review. Review of General Psychology, 2 (3), 271–299. https://doi.org/10.1037/1089-2680.2.3.271

Gross, J. J. (1998b). Antecedent- and response-focused emotion regulation: Divergent consequences for experience, expression, and physiology. Journal of Personality and Social Psychology, 74 (1), 224–237. https://doi.org/10.1037/0022-3514.74.1.224

Gross, J. J. (2013). Emotion regulation: Taking stock and moving forward. Emotion, 13 (3), 359–365. https://doi.org/10.1037/a0032135

Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26 (1), 1–26. https://doi.org/10.1080/1047840X.2014.940781

Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85 (2), 348–362. https://doi.org/10.1037/0022-3514.85.2.348

Gross, J. J., & Levenson, R. W. (1993). Emotional suppression: Physiology, self-report, and expressive behavior. Journal of Personality and Social Psychology, 64 (6), 970–986. https://doi.org/10.1037/0022-3514.64.6.970

Gruber, J., Harvey, A. G., & Gross, J. J. (2012). When trying is not enough: Emotion regulation and the effort–success gap in bipolar disorder. Emotion, 12 (5), 997–1003. https://doi.org/10.1037/a0026822

Guessoum, S. B., Lachal, J., Radjack, R., Carretier, E., Minassian, S., Benoit, L., & Moro, M. R. (2020). Adolescent psychiatric disorders during the COVID-19 pandemic and lockdown. Psychiatry Research, 291 , 113264. https://doi.org/10.1016/j.psychres.2020.113264

Harrer, M., Cuijpers, P., Furukawa, T. A., & Ebert, D. D. (2022). Doing meta-analysis with R: A hands-on guide (First edition). CRC Press.

Hayes, S. C., Strosahl, K. D., & Wilson, K. G. (1999). Acceptance and commitment therapy: An experiential approach to behavior change (pp. xvi, 304). Guilford Press.

Herrmann, J., Koeppen, K., & Kessels, U. (2019). Do girls take school too seriously? Investigating gender differences in school burnout from a self-worth perspective. Learning and Individual Differences, 69 , 150–161. https://doi.org/10.1016/j.lindif.2018.11.011

Higgins, J. P. T., & Green, S. (Eds.) (2011.). Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Retrieved May 13, 2024 from www.handbook.cochrane.org .

Hofmann, S. G., & Asmundson, G. J. G. (2008). Acceptance and mindfulness-based therapy: New wave or old hat? Clinical Psychology Review, 28 (1), 1–16. https://doi.org/10.1016/j.cpr.2007.09.003

Hystad, S. W., Eid, J., Laberg, J. C., Johnsen, B. H., & Bartone, P. T. (2009). Academic Stress and Health: Exploring the Moderating Role of Personality Hardiness. Scandinavian Journal of Educational Research, 53 (5), 421–429. https://doi.org/10.1080/00313830903180349

Ibda, H., Wulandari, T. S., Abdillah, A., Hastuti, A. P., & Mahsun, M. (2023). Student academic stress during the COVID-19 pandemic: A systematic literature review. International Journal of Public Health Science (IJPHS), 12 (1), 286. https://doi.org/10.11591/ijphs.v12i1.21983

Jiang, S., Ren, Q., Jiang, C., & Wang, L. (2021). Academic stress and depression of Chinese adolescents in junior high schools: Moderated mediation model of school burnout and self-esteem. Journal of Affective Disorders, 295 , 384–389. https://doi.org/10.1016/j.jad.2021.08.085

Junyan, F., & Minqiang, Z. (2020). What is the minimum number of effect sizes required in meta-regression? An estimation based on statistical power and estimation precision. Advances in Psychological Science, 28 (4), 673. https://doi.org/10.3724/SP.J.1042.2020.00673

Kim, B., Jee, S., Lee, J., An, S., & Lee, S. M. (2018). Relationships between social support and student burnout: A meta-analytic approach. Stress and Health, 34 (1), 127–134. https://doi.org/10.1002/smi.2771

Kim, S., Kim, H., Park, E. H., Kim, B., Lee, S. M., & Kim, B. (2021). Applying the demand–control–support model on burnout in students: A meta-analysis. Psychology in the Schools, 58 (11), 2130–2147. https://doi.org/10.1002/pits.22581

Kmet, Leanne M. ; Cook, Linda S. ; Lee, Robert C. (2004). Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields . https://doi.org/10.7939/R37M04F16

Kobylińska, D., & Kusev, P. (2019). Flexible Emotion Regulation: How Situational Demands and Individual Differences Influence the Effectiveness of Regulatory Strategies. Frontiers in Psychology , 10 . https://doi.org/10.3389/fpsyg.2019.00072

Koole, S. L. (2009). The psychology of emotion regulation: An integrative review. Cognition and Emotion, 23 (1), 4–41. https://doi.org/10.1080/02699930802619031

Kristensen, T. S., Borritz, M., Villadsen, E., & Christensen, K. B. (2005). The copenhagen burnout inventory: A new tool for the assessment of burnout. Work & Stress, 19 (3), 192–207. https://doi.org/10.1080/02678370500297720

Larsen, R. J. (2000). Toward a science of mood regulation. Psychological Inquiry, 11 (3), 129–141. https://doi.org/10.1207/S15327965PLI1103_01

Lau, S. C., Chow, H. J., Wong, S. C., & Lim, C. S. (2020). An empirical study of the influence of individual-related factors on undergraduates’ academic burnout: Malaysian context. Journal of Applied Research in Higher Education, 13 (4), 1181–1197. https://doi.org/10.1108/JARHE-02-2020-0037

Leppanen, J., Brown, D., McLinden, H., Williams, S., & Tchanturia, K. (2022). The Role of Emotion Regulation in Eating Disorders: A Network Meta-Analysis Approach. Frontiers in Psychiatry, 13. https://doi.org/10.3389/fpsyt.2022.793094

Libert, C., Chabrol, H., & Laconi, S. (2019). Exploration du burn-out et du surengagement académique dans un échantillon d’étudiants. Journal De Thérapie Comportementale Et Cognitive, 29 (3), 119–131. https://doi.org/10.1016/j.jtcc.2019.01.001

Lin, F., & Yang, K. (2021). The External and Internal Factors of Academic Burnout: 2021 4th International Conference on Humanities Education and Social Sciences (ICHESS 2021), Xishuangbanna, China. https://doi.org/10.2991/assehr.k.211220.307

Linehan, M. M. (1993). Cognitive-behavioral treatment of borderline personality disorder (pp. xvii, 558). Guilford Press.

Lo, H. H. M., Ngai, S., & Yam, K. (2021). Effects of Mindfulness-Based Stress Reduction on Health and Social Care Education: A Cohort-Controlled Study. Mindfulness, 12 (8), 2050–2058. https://doi.org/10.1007/s12671-021-01663-z

Luszczynska, A., Diehl, M., Gutiérrez-Doña, B., Kuusinen, P., & Schwarzer, R. (2004). Measuring one component of dispositional self-regulation: Attention control in goal pursuit. Personality and Individual Differences, 37 (3), 555–566. https://doi.org/10.1016/j.paid.2003.09.026

Luo, Y., Wang, Z., Zhang, H., Chen, A., & Quan, S. (2016). The effect of perfectionism on school burnout among adolescence: The mediator of self-esteem and coping style. Personality and Individual Differences, 88 , 202–208. https://doi.org/10.1016/j.paid.2015.08.056

Luo, Y., Deng, Y., & Zhang, H. (2020). The influences of parental emotional warmth on the association between perceived teacher–student relationships and academic stress among middle school students in China. Children and Youth Services Review, 114 , 105014. https://doi.org/10.1016/j.childyouth.2020.105014

Lynch, T. R., Trost, W. T., Salsman, N., & Linehan, M. M. (2007). Dialectical behavior therapy for borderline personality disorder. Annual Review of Clinical Psychology, 3 , 181–205. https://doi.org/10.1146/annurev.clinpsy.2.022305.095229

Madigan, D. J., & Curran, T. (2021). Does burnout affect academic achievement? A meta-analysis of over 100,000 students. Educational Psychology Review, 33 (2), 387–405. https://doi.org/10.1007/s10648-020-09533-1

Madigan, D. J., Kim, L. E., & Glandorf, H. L. (2023). Interventions to reduce burnout in students: A systematic review and meta-analysis. European Journal of Psychology of Education . https://doi.org/10.1007/s10212-023-00731-3

Maeda, Y., & Harwell, M. (2016). Guidelines for using the Q Test in Meta-Analysis. Mid-Western Educational Researcher, 28 (1). Retrieved May 22, 2024, from https://scholarworks.bgsu.edu/mwer/vol28/iss1/4

Marques, H., Brites, R., Nunes, O., Hipólito, J., & Brandão, T. (2023). Attachment, emotion regulation, and burnout among university students: A mediational hypothesis. Educational Psychology, 43 (4), 344–362. https://doi.org/10.1080/01443410.2023.2212889

Matud, M. P., Díaz, A., Bethencourt, J. M., & Ibáñez, I. (2020). Stress and Psychological Distress in Emerging Adulthood: A Gender Analysis. Journal of Clinical Medicine, 9 (9), 2859. https://doi.org/10.3390/jcm9092859

May, R. W., Bauer, K. N., & Fincham, F. D. (2015). School burnout: Diminished academic and cognitive performance. Learning and Individual Differences, 42 , 126–131. https://doi.org/10.1016/j.lindif.2015.07.015

Mennin, D. S., Holaway, R. M., Fresco, D. M., Moore, M. T., & Heimberg, R. G. (2007). Delineating components of emotion and its dysregulation in anxiety and mood psychopathology. Behavior Therapy, 38 (3), 284–302. https://doi.org/10.1016/j.beth.2006.09.001

Merino-Tejedor, E., Hontangas, P. M., & Boada-Grau, J. (2016). Career adaptability and its relation to self-regulation, career construction, and academic engagement among Spanish university students. Journal of Vocational Behavior, 93 , 92–102. https://doi.org/10.1016/j.jvb.2016.01.005

Meylan, N., Doudin, P.-A., Curchod-Ruedi, D., & Stephan, P. (2015). Burnout scolaire et soutien social: L’importance du soutien des parents et des enseignants. Psychologie Française, 60 (1), 1–15. https://doi.org/10.1016/j.psfr.2014.01.003

Mickenautsch, S., Yengopal, V., Mickenautsch, S., & Yengopal, V. (2024). Trial Number and Sample Size Do Not Affect the Accuracy of the I2-Point Estimate for Testing Selection Bias Risk in Meta-Analyses. Cureus, 16 , 4. https://doi.org/10.7759/cureus.58961

Midgley, C., Maehr, M., Hruda, L., Anderman, E., Anderman, L., Freeman, K., Gheen, M., Kaplan, A., Kumar, R., Middleton, M., Nelson, J., Roeser, R., & Urdan, T. (2000). The patterns of adaptive learning scales (PALS) 2000 [Dataset].

Miola, A., Cattarinussi, G., Antiga, G., Caiolo, S., Solmi, M., & Sambataro, F. (2022). Difficulties in emotion regulation in bipolar disorder: A systematic review and meta-analysis. Journal of Affective Disorders, 302 , 352–360. https://doi.org/10.1016/j.jad.2022.01.102

Miu, A. C., Szentágotai-Tătar, A., Balázsi, R., Nechita, D., Bunea, I., & Pollak, S. D. (2022). Emotion regulation as mediator between childhood adversity and psychopathology: A meta-analysis. Clinical Psychology Review, 93 , 102141. https://doi.org/10.1016/j.cpr.2022.102141

Modrego-Alarcón, M., López-Del-Hoyo, Y., García-Campayo, J., Pérez-Aranda, A., Navarro-Gil, M., Beltrán-Ruiz, M., Morillo, H., Delgado-Suarez, I., Oliván-Arévalo, R., & Montero-Marin, J. (2021). Efficacy of a mindfulness-based programme with and without virtual reality support to reduce stress in university students: A randomized controlled trial. Behaviour Research and Therapy, 142 , 103866. https://doi.org/10.1016/j.brat.2021.103866

Mohammadi Bytamar, J., Saed, O., & Khakpoor, S. (2020). Emotion Regulation Difficulties and Academic Procrastination. Frontiers in Psychology, 11 , 524588. https://doi.org/10.3389/fpsyg.2020.524588

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339 , b2535. https://doi.org/10.1136/bmj.b2535

Muchacka-Cymerman, A., & Tomaszek, K. (2018). Polish Adaptation of the ESSBS School-Burnout Scale: Pilot Study Results. Hacettepe University Journal of Education , 1–16. https://doi.org/10.16986/HUJE.2018043462

Naderi, Z., Bakhtiari, S., Momennasab, M., Abootalebi, M., & Mirzaei, T. (2018). Prediction of academic burnout and academic performance based on the need for cognition and general self-efficacy: A cross-sectional analytical study. Revista Latinoamericana De Hipertensión, 13 (6), 584–591.

Narimanj, A., Kazemi, R., & Narimani, M. (2021). Relationship between Cognitive Emotion Regulation, Personal Intelligence and Academic Burnout. Journal of Modern Psychological Researches, 16 (61), 65–74.

Neacsiu, A. D., Rizvi, S. L., & Linehan, M. M. (2010). Dialectical behavior therapy skills use as a mediator and outcome of treatment for borderline personality disorder. Behaviour Research and Therapy, 48 (9), 832–839. https://doi.org/10.1016/j.brat.2010.05.017

Neff, K. D. (2003). The development and validation of a scale to measure self-compassion. Self and Identity, 2 (3), 223–250. https://doi.org/10.1080/15298860309027

Nikdel, F., Hadi, J., & Ali, T. (2019). SOCIAL SCIENCES & HUMANITIES Students’ Academic Stress, Stress Response and Academic Burnout: Mediating Role of Self-Efficacy .

Noh, H., Chang, E., Jang, Y., Lee, J. H., & Lee, S. M. (2016). Suppressor Effects of Positive and Negative Religious Coping on Academic Burnout Among Korean Middle School Students. Journal of Religion and Health, 55 (1), 135–146. https://doi.org/10.1007/s10943-015-0007-8

Nolen-Hoeksema, S., Wisco, B. E., & Lyubomirsky, S. (2008). Rethinking Rumination. Perspectives on Psychological Science, 3 (5), 400–424. https://doi.org/10.1111/j.1745-6924.2008.00088.x

Nyklícek, I., & Temoshok, L. (2004). Emotional expression and health: Advances in theory, assessment and clinical applications . Routledge.

Ogbuanya, T. C., Eseadi, C., Orji, C. T., Omeje, J. C., Anyanwu, J. I., Ugwoke, S. C., & Edeh, N. C. (2019). Effect of Rational-Emotive Behavior Therapy Program on the Symptoms of Burnout Syndrome Among Undergraduate Electronics Work Students in Nigeria. Psychological Reports, 122 (1), 4–22. https://doi.org/10.1177/0033294117748587

Östberg, V., Almquist, Y. B., Folkesson, L., Låftman, S. B., Modin, B., & Lindfors, P. (2015). The Complexity of Stress in Mid-Adolescent Girls and Boys. Child Indicators Research, 8 (2), 403–423. https://doi.org/10.1007/s12187-014-9245-7

Park, E.-Y., & Shin, M. (2020). A Meta-Analysis of Special Education Teachers’ Burnout. SAGE Open, 10 (2), 2158244020918297. https://doi.org/10.1177/2158244020918297

Parkinson, B., & Totterdell, P. (1999). Classifying affect-regulation strategies. Cognition and Emotion, 13 (3), 277–303. https://doi.org/10.1080/026999399379285

Pines, A., & Aronson, E. (1988). Career Burnout: Causes and Cures . Free Press.

Popescu, B., Maricuțoiu, L. P., & De Witte, H. (2023). The student version of the Burnout assessement tool (BAT): Psychometric properties and evidence regarding measurement validity on a romanian sample. Current Psychology . https://doi.org/10.1007/s12144-023-04232-w

Prefit, A.-B., Cândea, D. M., & Szentagotai-Tătar, A. (2019). Emotion regulation across eating pathology: A meta-analysis. Appetite, 143 , 104438. https://doi.org/10.1016/j.appet.2019.104438

Prospero. (2022). Systematic review registration: Emotion regulation and academic burnout in youths: a meta-analysis. Retrieved May 22, 2024, from  https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=325570

Ramírez, M. T. G., & Hernández, R. L. (2007). ESCALA DE CANSANCIO EMOCIONAL (ECE) PARA ESTUDIANTES UNIVERSITARIOS: PROPIEDADES PSICOMÉTRICAS EN UNA MUESTRA DE MÉXICO. Anales de Psicología / Annals of Psychology, 23 (2).

Richards, J. M., & Gross, J. J. (2000). Emotion regulation and memory: The cognitive costs of keeping one’s cool. Journal of Personality and Social Psychology, 79 (3), 410–424. https://doi.org/10.1037/0022-3514.79.3.410

Richards, J. M., Butler, E. A., & Gross, J. J. (2003). Emotion regulation in romantic relationships: The cognitive consequences of concealing feelings. Journal of Social and Personal Relationships, 20 (5), 599–620. https://doi.org/10.1177/02654075030205002

Roemer, L., Orsillo, S. M., & Salters-Pedneault, K. (2008). Efficacy of an acceptance-based behavior therapy for generalized anxiety disorder: Evaluation in a randomized controlled trial. Journal of Consulting and Clinical Psychology, 76 (6), 1083–1089. https://doi.org/10.1037/a0012720

Salmela-Aro, K. (2017). Dark and bright sides of thriving – school burnout and engagement in the Finnish context. European Journal of Developmental Psychology, 14 (3), 337–349. https://doi.org/10.1080/17405629.2016.1207517

Salmela-Aro, K., & Tynkkynen, L. (2012). Gendered pathways in school burnout among adolescents. Journal of Adolescence, 35 (4), 929–939. https://doi.org/10.1016/j.adolescence.2012.01.001

Salmela-aro *, K., Näätänen, P., & Nurmi, J. (2004). The role of work-related personal projects during two burnout interventions: A longitudinal study. Work & Stress, 18(3), 208–230. https://doi.org/10.1080/02678370412331317480

Salmela-Aro, K., Kiuru, N., Leskinen, E., & Nurmi, J.-E. (2009). School burnout inventory (SBI). European Journal of Psychological Assessment, 25 (1), 48–57. https://doi.org/10.1027/1015-5759.25.1.48

Santos Alves Peixoto, L., Guedes Gondim, S. M., & Pereira, C. R. (2022). Emotion Regulation, Stress, and Well-Being in Academic Education: Analyzing the Effect of Mindfulness-Based Intervention. Trends in Psychology, 30 (1), 33–57. https://doi.org/10.1007/s43076-021-00092-0

Scales, P. C., Benson, P. L., Oesterle, S., Hill, K. G., Hawkins, J. D., & Pashak, T. J. (2016). The dimensions of successful young adult development: A conceptual and measurement framework. Applied Developmental Science, 20 (3), 150–174. https://doi.org/10.1080/10888691.2015.1082429

Schaufeli, W. B., Salanova, M., González-romá, V., & Bakker, A. B. (2002). The measurement of engagement and burnout: A two sample confirmatory factor analytic approach. Journal of Happiness Studies, 3 (1), 71–92. https://doi.org/10.1023/A:1015630930326

Schaufeli, W. B., Desart, S., & De Witte, H. (2020). Burnout assessment tool (BAT)—development, validity, and reliability. International Journal of Environmental Research and Public Health, 17 (24). https://doi.org/10.3390/ijerph17249495

Schmid, C. H., Stijnen, T., & White, I. (2020). Handbook of Meta-Analysis . CRC Press.

Segal, Z. V., Williams, J. M. G., & Teasdale, J. D. (2002). Mindfulness-based cognitive therapy for depression: A new approach to preventing relapse (pp. xiv, 351). Guilford Press.

Séguin, D. G., & MacDonald, B. (2018). The role of emotion regulation and temperament in the prediction of the quality of social relationships in early childhood. Early Child Development and Care, 188 (8), 1147–1163. https://doi.org/10.1080/03004430.2016.1251678

Seibert, G. S., Bauer, K. N., May, R. W., & Fincham, F. D. (2017). Emotion regulation and academic underperformance: The role of school burnout. Learning and Individual Differences, 60 , 1–9. https://doi.org/10.1016/j.lindif.2017.10.001

Shahidi, S., Akbari, H., & Zargar, F. (2017). Effectiveness of mindfulness-based stress reduction on emotion regulation and test anxiety in female high school students. Journal of Education and Health Promotion, 6 , 87. https://doi.org/10.4103/jehp.jehp_98_16

Shih, S.-S. (2013). The effects of autonomy support versus psychological control and work engagement versus academic burnout on adolescents’ use of avoidance strategies. School Psychology International, 34 (3), 330–347. https://doi.org/10.1177/0143034312466423

Shih, S.-S. (2015a). An Examination of Academic Coping Among Taiwanese Adolescents. The Journal of Educational Research, 108 (3), 175–185. https://doi.org/10.1080/00220671.2013.867473

Shih, S.-S. (2015b). The relationships among Taiwanese adolescents’ perceived classroom environment, academic coping, and burnout. School Psychology Quarterly: The Official Journal of the Division of School Psychology, American Psychological Association, 30 (2), 307–320. https://doi.org/10.1037/spq0000093

Stellern, J., Xiao, K. B., Grennell, E., Sanches, M., Gowin, J. L., & Sloan, M. E. (2023). Emotion regulation in substance use disorders: A systematic review and meta-analysis. Addiction, 118 (1), 30–47. https://doi.org/10.1111/add.16001

Tobin, D. L., Holroyd, K. A., Reynolds, R. V., & Wigal, J. K. (1989). The hierarchical factor structure of the Coping Strategies Inventory. Cognitive Therapy and Research, 13 (4), 343–361. https://doi.org/10.1007/BF01173478

Troy, A. S., Shallcross, A. J., & Mauss, I. B. (2013). A Person-by-Situation Approach to Emotion Regulation: Cognitive Reappraisal Can Either Help or Hurt. Depending on the Context. Psychological Science, 24 (12), 2505–2514. https://doi.org/10.1177/0956797613496434

Tull, M. T., & Aldao, A. (2015). Editorial overview: New directions in the science of emotion regulation. Current Opinion in Psychology, 3 , iv–x. https://doi.org/10.1016/j.copsyc.2015.03.009

Vinter, K., Aus, K., & Arro, G. (2021). Adolescent girls’ and boys’ academic burnout and its associations with cognitive emotion regulation strategies. Educational Psychology, 41 (8), 1061–1077. https://doi.org/10.1080/01443410.2020.1855631

Vizoso, C., Arias-Gundín, O., & Rodríguez, C. (2019). Exploring coping and optimism as predictors of academic burnout and performance among university students. Educational Psychology, 39 (6), 768–783. https://doi.org/10.1080/01443410.2018.1545996

von Hippel, P. T. (2015). The heterogeneity statistic I(2) can be biased in small meta-analyses. BMC Medical Research Methodology, 15 , 35. https://doi.org/10.1186/s12874-015-0024-z

Walburg, V. (2014). Burnout among high school students: A literature review. Children and Youth Services Review, 42 , 28–33. https://doi.org/10.1016/j.childyouth.2014.03.020

Webb, T. L., Miles, E., & Sheeran, P. (2012). Dealing with feeling: A meta-analysis of the effectiveness of strategies derived from the process model of emotion regulation. Psychological Bulletin, 138 (4), 775–808. https://doi.org/10.1037/a0027600

Weiss, N. H., Kiefer, R., Goncharenko, S., Raudales, A. M., Forkus, S. R., Schick, M. R., & Contractor, A. A. (2022). Emotion regulation and substance use: A meta-analysis. Drug and Alcohol Dependence, 230 , 109131. https://doi.org/10.1016/j.drugalcdep.2021.109131

Westhues, A., & Cohen, J. S. (1997). A comparison of the adjustment of adolescent and young adult inter-country adoptees and their siblings. International Journal of Behavioral Development, 20 (1), 47–65. https://doi.org/10.1080/016502597385432

Wu, K., Wang, F., Wang, W., & Li, Y. (2022). Parents’ Education Anxiety and Children’s Academic Burnout: The Role of Parental Burnout and Family Function. Frontiers in Psychology , 12 . https://doi.org/10.3389/fpsyg.2021.764824

Yang, H., & Chen, J. (2016). Learning Perfectionism and Learning Burnout in a Primary School Student Sample: A Test of a Learning-Stress Mediation Model. Journal of Child and Family Studies, 25 (1), 345–353. https://doi.org/10.1007/s10826-015-0213-8

Yang, C., Chen, A., & Chen, Y. (2021). College students’ stress and health in the COVID-19 pandemic: The role of academic workload, separation from school, and fears of contagion. PLoS ONE, 16 (2), e0246676. https://doi.org/10.1371/journal.pone.0246676

Yildiz, M. A. (2017). Pathways to positivity from perceived stress in adolescents: Multiple mediation of emotion regulation and coping strategies. Current Issues in Personality Psychology, 5 (4), 272–284. https://doi.org/10.5114/cipp.2017.67894

Yu, X., Wang, Y., & Liu, F. (2022). Language learning motivation and burnout among english as a foreign language undergraduates: The moderating role of maladaptive emotion regulation strategies. Frontiers in Psychology , 13 .  https://www.frontiersin.org/articles/10.3389/fpsyg.2022.808118

Zahniser, E., & Conley, C. S. (2018). Interactions of emotion regulation and perceived stress in predicting emerging adults’ subsequent internalizing symptoms. Motivation and Emotion, 42 (5), 763–773. https://doi.org/10.1007/s11031-018-9696-0

Download references

Acknowledgements

This work was supported by two grants awarded to the corresponding author from the Romanian National Authority for Scientific Research, CNCS—UEFISCDI (Grant number PN-III-P4-ID-PCE-2020-2170 and PN-III-P2-2.1-PED-2021-3882)

Author information

Authors and affiliations.

Evidence-Based Psychological Assessment and Interventions Doctoral School, Babes-Bolyai University of Cluj-Napoca, Cluj-Napoca, Napoca, Romania

Ioana Alexandra Iuga

DATA Lab, The International Institute for the Advanced Studies of Psychotherapy and Applied Mental Health, Babes-Bolyai University Cluj-Napoca, Cluj-Napoca, Romania

Ioana Alexandra Iuga & Oana Alexandra David

Department of Clinical Psychology and Psychotherapy, Babeş-Bolyai University, No 37 Republicii Street, 400015, Cluj-Napoca, Napoca, Romania

Oana Alexandra David

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Oana Alexandra David .

Ethics declarations

Competing interests.

The authors declare that they have no known competing financial interests or personal relationships that influence the work reported in this paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 26534 KB)

Supplementary file2 (docx 221 kb), supplementary file3 (docx 315 kb), supplementary file4 (docx 16 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Iuga, I.A., David, O.A. Emotion Regulation and Academic Burnout Among Youth: a Quantitative Meta-analysis. Educ Psychol Rev 36 , 106 (2024). https://doi.org/10.1007/s10648-024-09930-w

Download citation

Accepted : 01 August 2024

Published : 10 September 2024

DOI : https://doi.org/10.1007/s10648-024-09930-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emotion regulation
• Academic burnout
• Meta-analysis
• Find a journal
• Publish with us
• Track your research

• Systematic Review
• Open access
• Published: 07 September 2024

Assessing the effectiveness of greater occipital nerve block in chronic migraine: a systematic review and meta-analysis

• Muhamad Saqlain Mustafa 1 , 7 ,
• Shafin bin Amin 2 , 8 ,
• Aashish Kumar 2 , 8 ,
• Muhammad Ashir Shafique 1 , 7 ,
• Syeda Mahrukh Fatima Zaidi 3 , 9 ,
• Syed Ali Arsal 2 , 8 ,
• Burhanudin Sohail Rangwala 1 , 7 ,
• Muhammad Faheem Iqbal 3 , 9 ,
• Adarsh Raja 2 , 8 ,
• Abdul Haseeb 1 , 7 ,
• Inshal Jawed 3 , 9 ,
• Khabab Abbasher Hussien Mohamed Ahmed   ORCID: orcid.org/0000-0003-4608-5321 5 ,
• Syed Muhammad Sinaan Ali 6 , 10 &
• Giustino Varrassi 4  

BMC Neurology volume  24 , Article number:  330 ( 2024 ) Cite this article

1 Altmetric

Metrics details

Background & aims

Chronic migraine poses a global health burden, particularly affecting young women, and has substantial societal implications. This study aimed to assess the efficacy of Greater Occipital Nerve Block (GONB) in individuals with chronic migraine, focusing on the impact of local anesthetics compared with placebo.

A meta-analysis and systematic review were conducted following the PRISMA principles and Cochrane Collaboration methods. Eligible studies included case-control, cohort, and randomized control trials in adults with chronic migraine, adhering to the International Classification of Headache Disorders, third edition (ICHD3). Primary efficacy outcomes included headache frequency, duration, and intensity along with safety assessments.

Literature searches across multiple databases yielded eight studies for qualitative analysis, with five included in the final quantitative analysis. A remarkable reduction in headache intensity and frequency during the first and second months of treatment with GONB using local anesthetics compared to placebo has been reported. The incidence of adverse events did not differ significantly between the intervention and placebo groups.

The analysis emphasized the safety and efficacy of GONB, albeit with a cautious interpretation due to the limited number of studies and relatively small sample size. This study advocates for further research exploring various drugs, frequencies, and treatment plans to enhance the robustness and applicability of GONB for chronic migraine management.

Peer Review reports

Introduction

Among headache disorders, migraine is particularly ranked second worldwide in terms of disability and is the leading cause of disability among young women, according to the Global Burden of Disease 2019 data [ 1 ]. Recent findings indicate that the global prevalence of migraine is approximately 15%, which translates to 4.9% of all ill health measured in years lived with disability (YLDs) [ 2 ]. Women are more likely to experience migraine than men, particularly those aged 15–49 years [ 3 ]. Migraine has a substantial societal and financial impact owing to both direct and indirect costs resulting from decreased productivity and missed work [ 4 ].

Migraine is a complex neurovascular disorder that affects sensory processing and is characterized by a range of symptoms, with headache being the most common symptom [ 5 ]. Chronic migraine (CM) is defined as the frequent occurrence of headache episodes, with at least 15 or more episodes (which, on at least 8 days/month, have the features of migraine headache) occurring per month for more than three months [ 6 ]. Several medications are available for the preventive treatment of migraine, including anticonvulsants, antidepressants, beta-blockers, calcium channel blockers, botulinum toxin A, and more recently, drugs that block the calcitonin gene-related peptide (CGRP) pathway (i.e., monoclonal antibodies and antagonists) [ 7 ]. Despite the potential of anti-CGRP monoclonal antibodies (mAbs) in managing chronic migraine, a remarkable proportion of patients do not respond to this treatment [ 8 ]. Approximately 25% of patients are unresponsive to anti-CGRP monoclonal antibodies [ 9 ].

An important component of the brainstem, the Trigeminocervical Complex (TCC) acts as a central processing unit for pain and sensory data from the head and neck. This is the point of convergence of the upper cervical spinal nerves and the trigeminal nerve, which supplies feeling to the face, head, and some regions of the neck [ 10 , 11 ].

One of the TCC’s primary functions is the confluence of the occipital and trigeminal nerves there. The trigeminal nerve transmits sensory data from the face, scalp, and meninges through its three main branches (ophthalmic, maxillary, and mandibular). In the meanwhile, feelings from the back of the head are transmitted by the occipital nerves, which originate from the upper cervical spinal roots [ 10 , 11 ]. Wide-ranging integration of sensory inputs from the head and neck is made possible by the network formed when these neurons converge at the TCC. The brainstem area known as the trigeminocervical complex is crucial to migraine pain processing since it is responsible for processing pain signals originating from the head and neck. [ 10 , 11 ]. The face, head, and neck region’s sensory data—especially pain—are integrated by the TCC. Because of this integration, the TCC is an important piece of the migraine jigsaw when it comes to interpreting the location and degree of pain. The trigeminal, occipital, and TCC nerves are intricately intertwined with one another. A series of neurological events are set off during a migraine episode, beginning with the stimulation of the trigeminal nerve. This activation increases pain signals by causing the production of inflammatory chemicals around the TCC and blood arteries in the brain [ 10 , 11 ]. Accompanying this, the occipital nerves may also be affected, particularly if the headache radiates to the rear of the head. Because of its connection, the TCC is further stimulated by pain signals from the occipital area, worsening the migraine sensation (it produces a feedback loop) [ 10 , 11 ].

The main sensory nerve that serves the occipital region is the Greater Occipital Nerve (GON), which predominantly originates from the C2 dorsal root. The GON block is used in acute and preventive headache treatments as it targets the anatomical and functional connections between the trigeminal and cervical fibers within the trigemino-cervical complex (TCC). The rationale for using GON blocks is based on the integration of sensory neurons from C2 in the upper cervical spinal cord with neurons in the trigeminal nucleus caudalis (TNC). However, the precise mechanisms by which GON blocks may affect the TCC and potentially reduce its activity are still being researched [ 12 ]. However, there is currently no standard protocol for GONB. Local anesthetics function by preventing the activation of voltage-gated sodium channels, which reduce the transmission of sensory signals originating from areas innervated by the greater occipital nerve, such as the medial region of the posterior scalp [ 13 , 14 ], thereby preventing the activation of convergent neurons in the trigeminal-cervical complex. Combination therapy with corticosteroids may reduce inflammation, thereby attenuating pain, however, this role of corticosteroids also seems to be under debate.

The current management of chronic migraines is inadequate, as it lacks clear guidelines despite the various treatment options available. The evidence supporting the efficacy of GONB in preventing chronic migraines is limited and not recent [ 15 , 16 , 17 ]. However, the emergence of new clinical trials offers a promising opportunity for this study to provide valuable insights to healthcare providers. This study aims to fill the knowledge gaps by conducting a comprehensive systematic review and meta-analysis, providing healthcare professionals with a more complete understanding of the collective results of this approach for the treatment of chronic migraines.

A meta-analysis and a comprehensive systematic review were conducted to assess the efficacy of GONB in patients with CM, adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 18 ]. The PICO framework, a cornerstone of evidence-based medicine, organizes clinical questions and study designs into Population, Intervention, Comparison, and Outcome. In our research on chronic migraine treatment, we examine the efficacy of greater occipital nerve block (Intervention) with local anesthetics alone versus a placebo (Comparison) among adults with chronic migraine (Population), focusing on changes in migraine intensity measured by VAS, frequency, and adverse effects (Outcome).

Eligibility criteria

Inclusion criteria for studies considered in this meta-analysis encompassed randomized controlled trials (RCTs) evaluating the efficacy of greater occipital nerve block (GONB) with local anesthetics alone compared to a placebo in adult individuals diagnosed with chronic migraine. Studies were required to report outcomes including changes in migraine intensity measured by Visual Analog Scale (VAS), frequency of migraine episodes, and documentation of adverse effects. Exclusion criteria comprised studies that incorporated corticosteroids in conjunction with local anesthetics for GONB, non-randomized or non-controlled trials, studies with insufficient data for outcome assessment, and those involving populations other than adults with chronic migraine.

The primary efficacy endpoints were the change in headache intensity as measured by any scale, the frequency of headache (days per month) in the intervention group compared to the placebo group at a specific point in time, and the intensity of headache in the intervention group compared to the placebo group. To assess safety, the analysis focused on the number of participants who experienced at least one adverse event (AE) and the total number of participants who experienced AEs.

Literature search and study selection

A systematic search of PubMed, Medline, Scopus, Embase, Cochrane, Web of Science, and PsycINFO was performed as of June 2023 by two authors AR and AH. All languages and publication dates were considered and the search strategy involved both free and restricted terms pertaining to migraine and GONB, using key word ‘Chronic migraine’ or Migraine’ or ‘Greater Occipital Nerve Block’. Duplicates were eliminated and the titles and abstracts of the remaining articles were assessed to identify relevant studies. Subsequently, a full-text assessment was performed by two independent investigators (AK and BSR) and any discrepancies were resolved by a third investigator (MSM). The PRISMA flowchart (Fig.  1 ) illustrates the selection process.

Prisma flow chart

Data extraction

We utilized a standard Microsoft Excel 2021 spreadsheet to gather data from each study included in a predetermined format. Two unbiased investigators (MAS and SMFZ) collected the following information from each study: author, year of publication, population, intervention and comparison drugs, techniques, primary and secondary outcomes, funding and potential conflicts of interest. If a disagreement arose, a third investigator made the final decision (GV).

Statistical analysis

Statistical analysis was conducted using Review Manager 5.3.22 and Comprehensive Meta-analysis. In order to account for anticipated between-study heterogeneity, we employed random-effects models in our meta-analysis of continuous outcomes. We reported the effect sizes as weighted mean differences (MD) with 95% confidence intervals (CI) for trials with similar results. The I 2 statistics were used to assess the statistical heterogeneity of the pooled estimates. While recognizing that statistical heterogeneity may not be significant when I 2 is < 40%, we performed this test. Regrettably, due to the limited number of included papers, we were unable to carry out a subgroup analysis or funnel plot assessment of publication bias.

Studies selection

The initial literature search yielded 3174 studies. After a detailed review of the selected studies and removal of duplicate entries, 1964 articles remained. These articles were then evaluated based on their titles and abstracts to determine whether they met the inclusion criteria for our study and those that did not were excluded. A comprehensive screening of the full text was performed in the remaining 30 studies. Studies which did not meet the inclusion criteria were excluded. The final quantitative analysis included five studies and 3 studies were included in the qualitative assessment as these studies used other drugs like corticosteroids thus with different interventions. A visual representation of the PRISMA flowchart effectively illustrated the study selection process (Fig.  1 ).

Quality assessment

In assessing the quality of RCTs, we extensively utilized the Cochrane Risk of Bias tool which categorizes studies into three risk levels: high, uncertain, and low, across seven specific domains encompassing aspects of selection, comparability, and outcome. Following rigorous evaluation, all studies included in our analysis were consistently classified as having low risk across these domains. A detailed presentation of the Risk of Bias assessment is shown in Fig.  2 .

Risk of bias Assessment ( A ) Qualitative ( B ) Quantitative

Study and patient characteristics

All the included studies assessed outcomes in patients aged 18–75 years. The intervention group in three studies [ 19 , 20 , 21 ] used bupivacaine 0.5% 1.5 ml with or without 1 ml of saline (0.9%); one study [ 22 ] used lidocaine 2% 1 ml with 1 ml of saline solution (0.9%); and lastly, one study [ 23 ] used lidocaine 2% 2 ml as the interventional group. In the control groups, a saline solution of 0.9% (1.5, 2, or 2.5 ml) was used as a placebo. A total of 268 patients were included in all studies, ranging in age from 18 to 75 years. The studies differed in their follow-up procedures. Two studies were followed up at 4 weeks, one study was followed up for up to 2 months, and two studies were checked every month for up to 3 months A summary of patients’ baseline characteristics is provided in Table  1 .

Effect of GONB on headache intensity

In the initial month following GONB treatment, the meta-analysis of three studies showed a significant reduction in headache intensity as measured by the Visual Analog Scale (VAS). The standardized mean difference (SMD) was − 0.653, with a 95% confidence interval (CI) of -0.996 to -0.311 and a p-value of 0.0001. This indicates that the local anesthetic group experienced a greater reduction in headache intensity compared to the placebo group. Importantly, the I² value of 0% suggests that there was no observed heterogeneity among the studies, indicating consistent results across the studies analyzed. (Fig.  3 )

Forest plot illustrating the effect of GONB on headache intensity, evaluated using VAS within the initial month

In the second month, an analysis of five studies continued to show a significant reduction in headache intensity with an SMD of -0.628 (95% CI -1.148 to -0.107; p  = 0.018). However, the I² value increased to 74%, indicating substantial heterogeneity among the studies. This heterogeneity was primarily due to one study (Inan et al.), which had an outlier SMD of 0.136. (Fig.  4 ) A leave-one-out analysis was conducted to address this issue and is shown in Fig.  5 .

Forest plot illustrating the impact of GONB on headache intensity, evaluated using VAS during the second month

Forest plot illustrating the effect of GONB on headache frequency within the initial month

Headache frequency

Within the initial month, the analysis of two studies showed a significant reduction in headache frequency, with an SMD of -0.755 (95% CI -1.133 to -0.377; p  = 0.0001). The results indicate a notable decrease in headache frequency in the local anesthetic group compared to the placebo group. The I² value of 0% indicates no heterogeneity between the studies, suggesting that the results were consistent. (Fig.  6 )

Forest plot illustrating the impact of GONB on headache frequency during the second month

At the two-month mark, the analysis of four studies also showed a significant reduction in headache frequency with an SMD of -0.577 (95% CI -0.887 to -0.266; p  = 0.0001). The low I² value of 8.9% indicates minimal heterogeneity among the studies, reinforcing the consistency of the observed effect (Fig.  7 ).

Forest plot displaying adverse events associated with the use of GONB

Adverse events

The meta-analysis of two studies on adverse events revealed no significant difference between the GONB treatment and placebo groups. The odds ratio (OR) was 1.379 with a 95% CI of 0.599 to 3.177 and a p-value of 0.450. The confidence interval crosses one, indicating that there is no clear increased risk of adverse events associated with GONB treatment. Additionally, the I² value of 0% suggests no heterogeneity between the studies, indicating consistent findings regarding the safety profile of GONB (Fig.  7 ).

We conducted an updated meta-analysis of GONB in patients with CM, incorporating findings from five RCTs. All RCTs used local anesthetics for GONB, while 0.9% saline served as the placebo. Our study focused on evaluating the impact of GONB on headache frequency, intensity, and associated adverse effects. The results demonstrated the beneficial effects of local anesthetics in reducing both the frequency and intensity of headaches during the first and second months of treatment. However, the outcomes related to adverse effects did not reach statistical significance. This meta-analysis included studies employing two distinct local anesthetics: 0.5% bupivacaine and 2% lidocaine. This suggests that the use of any local anesthetic could yield positive outcomes when compared with the effects of a placebo. Despite the positive results observed, we approached the evidence with caution because of the assessment of low certainty. Therefore, additional studies are warranted to further substantiate our findings and to enhance the reliability of the conclusions drawn from our meta-analysis.

Our meta-analysis demonstrated that GONB treatment significantly reduces both headache intensity and frequency in the initial and subsequent months post-treatment compared to placebo. During the first month, the studies consistently showed a marked reduction in headache intensity with no observed heterogeneity, indicating uniform results across the studies analyzed. In the second month, while the reduction in headache intensity remained remarkable, some heterogeneity was noted due to an outlier study. Similarly, the analysis revealed a notable decrease in headache frequency within the first month, again with consistent findings and no heterogeneity between the studies. By the second month, the reduction in headache frequency continued to be noteworthy, with minimal heterogeneity observed, reinforcing the consistency of the treatment effect. Furthermore, the analysis of adverse events indicated no significant difference between the GONB treatment and placebo groups, suggesting that GONB does not increase the risk of adverse events. The studies consistently supported the safety profile of GONB, with no observed heterogeneity. In terms of both safety and efficacy, our findings suggest that the use of local anesthetics in GONB is generally safe, as we did not identify any notable adverse effects in our intervention group. However, the certainty of our evidence is moderate, primarily because our results did not reach statistical significance, potentially influenced by the limited number of studies and relatively short follow-up phase. In our updated meta-analysis, building upon the original study by Velezquez et al. [ 24 ], we included an additional randomized RCT, contributing to a more comprehensive quantitative analysis. Although most of our study findings align with Velezquez’s findings [ 24 ], demonstrating the safety and effectiveness of GONB in treating chronic migraine, it is important to acknowledge some variations. Velezquez highlighted occasional negative effects associated with local anesthetics but found no remarkable side effects. In contrast, our study did not yield statistically significant outcomes in defining these results. A noteworthy distinction lies in the consideration of adjuvants: while our study did not account for steroids or other adjuvants, Velezquez considered steroids for every study outcome. This discrepancy underscores the need for further exploration and standardization of variables in future research to establish a more definitive understanding of the safety and efficacy of GONB in the management of chronic migraine.

Our findings strongly suggest that GONB is a safe and effective method for treating migraine. This assertion is consistent with existing research that characterizes GONB as a highly effective and safe therapy with minimal adverse effects, recommending its consideration when alternative treatments are unsuccessful [ 21 ]. This viewpoint is further supported by another study that affirms our findings, emphasizing a preference for GONB in cases of resistant migraine [ 22 ]. Moreover, evidence suggests the potential applicability of GONB in the treatment of various types of headaches [ 23 , 25 ]. A retrospective cohort study also indicated that GONB may be beneficial in addressing acute migraine episodes, albeit with a cautionary note regarding the potential negative effects occurring during the procedure rather than during the follow-up period [ 26 ]. Additional observational studies [ 25 , 27 ] reinforce our findings. However, a study comparing the effectiveness of GONB with placebo in preventing migraine revealed that while there was no marked change in headache frequency, GONB still played a remarkable role in lowering intensity [ 28 ]. Notably, these studies underscored the benefits of GONB, often involving the adjunct use of steroids. In a randomized controlled trial that focused on patients treated with bilateral GONB, the results indicated that the administration of a local anesthetic was associated with lower frequency, reduced intensity, and increased pressure thresholds. However, it is important to note that this study predominantly involved female participants [ 29 ]. However, it is essential to acknowledge that trials exclusively assessing the independent use of local anesthetics in GONB are currently lacking, as steroids are commonly employed as adjuvants in the majority of studies. This finding suggests the need for further investigation to delineate the unique contributions of local anesthetics to GONB outcomes.

Prior research has emphasized the necessity of comparing various treatment plans for GONB, incorporating diverse anesthetics and adjuncts to comprehensively evaluate its effectiveness, the need for additional intervention, and safety considerations, it is crucial to note that we did not incorporate any adjuncts, preventing us from commenting on their potential impact on the treatment outcomes. The absence of adjunct utilization in our study underscores the need for further exploration of how these additions may influence the overall efficacy and safety of GONB. Most trials in our analysis used weekly injections, resulting in a lack of comprehensive data for comparing various frequencies. Nevertheless, some studies have suggested the potential advantages associated with monthly use [ 26 ]. The American Headache Society also suggests and has shown interest in the efficacy of nerve blocks for headache treatment. Their endorsement highlights the growing recognition of nerve blocks as a valuable therapeutic option for managing headaches [ 30 , 31 ].

Included studies present diverse methodologies in terms of dosage, injection sites, duration and timing of the intervention, and primary endpoints for the evaluation of GONB efficacy in migraine treatment. The administration and makeup of the GONB differed substantially across the studies. For example, Gul et al. [ 20 ] used 0.5% bupivacaine diluted in 1 ml, while Inan et al. [ 19 ] used a slightly larger volume of the same concentration. Ozer et al. [ 22 ] combined 2% lidocaine with saline, and Ashkenazi et al. [ 32 ] mixed lidocaine and bupivacaine. These variations could lead to differences in efficacy and side effects. The addition of corticosteroids, as observed in Dilli et al. [ 33 ], introduces another variable that may enhance the anti-inflammatory effects but could also influence the outcome independently of the nerve block’s anesthetic action. Although the studies targeted the GON, the exact injection sites varied slightly. Most studies, such as those by Gul et al. [ 20 ], Inan et al. [ 19 ], and Cuadrado et al. [ 34 ], selected a site approximately 2 cm lateral and 2 cm inferior to the external occipital protuberance. Palamar et al. [ 21 ] used ultrasound guidance, which might improve accuracy and potential efficacy. Ashkenazi et al. [ 32 ] included additional trigger point injections (TPIs), which could complicate the specific effects of the GONB.

The administration of GONB varied in frequency and duration among different studies. While some research, such as that conducted by Gul et al. [ 20 ] and Inan et al. [ 19 ], administered the blocks weekly for four weeks, others like Chowdury et al. [ 23 ] extended the injections over a period of 12 weeks. On the other hand, Cuadrado et al. [ 34 ] and Dilli et al. [ 33 ] examined single-time administrations. These discrepancies in timing may affect both short-term and long-term outcomes, with more frequent administrations potentially leading to more sustained relief, but also increasing the risks of cumulative side effects. The primary endpoints of the studies varied but generally included measures of headache frequency and intensity. For instance, Gul et al. [ 20 ] and Palamar et al. [ 21 ] focused on the number of headache days per month, while Inan et al. assessed both frequency and intensity. Ozer et al. [ 22 ] and Cuadrado et al. [ 34 ] emphasized the reduction in headache frequency, while Dilli et al. [ 33 ] sought a 50% reduction in migraine frequency as a measure of success. The variation in endpoints underscores the multifaceted nature of migraine impact and the significance of selecting appropriate, consistent measures for evaluating the efficacy of treatments.

Despite the differences in methodology, the studies collectively indicate that GONB can effectively decrease the frequency and severity of migraines. The consistent reporting of substantial improvements across a range of dosages, injection techniques, and primary outcomes reinforces the potential usefulness of GONB in clinical practice. However, the variation in methodologies highlights the need for standardized protocols to improve the comparability and generalizability of the findings. While the reviewed studies indicate promising outcomes for GONB in migraine treatment, the variability in dosage, injection sites, administration timing, and primary endpoints necessitates caution.

Examining these frequencies is particularly vital because of the invasive nature of the procedure, which offers valuable insights into its safety profile. An essential aspect of chronic migraine management is patient adherence, which markedly contributes to treatment success. It is imperative to assess the level of adherence to GONB. Unfortunately, we could not find relevant research on participants discontinuing their medication owing to side effects, hindering our ability to determine the tolerability of the treatment. Another unresolved concern revolves around the choice between unilateral and bilateral GONB and their relative efficacy. A retrospective cohort study comparing patients who underwent bilateral versus unilateral GONB demonstrated equal effectiveness [ 35 ]. However, a definitive conclusion remains elusive as additional evidence from diverse studies is lacking. Addressing these gaps in research would contribute substantially to refining our understanding of GONB’s optimal parameters for improved outcomes in chronic migraine management. Longitudinal studies and studies on the frequency of nerve block use are needed to assess long-term efficacy.

Limitations

Although this meta-analysis offers valuable insights, it is crucial to acknowledge its limitations. First, the small sample size resulting from the limited availability of new studies may compromise the reliability and accuracy of our findings. Although incorporating more studies could alleviate this concern, the scarcity of available data remains an issue. Second, the absence of sufficient data from recent trials prevented consideration of baseline characteristics, hindering our ability to perform meta-regression. This limitation underscores the importance of comprehensive data collection in future studies to increase the depth of our analyses. Third, oversight of not accounting for pretreatment medications taken by patients during the procedure might introduce a confounding factor. Although the existing data may be insufficient to draw definitive conclusions, recognizing and addressing this aspect in future research is essential for a more nuanced understanding. Moreover, this meta-analysis did not explicitly address patient comorbidities. These factors could potentially influence the safety of the procedure in patients with various comorbidities. Future studies should delve into these aspects to provide a more comprehensive assessment of the safety profile of the procedure in diverse patient populations. In conclusion, although this meta-analysis provides valuable insights, researchers must remain cognizant of these limitations. Addressing these concerns in future studies will enhance the robustness and applicability of these findings in clinical settings.

Based on our investigation, we ascertained that the administration of Greater Occipital Nerve Blocks (GONB) with local anesthetic leads to a notable reduction in both the intensity and frequency of headaches when compared to placebo. Additionally, our research underscores the effectiveness of GONBs and affirms their satisfactory safety profile. However, it is important to acknowledge that our confidence in these findings is somewhat tempered by the limited number of studies and relatively modest sample size that underpins our conclusions. Therefore, we advocate that future studies should broaden their scope by incorporating larger and more diverse sample sizes. These studies should also explore a range of drugs, frequencies, and treatment plans to augment the robustness and applicability of the results, thereby providing a more comprehensive understanding of the potential benefits of GONBs for headache management.

Data availability

The data are available within the article and supplementary files. The authors confirm that data supporting the findings of this study are available in the article and supplementary files.

Steiner TJ, Stovner LJ, Jensen R, Uluduz D, Katsarava Z. Migraine remains second among the world’s causes of disability, and first among young women: findings from GBD2019. J Headache Pain. 2020;21(1):137.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Steiner TJ, Stovner LJ. Global epidemiology of migraine and its implications for public health and health policy. Nat Rev Neurol. 2023;19(2):109–17.

Article   PubMed   Google Scholar  

Stovner LJ, Nichols E, Steiner TJ, Abd-Allah F, Abdelalim A, Al-Raddadi RM, et al. Global, regional, and national burden of migraine and tension-type headache, 1990–2016: a systematic analysis for the global burden of Disease Study 2016. Lancet Neurol. 2018;17(11):954–76.

Article   Google Scholar  

Ferrari MD. The economic burden of migraine to society. PharmacoEconomics. 1998;13(6):667–76.

Article   PubMed   CAS   Google Scholar  

Goadsby PJ, Holland PR. An update: pathophysiology of Migraine. Neurol Clin. 2019;37(4):651–71.

The International Classification of Headache Disorders, 3rd edition (beta version). Cephalalgia. 2013;33(9):629–808.

Urits I, Yilmaz M, Bahrun E, Merley C, Scoon L, Lassiter G, et al. Utilization of B12 for the treatment of chronic migraine. Best Pract Res Clin Anaesthesiol. 2020;34(3):479–91.

Hong JB, Lange KS, Overeem LH, Triller P, Raffaelli B, Reuter U. A scoping review and Meta-analysis of Anti-CGRP monoclonal antibodies: Predicting Response. Pharmaceuticals (Basel). 2023;16(7).

Han L, Liu Y, Xiong H, Hong P. CGRP monoclonal antibody for preventive treatment of chronic migraine: an update of meta-analysis. Brain Behav. 2019;9(2):e01215.

Article   PubMed   PubMed Central   Google Scholar  

Al-Khazali HM, Krøll LS, Ashina H, Melo-Carrillo A, Burstein R, Amin FM et al. Neck pain and headache: Pathophysiology, treatments and future directions. Musculoskeletal Science and Practice [Internet]. 2023;66:102804. https://doi.org/10.1016/j.msksp.2023.102804

Bartsch T, Goadsby PJ. The trigeminocervical complex and migraine: current concepts and synthesis. Curr Sci Inc. 2003;7:371–6. https://doi.org/10.1007/s11916-003-0036-y .

Article   CAS   Google Scholar  

Chowdhury D, Datta D, Mundra A. Role of Greater Occipital nerve Block in Headache disorders: a narrative review. Neurol India. 2021;69(Supplement):S228–56.

Anthony M. Headache and the greater occipital nerve. Clin Neurol Neurosurg. 1992;94(4):297–301.

Selekler MH. [Greater occipital nerve blockade: trigeminicervical system and clinical applications in primary headaches]. Agri. 2008;20(3):6–13.

PubMed   Google Scholar  

Shauly O, Gould DJ, Sahai-Srivastava S, Patel KM. Greater Occipital nerve block for the treatment of chronic migraine headaches: a systematic review and Meta-analysis. Plast Reconstr Surg. 2019;144(4):943–52.

Tang Y, Kang J, Zhang Y, Zhang X. Influence of greater occipital nerve block on pain severity in migraine patients: a systematic review and meta-analysis. Am J Emerg Med. 2017;35(11):1750–4.

Zhang H, Yang X, Lin Y, Chen L, Ye H. The efficacy of greater occipital nerve block for the treatment of migraine: a systematic review and meta-analysis. Clin Neurol Neurosurg. 2018;165:129–33.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Inan LE, Inan N, Karadaş Ö, Gül HL, Erdemoğlu AK, Türkel Y, et al. Greater occipital nerve blockade for the treatment of chronic migraine: a randomized, multicenter, double-blind, and placebo-controlled study. Acta Neurol Scand. 2015;132(4):270–7.

Gul HL, Ozon AO, Karadas O, Koc G, Inan LE. The efficacy of greater occipital nerve blockade in chronic migraine: a placebo-controlled study. Acta Neurol Scand. 2017;136(2):138–44.

Palamar D, Uluduz D, Saip S, Erden G, Unalan H, Akarirmak U. Ultrasound-guided greater occipital nerve block: an efficient technique in chronic refractory migraine without aura? Pain Physician. 2015;18(2):153–62.

Özer D, Bölük C, Türk Börü Ü, Altun D, Taşdemir M, Köseoğlu Toksoy C. Greater occipital and supraorbital nerve blockade for the preventive treatment of migraine: a single-blind, randomized, placebo-controlled study. Curr Med Res Opin. 2019;35(5):909–15.

Chowdhury D, Tomar A, Deorari V, Duggal A, Krishnan A, Koul A. Greater occipital nerve blockade for the preventive treatment of chronic migraine: a randomized double-blind placebo-controlled study. Cephalalgia. 2023;43(2):3331024221143541.

Velásquez-Rimachi V, Chachaima-Mar J, Cárdenas-Baltazar EC, Loayza-Vidalon A, Morán-Mariños C, Pacheco-Barrios K, et al. Greater occipital nerve block for chronic migraine patients: a meta-analysis. Acta Neurol Scand. 2022;146(2):101–14.

Bovim G, Sand T. Cervicogenic headache, migraine without aura and tension-type headache. Diagnostic blockade of greater occipital and supra-orbital nerves. Pain. 1992;51(1):43–8.

Allen SM, Mookadam F, Cha SS, Freeman JA, Starling AJ, Mookadam M. Greater Occipital nerve block for Acute Treatment of Migraine Headache: a large Retrospective Cohort Study. J Am Board Fam Med. 2018;31(2):211–8.

Austin M, Hinson MR. Occipital nerve Block. StatPearls. Treasure Island (FL) ineligible companies. Disclosure: Melissa Hinson declares no relevant financial relationships with ineligible companies.: StatPearls Publishing Copyright © 2024. StatPearls Publishing LLC.; 2024.

Afridi SK, Shields KG, Bhola R, Goadsby PJ. Greater occipital nerve injection in primary headache syndromes–prolonged effects from a single injection. Pain. 2006;122(1–2):126–9.

Santos Lasaosa S, Cuadrado Pérez ML, Guerrero Peral AL, Huerta Villanueva M, Porta-Etessam J, Pozo-Rosich P, et al. Consensus recommendations for anaesthetic peripheral nerve block. Neurologia. 2017;32(5):316–30.

Rothrock JF. Occfipital nerve blocks. Headache the Journal of Head and Face Pain [Internet]. 2010;50(5):917–8. https://doi.org/10.1111/j.1526-4610.2010.01668.x

Blumenfeld A, Ashkenazi A, Napchan U, Bender SD, Klein BC, Berliner R et al. Expert Consensus Recommendations for the performance of Peripheral nerve blocks for headaches – A Narrative review. Headache the Journal of Head and Face Pain [Internet]. 2013;53(3):437–46. https://doi.org/10.1111/head.12053

Ashkenazi A, Matro R, Shaw JW, Abbas MA, Silberstein SD. Greater occipital nerve block using local anaesthetics alone or with triamcinolone for transformed migraine: a randomised comparative study. J Neurol Neurosurg Psychiatry. 2008;79(4):415–7.

Dilli E, Halker R, Vargas B, Hentz J, Radam T, Rogers R, et al. Occipital nerve block for the short-term preventive treatment of migraine: a randomized, double-blinded, placebo-controlled study. Cephalalgia. 2015;35(11):959–68.

Cuadrado ML, Aledo-Serrano Á, Navarro P, López-Ruiz P, Fernández-de-Las-Peñas C, González-Suárez I, et al. Short-term effects of greater occipital nerve blocks in chronic migraine: a double-blind, randomised, placebo-controlled clinical trial. Cephalalgia. 2017;37(9):864–72.

Karaoğlan M, Durmuş İE, Küçükçay B, Takmaz SA, İnan LE. Comparison of the clinical efficacy of bilateral and unilateral GON blockade at the C2 level in chronic migraine. Neurol Sci. 2022;43(5):3297–303.

Download references

Acknowledgements

We would like to thank the Paolo Procacci Foundation for their support.

The study was funded by the Paolo Procacci Foundation.

Author information

Authors and affiliations.

Department of Medicine, Jinnah Sindh Medical University, Karachi, 75510, Pakistan

Muhamad Saqlain Mustafa, Muhammad Ashir Shafique, Burhanudin Sohail Rangwala & Abdul Haseeb

Department of Medicine, Shaheed Mohtarma Benazir Bhutto Medical College, Karachi, 75400, Pakistan

Shafin bin Amin, Aashish Kumar, Syed Ali Arsal & Adarsh Raja

Department of Medicine, Dow University of Health Science, Karachi, 74200, Pakistan

Syeda Mahrukh Fatima Zaidi, Muhammad Faheem Iqbal & Inshal Jawed

Fondazione Paolo Procacci, Roma, 00193, Italy

Giustino Varrassi

Faculty of Medicine, University of Khartoum, Khartoum, 11111, Sudan

Khabab Abbasher Hussien Mohamed Ahmed

Department of Medicine, Liaquat National Hospital and Medical College, Karachi, Pakistan

Syed Muhammad Sinaan Ali

Jinnah Sindh Medical University, Rafiqi H J Shaheed Road, Karachi, 75510, Pakistan

Shaheed Mohtarma Benazir Bhutto Medical College, Lyari Hospital Rd, Rangiwara Karachi, Karachi, 75400, Pakistan

Dow University of Health Sciences, Mission Rd, New Labour Colony Nanakwara, Karachi, 74200, Pakistan

Liaquat National Hospital & Medical College, Stadium Road, Karachi, 74800, Pakistan

You can also search for this author in PubMed   Google Scholar

Contributions

M.S.M., S.B.A., A.K., M.A.S., S.M.F.Z., S.A.A. and B.S.R. wrote the main manuscript, visualized, validated and analyzed data. M.F.I., A.R., A.H., I.J., K.A.H.M.A., S.M.S.A. and G.V. wrote the main manuscript, conceived, visualized, validated, reviewed and edited.All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Corresponding author

Correspondence to Khabab Abbasher Hussien Mohamed Ahmed .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mustafa, M.S., bin Amin, S., Kumar, A. et al. Assessing the effectiveness of greater occipital nerve block in chronic migraine: a systematic review and meta-analysis. BMC Neurol 24 , 330 (2024). https://doi.org/10.1186/s12883-024-03834-6

Download citation

Received : 19 April 2024

Accepted : 28 August 2024

Published : 07 September 2024

DOI : https://doi.org/10.1186/s12883-024-03834-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Chronic migraine
• Greater occipital nerve block (GONB)
• Local anesthetics

BMC Neurology

ISSN: 1471-2377

• General enquiries: [email protected]

About Systematic Reviews

Are Systematic Reviews Qualitative or Quantitative?

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

A systematic review is designed to be transparent and replicable. Therefore, systematic reviews are considered reliable tools in scientific research and clinical practice. They synthesize the results using multiple primary studies by using strategies that minimize bias and random errors. Depending on the research question and the objectives of the research, the reviews can either be qualitative or quantitative. Qualitative reviews deal with understanding concepts, thoughts, or experiences. Quantitative reviews are employed when researchers want to test or confirm a hypothesis or theory. Let’s look at some of the differences between these two types of reviews.

To learn more about how long it takes to do a systematic review , you can check out the link to our full article on the topic.

Differences between Qualitative and Quantitative Reviews

The differences lie in the scope of the research, the methodology followed, and the type of questions they attempt to answer. Some of these differences include:

Research Questions

As mentioned earlier qualitative reviews attempt to answer open-ended research questions to understand or formulate hypotheses. This type of research is used to gather in-depth insights into new topics. Quantitative reviews, on the other hand, test or confirm existing hypotheses. This type of research is used to establish generalizable facts about a topic.

Type of Sample Data

The data collected for both types of research differ significantly. For qualitative research, data is collected as words using observations, interviews, and interactions with study subjects or from literature reviews. Quantitative studies collect data as numbers, usually from a larger sample size.

Data Collection Methods

To collect data as words for a qualitative study, researchers can employ tools such as interviews, recorded observations, focused groups, videos, or by collecting literature reviews on the same subject. For quantitative studies, data from primary sources is collected as numbers using rating scales and counting frequencies. The data for these studies can also be collected as measurements of variables from a well-designed experiment carried out under pre-defined, monitored conditions.

Data Analysis Methods

Data by itself cannot prove or demonstrate anything unless it is analyzed. Qualitative data is more challenging to analyze than quantitative data. A few different approaches to analyzing qualitative data include content analysis, thematic analysis, and discourse analysis. The goal of all of these approaches is to carefully analyze textual data to identify patterns, themes, and the meaning of words or phrases.

Quantitative data, since it is in the form of numbers, is analyzed using simple math or statistical methods. There are several software programs that can be used for mathematical and statistical analysis of numerical data.

Presentation of Results

Learn more about distillersr.

(Article continues below)

Final Takeaway – Qualitative or Quantitative?

3 reasons to connect.

• Open access
• Published: 09 September 2024

A quantitative content analysis of topical characteristics of the online COVID-19 infodemic in the United States and Japan

• Matthew Seah 1 &
• Miho Iwakuma 1  

BMC Public Health volume  24 , Article number:  2447 ( 2024 ) Cite this article

Metrics details

The COVID-19 pandemic has spurred the growth of a global infodemic. In order to combat the COVID-19 infodemic, it is necessary to understand what kinds of misinformation are spreading. Furthermore, various local factors influence how the infodemic manifests in different countries. Therefore, understanding how and why infodemics differ between countries is a matter of interest for public health. This study aims to elucidate and compare the types of COVID-19 misinformation produced from the infodemic in the US and Japan.

COVID-19 fact-checking articles were obtained from the two largest publishers of fact-checking articles in each language. 1,743 US articles and 148 Japanese articles in their respective languages were gathered, with articles published between 23 January 2020 and 4 November 2022. Articles were analyzed using the free text mining software KH Coder. Exploration of frequently-occurring words and groups of related words was carried out. Based on agglomeration plots and prior research, eight categories of misinformation were created. Lastly, coding rules were created for these eight categories, and a chi-squared test was performed to compare the two datasets.

Overall, the most frequent words in both languages were related to health-related terms, but the Japan dataset had more words referring to foreign countries. Among the eight categories, differences with chi-squared p  ≤ 0.01 were found after Holm-Bonferroni p value adjustment for the proportions of misinformation regarding statistics (US 40.0% vs. JP 25.7%, ϕ 0.0792); origin of the virus and resultant discrimination (US 7.0% vs. JP 20.3%, ϕ 0.1311); and COVID-19 disease severity, treatment, or testing (US 32.6% vs. JP 45.9%, ϕ 0.0756).

Conclusions

Local contextual factors were found that likely influenced the infodemic in both countries; representations of these factors include societal polarization in the US and the HPV vaccine scare in Japan. It is possible that Japan’s relative resistance to misinformation affects the kinds of misinformation consumed, directing attention away from conspiracy theories and towards health-related issues. However, more studies need to be done to verify whether misinformation resistance affects misinformation consumption patterns this way.

Peer Review reports

Introduction

The COVID-19 pandemic has brought into the spotlight the growing infodemic : the “excessive amount of unfiltered information concerning a problem such that the solution is made more difficult” [ 1 ]. Between the mainstream media, statements made by politicians, social media platforms, instant messaging services, and changing guidelines released by official institutions, the typical person is constantly inundated with a barrage of information that presents both the challenge of discerning reliable information, as well as the option to take fringe or pseudoscientific theories as the truth. This represents a public health concern, as COVID-19 misinformation or “fake news” may spread anti-vaccine views or promote racial discrimination [ 2 ].

A multi-pronged approach is necessary to mitigate the impact of the infodemic, as no single intervention can achieve the breadth required to match the scale of the worldwide flow of information. Eysenbach proposes four pillars of infodemic management in his 2020 paper: infoveillance and infodemiology (surveillance of information supply and demand, as well as its quality); building eHealth literacy; improving the translation of knowledge between academia and larger outlets such as policymakers, mainstream media, and social media; and the peer-review process and fact-checking [ 3 ].

“Fact-checking” refers to the process of evaluating a statement for its factual accuracy or whether it has been framed in a misleading manner due to omission of context. Fact-checking has its origins in American TV segments devoted to checking the accuracy of statements made by American presidential candidates [ 4 ], though most current fact-checking content is produced by websites such as Snopes or FactCheck.org in the form of articles or videos.

Fact-checking alone cannot be the ultimate counter to misinformation – not only does it have limited effects on correcting perceptions of misinformation due to the strong biases and emotions involved when interacting with such information [ 4 , 5 ], the local politics of truth [ 6 ], i.e. the historical and cultural contexts of the region, inform behavior and beliefs to a significant degree; for instance, close-contact burial practices in parts of west Africa stricken by ebola [ 7 ], or vaccine hesitancy in Japan following the HPV vaccine scare in 2013 [ 8 ]. Interventions targeting an infodemic need to take into account the nature and context of the region to be effective.

One of the few extant studies comparing the COVID-19 infodemics and national contexts across countries was published by Zeng et al. [ 9 ], in which they analyzed fact-checking article contents from the US, China, India, Germany, and France. Some key findings included the fact that non-health misinformation (e.g. regarding politics, or the origin of the virus) is nearly twice as common as health misinformation (e.g. COVID-19 being “just a cold”); Germany is relatively resilient to misinformation compared to the US or India owing to its low societal polarization and high trust in the news media; misinformation regarding the spread of COVID-19 or travel restrictions is common in China, likely due to China being the early epicenter of the pandemic as well as large-scale travel movements that occur around Chinese New Year; and wedge-driving misinformation along religious lines is common in India owing to the longstanding conflict between the nation’s Muslim and Hindu populations.

Although there is already an abundance of cross-cultural research between the US and Japan, a comparative study of infodemics in these countries has yet to be done, and much has changed in the time since the publication of the Zeng paper – noteworthy developments including the progress made in global vaccination campaigns [ 10 ], and the emergence of the highly transmissible delta and omicron variants [ 11 ]. Furthermore, the national contexts of the US and Japan differ to a notable extent, in geographical, sociocultural, and historical terms, making it reasonable to expect differences in the types of misinformation that would gather more traction. Therefore, this research aims to provide an updated understanding of the COVID-19 infodemics in the US and Japan through a quantitative content analysis of the types of misinformation that appear in fact-checking articles.

Methodology

Data selection and gathering.

In order to find the types of COVID-19 misinformation that gathered significant traction in the US and Japan, COVID-19 fact-checking articles were gathered from the top two largest fact-checking publishers: Politifact and FactCheck.org for the US, and Buzzfeed and InFact for Japan. All articles were written in their respective countries’ languages (English for the US, Japanese for Japan). A summary of the data sources used is shown in Table  1 below. Articles included were published between 23 January 2020 and 4 November 2022.

Article URLs were scraped from the COVID-19 sections of each source in Python, using the Selenium library in Chrome 108.0.5359.124. Following this, a separate program was used to visit the listed URLs and scrape the article contents using the news-please library [ 16 ]. (Source codes can be accessed at https://github.com/seahmatthew/KyotoU-PublicHealth2023 .)

Data analysis in KH coder

The open-source quantitative text analysis program KH Coder [ 17 ], developed by Koichi Higuchi at Ritsumeikan university, was used to analyze the article contents, with the US and Japan datasets in separate projects. As of January 2023, there are 5,761 published research articles which make use of KH Coder [ 18 ], many of which cover health-related research topics. Its strengths include functions for statistical analysis (e.g., term frequency) of large data files, as well as the KWIC Concordance function [ 19 ] which provides the capability to easily refer to the original data from any given result.

Word Frequency [ 19 ] was used to obtain an overview of the data as a preliminary step. Following this, Hierarchal Cluster Analysis [ 19 ] was used to explore groups of related words, and also to build the lists of terms to force pickup (such as “toilet paper” or “Moderna”) which would not be picked up by default, and irrelevant terms to force ignore (such as “website” or “article”), which introduce noise due to appearing very frequently but not being indicative of any relevant themes. This took a process of trial and error especially when building the force ignore lists, as blocking certain seemingly irrelevant terms would sometimes turn out to hide an otherwise useable article.

After substantive force pickup/ignore lists had been built for each languages, the lists were compared to ensure that relevant keywords were ignored in both languages, although words that appear frequently as syntactic features in each language (such as “pants [on] fire” or “subject”) were not duplicated in the same way.

Next, Hierarchal Cluster Analysis was re-run using the finalized force pickup/ignore lists to gather the terms to form the document coding files. For the U.S. dataset, the minimum Term Frequency (TF) was set to 90, Document Frequency (DF) to 1, and only nouns, proper nouns, and terms from the force pickup list were analyzed to minimize noise. For the Japan dataset, the minimum TF was set to 10, DF to 1, and only nouns, proper nouns, location names, and terms from the force pickup list were analyzed. For both datasets, the Ward method and Jaccard frequency were used, with the number of clusters shown being auto-chosen.

Based on the agglomeration plot turning points from the Hierarchal Cluster analyses, the prior Zeng paper [ 9 ], and familiarity with the data, it was decided to split the data into eight categories. From the categories and keywords found, coding files were built for the US and Japan datasets and applied to obtain the frequencies for each category. Articles could be assigned to multiple categories, and manual sorting was used to classify articles through a first pass after automatic sorting. Articles that failed to be classified in any category after both automatic and manual sorting were assigned to a separate Miscellaneous category.

After the code frequencies for each language had been obtained, chi-squared tests were carried out to test whether there were differences in the frequencies across countries. Holm-Bonferroni adjustment was used to adjust the p values.

The agglomeration plots produced from the Hierarchal Cluster analyses are shown below in Fig.  1 . The turning points show that somewhere in the range of seven categories would be ideal, but considering prior research and familiarity with the data, it was decided to generate eight categories.

Agglomeration plots produced by Hierarchal Cluster Analysis of the US (left) and Japan (right) datasets

The coding files created based on the categories and keywords found are shown in Table  2 . A total of eight categories were created: government policy; resource shortages; statistics; measures to stem the spread of infection; masks and transmission; origin of the virus and resultant discrimination; COVID-19 disease severity, treatment, or testing; and vaccine efficacy, contents, or safety. Each category contains a set of keywords in its respective language that results in close association; for instance, “lockdown”, “quarantine”, and “border” associate highly with articles about measures taken to stem the spread of infection.

A summary of the top 50 words with the highest tf (term frequency) is shown in Table  3 . Both the U.S. and Japan lists are topped by words pertaining to vaccination, masks, cases and testing, likely because these words are likely to appear across a broad range of categories. For instance, words pertaining to vaccination could appear in both articles about supposed deleterious health effects of vaccination, as well as articles about vaccination program plans or vaccine-related conspiracy theories.

A summary of the code frequencies, chi-squared test p values, and relevant excerpts from the data is provided below in Table  4 . Articles that contained none of the eight predetermined codes are grouped in the “Miscellaneous” category. Chi-squared tests were carried out to compare the code frequencies across datasets, and p value correction was done using the Holm-Bonferroni method. Three categories stood out due to their relatively low p values and relatively high effect sizes: statistics, the origin of the virus and resultant discrimination, and COVID-19 severity, treatment, and testing.

Versions of Tables  2 and 3 , and 4 with the original Japanese text are available in Supp_012024.docx.

The effect sizes ϕ for each category are shown below in Table  5 . Only the category on the origin of the virus and resultant discrimination showed an effect size exceeding 0.1, a small effect. The two categories of statistics, and COVID-19 severity, treatment, and testing showed the next-highest effect sizes of > 0.07. Hence, these three categories were chosen for further discussion.

Similarities and differences between US and Japan categories

Selective reading of articles with high tf (term frequency) for the chosen categories produced a handful of similarities and differences. Within the statistics category (which was more common in the US dataset, 40.1% vs. 25.7%, ϕ 0.0792), misinformation from both countries tended to downplay the severity of the COVID-19 mortality rate, or otherwise make factually false statistical assertions. US misinformation tended to make more (invalid) comparisons to influenza, and there were false assertions that the US was performing statistically better in terms of mortality rate than other countries, while Japanese misinformation contained more assertions that vaccines increase mortality rate. Many of the US articles in this category were based on quotes from then-President Donald Trump.

Within the category regarding the origin of the virus and resultant discrimination (which was more common in the Japan dataset, 20.3% vs. 7.0%, ϕ 0.1311), misinformation from both countries asserted that COVID-19 was artificially made in the Wuhan Institute of Virology. However, US misinformation tended to focus on federal funding for the institute, and some articles tied the origin of the pandemic to Chinese meat-eating practices. Japanese misinformation focused more on Chinese people within Japan itself, such as warning of incoming tourist swarms or Chinese nationals taking up space in hospitals.

Within the category of COVID-19 severity, treatment, or testing (which was more common in the Japan dataset, 46.0% vs. 32.6%, ϕ 0.0756), both countries had misinformation about treatments for COVID-19, as well as about testing kits. While both countries mentioned ivermectin, hydroxychloroquine and marijuana as COVID-19 treatments were exclusive to the US dataset, while green tea and hot water were exclusive to the Japan dataset. More US articles tended to downplay the severity of infection by likening it to the flu. There were pieces of misinformation in the US that stemmed from misinterpretation of test kits, while there were Japanese assertions that COVID-19 test kits are faulty or ineffective.

Overall, non-health misinformation appeared more frequently than health misinformation, echoing findings from other studies analyzing fact-checking articles [ 9 ] or social media posts [ 20 ].

In addition, while the category frequencies for masks and transmission did not appear to differ, the contents of articles in these categories showed differences: articles from the US dataset tended to be regarding misinformation on the effectiveness of masks as a means for preventing transmission, while articles from the Japan dataset tended to be on ancillary topics, such as the country of manufacture of masks, or mask shortages. Mask-wearing as a means for preventing disease transmission while sick is an established aspect of Japanese culture [ 21 ].

National contextual factors that affect misinformation consumption

As outlined above, there are some differences in the contents of the COVID-19 misinformation circulating in the US and Japan. A few of the numerous contextual factors that may have influenced these differences will be described further below.

Importantly, it should not be assumed that a cause-and-effect relationship is at play, as a myriad of factors influence consumer (and macro-level) information-seeking habits. For instance, on the micro level, there are consumer culture factors that influence patterns of consumption, such as social influences or social class [ 22 ]; on the macro level, society-level factors such as the quality of official communications can affect attitudes towards health measures [ 23 ]. Some evidence also exists to suggest that in certain countries, the demand for certain kinds of misinformation fluctuates based on the epidemic curve [ 9 ]. While a comprehensive list of every potential influencing factor would be beyond the scope of this research, it can be seen that local context can indeed influence information-seeking habits. Understanding the concerns and mindsets of those grappling with the infodemic should be a priority in determining what countermeasures to take (e.g., targeted messaging, rapid response, etc.).

On the topic of the high prevalence of political figures involved in US misinformation, a survey conducted by the Reuters Institute for the Study of Journalism in 2020 [ 24 ] found that American information-seeking habits surrounding COVID-19 are strongly tied to political affiliation. Left-leaning respondents were likely to trust the news media and unlikely to trust the government; the opposite was true for right-leaning participants. Trump was himself a major direct source of COVID-19 misinformation [ 25 ], and many of the erroneous claims he made are reflected in the data, especially in the Statistics and Origin categories. The significant sway a person’s political beliefs hold over their information-seeking behavior in the US is likely to be associated with the country’s highly polarized political climate. This finding of the high frequency of misinformation from politicians in the US is echoed in the Zeng paper [ 9 ], and the same paper found that this connection between societal polarization and political misinformation was also clear in India.

In the Japanese dataset, articles pertaining to the origin of COVID-19 from China were much more frequent and pointed in general; as opposed to US articles which mostly addressed conspiracy theories of American funding for the Wuhan Institute of Virology or the animal origins of the virus, articles in this category in the Japan dataset tended to focus directly on Chinese nationals, either as disproportionate occupants of Japanese medical institutions, or as spreaders of COVID-19 inbound from China. Japan’s relative geographical proximity to China and popularity as a Chinese tourist destination, as well as existing anti-Chinese sentiment that has been worsening progressively since the 1980s [ 26 ], may explain to some extent the personal nature of Japanese misinformation in this category.

At first glance, it may seem surprising that both the US and Japan have similar proportions of articles discussing vaccine efficacy, contents, or safety, especially given the heavy role US political figures played in leading supporters to act contrary to evidence-based findings [ 27 ]. In an article published in the Japanese journal Chiryo in 2021, the founders of HPV vaccine awareness group MinPapi describe how vaccine hesitancy in Japan may have been exacerbated by the human papillomavirus (HPV) vaccine side effect scare in 2013 [ 28 ]; years later, addressing vaccine hesitancy through their new website CoviNavi continues to be a challenge.

Additionally, a 2021 survey conducted in Japan showed that Japanese respondents were uncertain in general about what sources of COVID-19 information they could trust [ 20 ]. 24.7% of respondents believed there was no information source they could trust, and only 26.0% of respondents felt they could trust health experts. This stands in stark contrast to the results from the aforementioned Reuters study, where over 80% of American respondents on both sides of the political spectrum felt they could trust health experts. This difference in response to the infodemic – picking sides, as opposed to being assailed by uncertainty – may actually help to explain why vaccine misinformation is relatively common in both countries; one possible interpretation is that a limited segment of the American audience consumes vaccine misinformation in greater per capita amounts, while a more general segment of the Japanese audience consumes vaccine misinformation in lower per capita amounts.

Disinformation resilience and its effects on misinformation consumption

In a 2020 paper, Humprecht et al. outline a framework for cross-national comparisons of disinformation (henceforth “misinformation”) resilience : the degree to which online misinformation is likely to receive exposure and be spread [ 29 ]. Political factors limiting misinformation resilience include societal polarization, and frequency of populist communication; media-related factors include low trust in news media, weak public news services, and audience fragmentation; economic factors include a large advertisement market size, and high social media usage. Using this framework in a comparison of the US with 16 other mainly European countries, the authors found that the US scored the lowest in misinformation resilience, owing to its fragmented media landscape, large ad market, low trust in news, highly polarized society, and frequent populist communication.

In comparison to the US, Japan scores notably lower in terms of populist communication [ 30 ]; NHK, the public broadcasting network, attains comparable viewership to other networks [ 31 ] as opposed to American public broadcasters with one- to two-thirds the viewership of major American TV networks [ 32 , 33 ]; major TV news networks in Japan attain roughly two times the viewer share of US TV network providers, with Yahoo! News dominating the online news market with over 50% weekly usage [ 34 ]. While a formal comparison has yet to be done in the literature, these factors suggest that Japan may be more resilient to misinformation than the US. It is possible that this affected the sizes of the datasets that could be obtained, leading to the US dataset being more than ten times as large than the Japan dataset.

While it stands to reason that increased misinformation resilience would lead to lower spread and consumption of misinformation, its effect on the types of misinformation consumed is less clear. In the Zeng study [ 9 ], Germany stood out as one of the studied countries with high misinformation resilience; compared to the other countries which tended to contain high proportions of articles on political conspiracy theories, lockdown measures, or transmission methods, misinformation from Germany was centered on COVID-19 treatment and vaccines, similarly to the Japan dataset used in this report. If we consider the nature of rumors and misinformation as an answer-seeking response to a perceived external threat [ 35 ], one possible interpretation of this pattern is that increased misinformation resilience in the midst of the pandemic contributes to lower distraction with non-key issues – the key issue in this context being the health impact of COVID-19 and how it can be avoided or treated. The “Miscellaneous” category is mostly comprised of articles on these non-key issues , including those bordering on absurdity or conspiracy; while this category was not notably differently sized between the US and Japan datasets, the Japan data had a noticeably lower proportion of misinformation along the lines of the “deity of death” US article.

Strengths and limitations of this study

In comparison to prior studies which used fact-checking articles as data, this study uses a larger sample size for the US dataset and offers a Japanese dataset for the first time. In particular, using KH Coder allowed for multiple categories to be assigned to a single article, which reflects the data more accurately than other studies [ 9 ] that are limited to a single category for each article. Additionally, quantitative content analysis using KH Coder allowed for counting the term frequencies in the large datasets, as well as for referring back to the original data when needed using the KWIK Concordance function.

However, as to the limitations of the study, the span of misinformation covered in this report is limited to that selected by the editorial teams in a “gatekeeping” process [ 36 ] for the four online news sources used; in particular, fact-checking in Japan is a relatively new endeavor, with the InFact team and website notably smaller than established fact-checking organizations from the US. This has negative implications for the generalizability of the Japan data, and a larger future dataset would likely give richer results. In addition, since the categorization processes were carried out automatically, there may be a handful of data points that have not been categorized correctly. More studies should be done to further verify the relationship between the misinformation resistance of a country and the types of misinformation that spread within it. Future studies of this nature will have larger and more varied datasets to work with, whether they are about COVID-19 or any other infodemic. Finally, the effect sizes found for the sections discussed here are all of small magnitude, meaning that it should not be inferred that certain segments of misinformation should receive disproportionate amounts of focus in countries that seem vulnerable to that kind of misinformation.

Practical implications

In combination with aggregated data from other countries, data on the types of misinformation which are comparatively common in the country provides policymakers a reference point when allocating resources to tackling misinformation, through means such as rapid-response messaging [ 37 ]. Of course, this data should be weighed against the actual likely impact of said misinformation spreading in the populace; any given piece vaccine misinformation is likely to do more harm overall than a wild claim of a vaccination center bearing a logo of a “deity of death”.

This research also opens up new avenues for further research – for instance, research to verify whether modifying our taking a culturally-relevant approach to tackling misinformation results in better correction outcomes. One possible example would be altering the tone of messaging to be firmer and more succinct in an environment like Japan, where misinformation likely spreads out of uncertainty instead of certainty in misinformation, while a more indirect approach may be more effective in places like the United States where misinformed beliefs are grounded in certainty.

Using quantitative content analysis, this study shows the similarities and differences in the COVID-19 infodemics in US and Japan since the start of the pandemic. Differences were found in the proportion of articles mentioning statistics, the origin of the virus and resultant discrimination, and COVID-19 severity, treatment and testing, though the effect sizes were seen to be small.

Several facets of national context appear to support the trends seen in the data, such as the history of the HPV vaccine in Japan leading to increased distrust of COVID-19 vaccines. In addition, application of a misinformation resilience framework appears to show that in countries with higher resilience, distracting non-key issues such as conspiracy theories attract less attention compared to key issues , which refer to COVID-19 health impacts and other health information in the context of the pandemic. Understanding the types of misinformation in circulation gives policymakers and educators direction in developing strategies to counter this misinformation.

Lastly, it should be reiterated that fact-checking, even when done through appropriate channels in a culturally relevant manner, cannot be relied upon as the sole measure with which to combat an infodemic. Not only does fact-checking have heavily limited effects on correcting misinformed beliefs [ 4 , 5 ], a deluge of fact-checking information may even backfire by contributing to information overload and avoidance in the intended audience [ 38 ], or by simply acting as a dissemination channel for the misinformation that would not have been spread otherwise [ 36 ]. Fact-checking has a place as one of the pillars of infodemic management – there is a need to uphold journalistic integrity, and to provide a reliable source for a more invested, informed reader subset. The other pillars of infoveillance and infodemiology, the gradual process of building eHealth literacy in the populace, and providing clear, timely translations of scientific findings to actionable messages need to be upheld in tandem as a long-term strategy for decreasing the impact of misinformation [ 3 ].

Data availability

The dataset supporting the conclusions of this article is available in the GitHub repository, https://doi.org/10.5281/zenodo.8282744 at https://github.com/seahmatthew/KyotoU-PublicHealth2023 [ 39 ].

Abbreviations

Coronavirus disease 2019

Human papillomavirus

Eysenbach G. Infodemiology and Infoveillance: Framework for an Emerging Set of Public Health Informatics Methods To Analyze Search, communication and publication behavior on the internet. J Med Internet Res. 2009;11(1):e11.

Article   PubMed   PubMed Central   Google Scholar  

Organization WH, Managing. the COVID-19 infodemic: Promoting healthy behaviours and mitigating the harm from misinformation and disinformation 2020 [updated 23 September 2020. https://www.who.int/news/item/23-09-2020-managing-the-covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-and-disinformation

Eysenbach G. How to fight an infodemic: the four pillars of infodemic management. J Med Internet Res. 2020;22(6):e21820.

Young DG, Jamieson KH, Poulsen S, Goldring A. Fact-checking effectiveness as a function of format and tone: evaluating FactCheck.org and FlackCheck.org. Journalism Mass Communication Q. 2018;95(1):49–75.

Article   Google Scholar  

Chou W-YS, Gaysynsky A, Vanderpool RC. The COVID-19 Misinfodemic: moving beyond fact-checking. Health Educ Behav. 2021;48(1):9–13.

Article   PubMed   Google Scholar  

Foucalt M, Power/Knowledge. Selected interviews and other writings 1972–1977. Great Britain: Harvester, Limited;; 1980.

Google Scholar  

Chandler C, Fairhead J, Kelly A, Leach M, Martineau F, Mokuwa E et al. Ebola: limitations of correcting misinformation. Lancet. 2014;385(9975).

Yagi A, Ueda Y, Nakagawa S, Ikeda S, Tanaka Y, Sekine M et al. Potential for cervical cancer incidence and death resulting from Japan’s current policy of prolonged suspension of its governmental recommendation of the HPV vaccine. Sci Rep. 2020;10(1).

Zeng J, Chan C-H. A cross-national diagnosis of infodemics: comparing the topical and temporal features of misinformation around COVID-19 in China, India, the US, Germany and France. Online Inf Rev. 2021;45(4):709–28.

Verbeke R, Lentacker I, De Smedt SC, Dewitte H. The dawn of mRNA vaccines: the COVID-19 case. J Controlled Release. 2021;333:511–20.

Article   CAS   Google Scholar  

Karim SSA, Karim QA. Omicron SARS-CoV-2 variant: a new chapter in the COVID-19 pandemic. Lancet. 2021;398(10317):2126–8.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Politifact. Latest Fact-Checks In Coronavirus. https://www.politifact.com/factchecks/list/?page=1&category=coronavirus

FactCheck.org. Issues: coronavirus. https://www.factcheck.org/issue/coronavirus/

Japan B. Fact Check. https://www.buzzfeed.com/jp/badge/factcheckjp

InFact. COVID-19 FactCheck. https://infact.press/tag/covid-19-factchecks/

Hamborg F, Meuschke N, Breitinger C, Gipp B, editors. news-please: A Generic News Crawler and Extractor. 15th International Symposium of Information Science; 2017.

Higuchi K. A two-step Approach to quantitative content analysis: KH coder tutorial using Anne of Green Gables (Part I). Ritsumeikan Social Sci Rev. 2016;52(3):77–91.

Higuchi K. Research done using KH Coder 2023. https://khcoder.net/bib.html

Higuchi K. KH Coder 3 Reference Manual. 2017.

Ishibashi M, Sekiya N. Exploratory study on the spread of rumors and psychological factors related to COVID-19. Japanese J Risk Anal. 2021;31(2):123–32.

Burgess A, Horii M. Risk, ritual and health responsibilisation: Japan’s ‘safety blanket’ of surgical face mask-wearing. Sociol Health Illn. 2012;34(8):1184–98.

Hoyer WD, MacInnis DJ, Pieters R. Consumer Behavior. 6 ed. Cengage Learning; 2012.

Nara Y. The role of Risk Communication in COVID-19 Public Health Management. Infect Control: Japanese J Infect Control. 2021;30(9):958–61.

Nielsen RK, Fletcher R, Newman N, Brennen JS, Howard PN. Navigating the ‘Infodemic’: How People in Six Countries Access and Rate News and Information about Coronavirus. 2020.

Editors NEJoM. Dying in a Leadership Vacuum. N Engl J Med. 2020;383(15):1479–80.

Sekiyama T. Japan’s policy toward China under strong anti-chinese sentiment: a case of terminating yen loans to China. East Asia. 2012;29(3):295–311.

Adams A, Hackstadt A, editors. Distrust in Institutions: Reference and Library Instruction During an Infodemic. DGO2021: The 22nd Annual International Conference on Digital Government Research; 2021 2021-06-09: ACM.

Kinoshita T. How should healthcare professionals reach the apathetic populace: on my experiences running MinPapi/Cov-Navi. Chiryo. 2021;103(12):1527–31.

Humprecht E, Esser F, Van Aelst P. Resilience to online disinformation: a Framework for cross-national comparative research. Int J Press/Politics. 2020;25(3):493–516.

Hawkins KA, Aguilar R, Castanho Silva B, Jenne EK, Kocijan B, Rovira Kaltwasser C. Global populism database. 2 ed. Harvard Dataverse; 2019.

VideoResearch. Weekly Top 10 Most-Watched TV Shows: VideoResearch; 2022. https://www.videor.co.jp/tvrating/

Center PR. Public Broadcasting Fact Sheet 2021. https://www.pewresearch.org/journalism/fact-sheet/public-broadcasting/

Center PR. Cable News Fact Sheet 2021. https://www.pewresearch.org/journalism/fact-sheet/cable-news/

Newman N, Fletcher R, Robertson CT, Eddy K, Nielsen RK. Reuters Institute Digital News Report 2022. Reuters Institute; 2022.

Difonzo N, Bordia P, Rumor. Gossip Urban Legends Diogenes. 2007;54(1):19–35.

Lwin MO, Lee SY, Panchapakesan C, Tandoc E. Mainstream News Media’s role in Public Health communication during crises: Assessment of Coverage and correction of COVID-19 misinformation. Health Commun. 2021;38(1).

Scales D, Gorman J, Jamieson KH. The Covid-19 infodemic — applying the epidemiologic model to Counter Misinformation. N Engl J Med. 2021;385(8):678–81.

Article   PubMed   CAS   Google Scholar  

Soroya SH, Farooq A, Mahmood K, Isoaho J, Zara S. -e. from information seeking to information avoidance: understanding the health information behavior during a global health crisis. Inf Process Manage. 2021;58(2).

Seah M, Iwakuma M. KyotoU-PublicHealth2023-MatthewSeah. 2023.

Download references

Acknowledgements

Not applicable.

This research was supported by operating funds allocated by Kyoto University to the Department of Medical Communication.

Author information

Authors and affiliations.

Department of Medical Communication, Kyoto University, Sakyo-ku Yoshida-konoe-cho, Kyoto, Japan

Matthew Seah & Miho Iwakuma

You can also search for this author in PubMed   Google Scholar

Contributions

This paper was written in whole by MS. The manuscript was reviewed by Associate Professor MI. Both the authors read and approved the final manuscript.

Corresponding author

Correspondence to Matthew Seah .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

: This material contains the chi-squared test assumptions, Holm-Bonferroni adjusted p values used in the results, and untranslated Japanese text for Tables 2 and 3, and 4

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Seah, M., Iwakuma, M. A quantitative content analysis of topical characteristics of the online COVID-19 infodemic in the United States and Japan. BMC Public Health 24 , 2447 (2024). https://doi.org/10.1186/s12889-024-19813-y

Download citation

Received : 07 September 2023

Accepted : 16 August 2024

Published : 09 September 2024

DOI : https://doi.org/10.1186/s12889-024-19813-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Multi-country
• Health information
• Fact checking
• Myth busting
• Myth correction

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

Title: Social Media Marketing - A Systematic Review of Quantitative Empirical Studies

Authors : Bernd W. Wirtz; Isabell Balzer

Addresses : N/A ' N/A

Abstract : Purpose - The main purpose of this paper is to review the available research in the area of social media marketing and provide an overview of the research potential to identify research gaps and define specific areas for further research. Method - Through various reading and structuring loops, we examined 355 peer-reviewed quantitative empirical articles according to key themes, research objectives, research perspective, statistical method, and data collection method. Descriptive statistics were used to analyze research foci and identify research gaps. Findings - We were able to identify four research periods within social media marketing research: (1) emergence of SMM research, (2) growing interest, (3) underpinning of the theories, and (4) paradigm shift in research. We also identified the most interesting (e.g., \'success with SMM\') and underrepresented social media marketing research topics (e.g., \'B2B influence factors\'). In addition, the lack of a two-way research orientation (customer and provider) is evident and should be addressed in the future. Finally, we uncovered the need for further exploratory research in SMM. Limitations - Because we focused on quantitative empirical research, we cannot provide insights into the entire SMM research body. In addition, we cannot guarantee the completeness of the data because relevant publications may not have been retrieved due to the limitation of the search terms. Moreover, there is a potential loss of information, as we have summarized information to present it better. Implications - Future research should focus on two-sided research dealing with manager responses in SNS and customer loyalty and comparing manager responses and consumer reviews on social media performance. In addition, future research could include exploratory research questions. Research on the effects of social media marketing in a business-to-business context is also of great importance in the future. Originality - There has been no systematic review of the quantitative literature on social media marketing. Therefore, we attribute originality to our approach and results.

Keywords : Social media marketing; literature review; empirical research; quantitative research.

DOI : 10.1504/JBM.2023.141304

Journal of Business and Management, 2023 Vol.29 No.1, pp.80 - 114

Published online: 05 Sep 2024 *

Keep up-to-date

• Our Newsletter ( subscribe for free )
• New issue alerts
• Inderscience is a member of publishing organisations including:

• Study Protocol
• Open access
• Published: 09 September 2024

Strengthening facility-based integrated emergency care services for time sensitive emergencies at all levels of healthcare in India: An implementation research study protocol

• Tej Prakash Sinha 1 ,
• Sanjeev Bhoi 1 ,
• Dolly Sharma 1 ,
• Sushmita Chauhan 1 ,
• Radhika Magan 1 ,
• Ankit Kumar Sahu 1 ,
• Stuti Bhargava 2 ,
• Patanjali Dev Nayar 3 ,
• Venkatnarayan Kannan 4 ,
• Rakesh Lodha 5 ,
• Garima Kacchawa 6 ,
• Narendra Kumar Arora 7 ,
• Moji Jini 8 ,
• Pramod Kumar Sinha 9 ,
• Satyajeet Verma 10 ,
• Pawan Goyal 11 ,
• K. V. Viswanathan 12 ,
• Kemba Padu 13 ,
• Pallavi Boro 14 ,
• Yogesh Kumar 15 ,
• Pratibha Gupta 16 ,
• Srikanth Damodaran 17 &
• Nasar Jubair 18  

Health Research Policy and Systems volume  22 , Article number:  125 ( 2024 ) Cite this article

Metrics details

The healthcare system in India is tiered and has primary, secondary and tertiary levels of facilities depending on the complexity and severity of health challenges at these facilities. Evidence suggests that emergency services in the country is fragmented. This study aims to identify the barriers and facilitators of emergency care delivery for patients with time-sensitive conditions, and develop and implement a contextually relevant model, and measure its impact using implementation research outcomes.

We will study 85 healthcare facilities across five zones of the country and focus on emergency care delivery for 11 time-sensitive conditions. This implementation research will include seven phases: the preparatory phase, formative assessment, co-design of Model “Zero”, co-implementation, model optimization, end-line evaluation and consolidation phase. The “preparatory phase” will involve stakeholder meetings, approval from health authorities and the establishment of a research ecosystem. The “formative assessment” will include quantitative and qualitative evaluations of the existing healthcare facilities and personnel to identify gaps, barriers and facilitators of emergency care services for time-sensitive conditions. On the basis of the results of the formative assessment, context-specific implementation strategies will be developed through meetings with stakeholders, providers and experts. The “co-design of Model ‘Zero’” phase will help develop the initial Model “Zero”, which will be pilot tested on a small scale (co-implementation). In the “model optimization” phase, iterative feedback loops of meetings and testing various strategies will help develop and implement the final context-specific model. End-line evaluation will assess implementation research outcomes such as acceptability, adoption, fidelity and penetration. The consolidation phase will include planning for the sustenance of the interventions.

In a country such as India, where resources are scarce, this study will identify the barriers and facilitators to delivering emergency care services for time-sensitive conditions across five varied zones of the country. Stakeholder and provider participation in developing consensus-based implementation strategies, along with iterative cycles of meetings and testing, will help adapt these strategies to local needs. This approach will ensure that the developed models are practical, feasible and tailored to the specific challenges and requirements of each region.

Peer Review reports

Introduction

An emergency Care System (ECS) is an essential component of universal health coverage and for large population around the world, as it is the first point of contact with the healthcare system [ 1 ]. A World Health Organization report in 2020 demonstrated that 10 of the leading causes of death in low- and middle-income countries (LMICs) are related to emergency conditions such as heart disease, stroke, lower respiratory infections, etc. [ 2 ]. Despite the higher burden of emergency conditions, emergency services utilization is substantially lower in LMICs, possibly due to limited access to ECS [ 3 ]. The 72nd World Health Assembly on 30 May 2019 emphasized that many health interventions are time sensitive and that an integrated ECS provider is an effective solution for the delivery of accessible, quality and time-sensitive healthcare services in the case of acute illnesses and injuries throughout the life-course [ 4 ].

India is now the most populous country in the world, with one sixth of the world’s population [ 5 ]. However the ECS in the country is inadequately developed, leading to high emergency-related mortality [ 6 ]. In 2019, the National Institute for Transforming India (NITI) Aayog and All India Institute of Medical Sciences (AIIMS) New Delhi conducted a pan-India study to assess the status of emergency and injury care at government (tertiary and secondary care) and private hospitals. The study demonstrated that, although emergency and injury cases accounted for one tenth of all patients visiting the health facilities, only 3–5% beds are available for catering to emergency admissions. A dedicated emergency department (ED) was available only in half of the studied facilities, whereas a dedicated triage area was present only in one third of these facilities. Most facilities lacked trained healthcare staff who are dedicated to and trained for ED services and lacked standard operating procedures for emergency care. There were no provisions for dedicated funding, legislation and regulations within the system [ 7 ]. This led to the recommendation for “developing a robust integrated ECS system which can comprehensively address all medical and surgical emergencies,inclusive of trauma related care across all levels of healthcare facilities” [ 6 , 7 ].

Time-sensitive conditions (TSC) are critically important in ECS due to their potential for severe consequences such as high mortality and morbidity. However, the lack of a clear definition for TSCs, as highlighted in a study by Wibring et al., leads to inconsistent treatment approaches and challenges in recognizing and managing these conditions effectively [ 8 ]. The Indian Council of Medical Research (ICMR), NITI-AIIMS study, and insights from the various registries of India have identified a high mortality rate associated with these conditions in India. Consequently, the authors of this study have deliberately chosen to focus on specific TSCs including chest pain, stroke, trauma, shock, unconsciousness, respiratory distress, poisoning, snake bite, burns, post-partum haemorrhage and seizures, with a particular emphasis on paediatric emergencies, to address these critical health challenges more effectively.

Despite the existence of strong, research-backed protocols to guide the management of these time-sensitive conditions, the application of these practices to patient care remains suboptimal in developing countries including India [ 9 ]. This evidence-practice gap, that is, the gap between efficacious interventions and their application in the real-life setting, can be bridged through implementation research (IR) [ 10 ]. A scoping systematic review of implementation strategies in ECS identified 197 studies from 2000 to 2017 [ 11 ]. However, only 42% of these studies focused on identifying evidence-practice gaps, whereas 39% of these studies provided limited detail on the IR strategies. These include studies on trauma [ 12 , 13 , 14 , 15 , 16 , 17 ], sepsis [ 18 , 19 , 20 ] and cardiac emergencies [ 21 , 22 ]. Few IR studies have focused on ECS issues such as triage in ED [ 18 , 23 ]. But there is lack of studies focusing on comprehensive ECS delivery (ranging from triage to referral) [ 7 ]. The AIIMS-NITI study has provided concrete evidence of a fragmented ECS in India, covering all phases from prehospital to in-hospital and follow-up care. In the current scenario, trauma patients are sent to a designated trauma centre, while maternity and paediatric cases are channelled to specialized facilities for mothers and children [ 24 ]. This situation highlights the critical need for an integrated system that can provide emergency treatment to any patient with a time-sensitive condition within a single, cohesive emergency care structure. Hence, the authors aim to conduct this novel IR study on integrated ECS delivery to address these gaps and enhance patient care.

The objectives of this study are (a) to identify the barriers and facilitators of emergency care delivery for patients with time-sensitive conditions and (b) to develop and implement a contextually relevant model, measuring its impact through implementation research outcomes such as acceptability, adoption, fidelity and penetration.

Study settings

This multi-centric study will be carried out in healthcare facilities (HCF) of five zones of India, that is, the north-west, north, east, south and north-east regions (Fig.  1 a further lists the different study sites along with the inclusion criteria). The study period is 3 years and will be centrally coordinated from All India Institute of Medical Sciences (AIIMS) and ICMR, New Delhi. In India, the public health system is divided into the primary level (primary health centre and sub-centre), secondary level (sub-district hospital and community health centre) and tertiary level (medical college and district hospital) [ 25 ]. For the study, one chain of HCF (from primary level to tertiary level of care) will be selected from each zone. Each chain will consist of a total of 17 HCFs, that is, one medical college, one district hospital (DH), one sub-district hospital (SDH), two community health centres (CHC), four primary health centres (PHC) and eight sub-centres/health and wellness centres (HWC; Fig.  1 b). So, a total of 85 HCFs were selected.

a Map of India depicting five zones, b presentation of a chain of linked healthcare facilities. CHC community health centre, PHC primary health centre, HWC health and wellness centre

These five study zones were selected for this study through purposive sampling, with a total of 17 healthcare facilities (HCF) or sites chosen in each zone. Naharlagun in Arunachal Pradesh was chosen due to its challenging geographical terrain and sparsely populated areas, which make providing healthcare facilities difficult. Ayodhya in Uttar Pradesh was included because it is a significant site for mass gatherings due to religious tourism. Nalhar in Haryana was selected because of its sparse healthcare facilities, causing residents to seek emergency services in nearby cities. Gaya in Bihar is a tourist hotspot that needs enhanced emergency services for potential disasters. Trivandrum in Kerala was chosen for comparison due to its relatively better healthcare practices, allowing us to adapt and implement best practices. These zones have been purposefully chosen because, according to the Global Burden of Diseases – 2021 report, the majority of deaths in these regions were related to ischemic heart diseases, stroke, breathlessness, trauma and infections [ 26 ].

Team description

The study team will comprise four groups: the central coordination team, the technical support unit, the implementers and the research teams. The detailed roles and responsibilities are as follows:

Central Coordination Team (at AIIMS New Delhi): This team will be responsible for the overall study management, coordination and oversight. It will consist of emergency medicine consultants and project scientists who will coordinate with project site staff to ensure the smooth implementation of the study protocol.

Technical support unit (at each study zones): The technical support unit (TSU) will consist of key administrative stakeholders from the state, medical colleges, districts and healthcare facilities (HCF). These stakeholders include state and district health authorities (Ministry of Health and Family Welfare officials, the Health Secretary, the director of Health Services, State National Health Mission officials and chief district medical Officers), as well as facility administrators (principals, medical superintendents, chief medical officers and chief nursing officers). Subject experts from emergency medicine, public health and external support organizations with expertise in implementation research are also part of this TSU. Their primary role will be to oversee local implementation efforts and ensure alignment with the directives from the central coordination team. They will participate in stakeholder meetings to assess the barriers and facilitators of ECS and will plan the individual implementation strategies. Additionally, they will act as liaisons between the central coordination team and local arrangements, facilitating effective communication and collaboration.

Implementers (at each facility): The implementers’ team will consist of key individuals from the facility administration, and facility stakeholders, including ED physicians, ED nurses, supporting healthcare staff and ambulance personnel. The ED team members will be responsible for providing care in terms of recognition, resuscitation and referral (3R) for TSC. They will be responsible for implementing the decided strategies at the facilities. They will follow the guidance of TSU and CCT, and provide feedback during meetings with the facility administrators.

Research teams (at each facility): The research team members will be recruited through this IR project and will be deployed at each facility during the study period. They will be retracted once a sustainable model is developed and implemented successfully. This team includes multiple sub-teams with specific roles:

Formative research and program evaluation team: Comprising project scientists, this team will conduct formative research and concurrent evaluations every 3 months to improve the implementation strategies. They will analyse collected data for learning and feedback. Project scientists in this team will conduct baseline, midline and end-line assessments, collecting both qualitative and quantitative data. This sub-team will provide feedback to the TSU and implementers.

Implementation support team (IST): This team will consist of nurses (emergency nurse coordinators) who will be available at the site throughout the implementation. They will assist implementers in applying the implementation strategies, help in clinical work, do data collection and perform community engagement.

Implementation phases

The study will be conducted in seven phases, that is, the preparatory phase, formative research, co-design of Model “Zero”, co-implementation, modification of Model “Zero”, end-line evaluations, and consolidation and dissemination of results. This implementation strategy is summarized in Fig.  2 .

Phases of implementation planned in this study. IS implementation strategies, TSC time-sensitive conditions

Phase 1 – Preparatory phase

The research protocol for the study sites was finalized with due permission and approval from the authorities. The next step is to establish a team and identify the roles of different team members at the central coordinating centre along with the team allocated to different sites. Further selection of chain of hospitals from different levels of healthcare facilities such as primary, secondary and tertiary was done in the respective districts.

Phase 2 – Formative assessment

In Phase 2, we will conduct formative research using a mixed methods approach, which includes both quantitative and qualitative assessments. The research will consist of several components: stakeholder mapping; assessment of the community, prehospital system and emergency department readiness for managing TSC; and data interpretation to identify and tailor implementation strategies for the following phase. Stakeholder mapping will be conducted through in-depth interviews and focus group discussions with key stakeholders, including state and district health authorities, facility administrators, healthcare providers, patients and ambulance personnel. These discussions will help us identify the necessary stakeholders to discuss barriers, facilitators of ECS for TSC and implementation strategies.

The readiness of the community will be assessed through interviews with the general public and bystanders/relatives of the patients arriving at the healthcare facilities. The readiness of the prehospital infrastructure will be evaluated using quantitative tools, live observation of ambulance system processes and qualitative interviews with ambulance personnel, command centre staff and their administrators. The readiness of the emergency department regarding the management of TSC will be assessed using quantitative tools such as surveys, questionnaires, existing performance data, clinical records and process documentation. Additionally, the research team will conduct live observations of the ED processes and carry out interviews, including in-depth interviews with facility administrators, physicians and nurses, as well as focus group discussions with allied healthcare staff.

The findings from this formative research will inform the subsequent phases of the study. Specifically, the data will be used to refine and tailor interventions on components of 3R in each TSC, to address site-specific barriers and facilitators and to develop targeted strategies for improving implementation processes. This approach ensures that the research is context-specific and directly applicable to the study sites and also the best practices from other sites are found, providing a robust foundation for refining and improving our implementation strategies.

Phase 3 – Co-design of Model “Zero”

The analysis of the formative research data will help us identify gaps, facilitators, barriers, good practices and opportunities. Following this analysis, nexus planning will be conducted in collaboration with the CCT, TSU and local stakeholders from each district. Nexus planning involves organizing meetings with relevant stakeholders, including chief investigators from CCT, subject experts, site investigators, key administrative stakeholders from the state, medical colleges, districts, healthcare facilities (HCF), facility administration and implementers such as senior ED physicians and senior ED nurses. The first implementation model, known as Model “Zero”, will be developed on the basis of the insights gained from these meetings. The development of Model “Zero” will follow the intervention mapping framework as outlined by Powell et al., which consists of five steps: needs assessment, defining proximal program objectives, enlisting intervention methods, designing implementation strategies, and monitoring and evaluation [ 27 ]. The needs assessment has already been completed during the formative research phase. We will then specify the objectives of the implementation model, focusing on recognition, resuscitation and referral (3R) of time-sensitive conditions. Intervention methods will be generated to align with these objectives, tailored to address the specific barriers and facilitators identified in the formative research. These methods will be operationalized into detailed implementation strategies, outlined in Table  1 and Supplementary Table 1, which will guide the activities for each of the 3R components. Finally, the research team will monitor the implementation process and evaluate the research outcomes (Table 2 ; Supplementary Table S2).

The planned implementation strategies with respect to 3R components are provided in Table  1 and relevant details are available in Supplementary Table S1. Regarding recognition, our strategies at the pre-facility level will involve creating and distributing educational resources within the community, providing training on identifying TSC and making use of the ambulance system. At the in-facility level, the focus will be on the implementation of a data collection system for triage, the formation of triage teams, the establishment of standardized protocols, the training of personnel, the regular auditing of processes and the development of essential infrastructure. In terms of resuscitation, pre-facility efforts involve similar educational and training initiatives, focusing on prehospital resuscitation, ensuring the availability of infrastructure and equipment and development of a prehospital data collection system. At the in-facility level, the focus shifts to developing clinical pathways, standard operating protocols, the reorganization of ED resuscitation bays, trainer training, continuous staff training, academic partnerships, quality improvement projects and identification of local champions for change. Finally, for improving the referral system, the initial methods will involve developing referral protocols and systems before patients are admitted to a HCF. For patients in the facility, efforts are made to standardize referral protocols, conduct monthly audits, promote information exchange and train healthcare staff on referral policies.

To evaluate successful implementation, four implantation research outcomes (IRO) will be investigated, that is, “Acceptability”, “Adoption”, “Fidelity” and “Penetration”, as proposed by Proctor et al. [ 28 ]. For each implementation strategy (described in Table  1 ), we will evaluate the implementation process using these IROs. The detailed implementation research outcomes and their corresponding implementation strategies are presented in Table  2 , and proposed calculations are provided in Supplementary Table S2. We will evaluate the acceptability of community training among the general public, prehospital training among ambulance personnel, triage implementation among triage staff, SOP for TSC management among triage staff and SOP for referral of patients with TSC. The adoption of an implementation strategy will be assessed in the initial phase of the study. We will examine the adoption of triage by triage teams, SOP for TSC management and referral policy. We will investigate the fidelity of triage implementation in adherence to prescribed guidelines, resuscitation adherence for each TSC and fidelity of referrals. We will assess the penetration of our implementation strategy in the end-line evaluation phase. Penetration of infrastructure readiness and equipment availability in ambulances, prehospital data collection systems, triage implementation at the facility level, triage team implementation at the facility level, infrastructure for triage, proper ED resuscitation bays as per standards and referral systems at the facility level. These assessments will provide a comprehensive evaluation of the implementation process, ensuring that each strategy is effectively contributing to the desired outcomes.

Phase 4 – Co-implementation

After the nexus planning for developing Model “Zero”, which focuses on recognition, resuscitation and referral (3R); TSC; and site-specific strategies, the implementation of the planned strategies will be carried out collaboratively by the site implementers, the research team and the implementation support team of the IR project. This phase will involve the execution of all relevant planned implementation strategies as detailed in Table  1 and Supplementary Table S1. These strategies encompass various aspects, including targeted skill enhancement training, process improvements, resource allocation and quality control measures. The on-ground participatory research will facilitate the modification and contextualization of Model “One” and subsequent models, ensuring that the strategies remain tailored and effective in the local context.

Phase 5 – Model optimization

In Phase 5, we focus on the continuous optimization of the implementation model as shown in Fig. 3 . On the basis of the insights from formative assessment (phase 2), new implementation strategies will be developed to address the identified issues (phase 3). These strategies will then be co-implemented (phase 4). The “Model optimization” phase will be a cyclical phase consisting of phases 2–4; evaluation of the performance of the current model, which will be closely monitored; assessment of model’s effectiveness using IROs; and interviewing stakeholders for perceived performance to gauge its success [ 27 ]. If the performance does not meet the desired standards, internal quality improvement initiatives will be undertaken as per Deming’s cycle of Plan-Do-Study-Act [ 29 ]. These initiatives aim to enhance performance and address any residual inertia within the system. The entire process is cyclic, involving regular model evaluations and stakeholder meetings to ensure that the implementation strategies remain effective and responsive to the evolving needs of the study sites. This iterative loop of assessment, implementation and evaluation ensures that the model is continually refined and optimized for better outcomes.

Model optimization loop for implementation strategy improvement. This figure illustrates the continuous process of identifying gaps and barriers, implementing new strategies, evaluating the model and considering stakeholder perceptions. The ± signs indicate the positive or negative influence of each step on the process. Positive signs (+) indicate steps that enhance the process, while negative signs (–) indicate steps that may introduce challenges or resistance (inertia). The loops emphasize ongoing evaluation and adjustment to improve the model’s performance and address stakeholder feedback. Quality improvement loop is a component of the larger model optimization loop

Phase 6 – End-line evaluations

The end-line evaluation will measure implementation-related outcomes (IRO) such as acceptability, adoption, fidelity and penetration as outlined in Table 2 and Supplementary Table S2. The readiness of the community, prehospital services and emergency departments (ED) for managing TSC will be reassessed similarly to the formative assessment. Quantitative tools such as surveys, performance data, clinical records and live observations of ED processes, along with qualitative methods such as in-depth interviews with facility administrators, physicians and nurses, and focus group discussions with allied healthcare staff, will be used to evaluate the final model. This final model will be tailored to the specific context (geographic location and level of HCF).

Phase 7 – Consolidation and dissemination of results

The study team will withdraw the support system at HCFs as an exit plan. The research team will conduct a meeting with the ISU to prepare an effective implementation plan based on the lessons from the comprehensive through end-line evaluations. The report writing will be done after a detailed data analysis of all the gathered data. The results of the IR study will be disseminated to all stakeholders (local, district, state and national level).

Data collection and analysis

Mixed-methods research will be conducted in this implementation research project. For the qualitative study, researchers at specific sites will conduct audio recordings with the consent of the interviewees. The recorded interviews will be transcribed verbatim and translated into English. To ensure accuracy, 10% of the transcriptions will be validated by re-listening to the recordings. Thematic analysis with inductive coding will be performed to understand the context and identify barriers and facilitators of emergency services for TSC at pre-facility and in-facility levels [ 30 ]. This analysis will be done according to interviewee category (physicians, nurses, administrators, etc.), site and healthcare facility (HCF) level. We will compare the zone-wise barriers and facilitators and create frequency tables for each domain and subdomain to list the barriers and facilitators on the basis of the constructs.

For the quantitative methodology, the readiness of the system will be assessed using quantitative tools and live observations of TSC management at each HCF across different zones as described in the “Phase 2 – Formative assessment”. Quantitative variables will be represented as means or medians, while categorical variables will be represented as frequencies and percentages. From the combined quantitative and qualitative data, IRO such as “Acceptability”, “Adoption”, “Fidelity” and “Penetration” will be derived. Detailed calculations for IRO are provided in Supplementary Table 2. “Acceptability” and “Adoption” will be assessed in the early phase of the study, while “Fidelity” and “Penetration” of intervention strategies will be evaluated at 3-month intervals to monitor trends and inform “Phase 5 – Model optimization”. Quantitative data analysis will be performed using the latest version of SPSS, and qualitative data analysis will be conducted using the latest version of NVivo.

This study aims to address critical gaps in India’s emergency care systems by implementing an integrated model focusing on time-sensitive conditions. We will use a multi-phase approach, including formative assessment, model co-design and co-implementation to ensure strategies are contextually relevant and tailored to the specific needs of each healthcare facility. The combination of qualitative and quantitative methodologies will provide a comprehensive understanding of barriers and facilitators to effective ECS delivery, and the iterative model optimization will ensure continuous improvement.

During the formative assessment phase, we will identify key gaps in community readiness, prehospital infrastructure and emergency department management of TSC. These insights will inform the development of Model “Zero” through collaborative nexus planning with key stakeholders. In the co-implementation phase, we will execute targeted strategies such as skill enhancement training, process improvements and resource allocation, tailored to each facility’s needs. This participatory approach will facilitate model adaptation to local contexts, ensuring its effectiveness and sustainability. In the model optimization phase, we will refine strategies through continuous evaluation and stakeholder feedback. The end-line evaluation will assess the impact of these strategies on implementation-related outcomes, such as acceptability, adoption, fidelity and penetration, using quantitative and qualitative tools. This comprehensive evaluation framework will ensure that the final model is both effective and adaptable to various healthcare settings. At the end, we will implement an exit strategy, gradually withdrawing research support from healthcare facilities while finalizing an effective implementation plan. The findings will be disseminated to stakeholders at all levels to ensure sustainable adoption and adaptation of the integrated emergency care system model.

This study will systematically conduct implementation research on India’s emergency care systems by focusing on time-sensitive conditions through a contextually tailored, integrated model. The multi-phase implementation strategy, incorporating both qualitative and quantitative methodologies, will ensure continuous optimization and effectiveness. The findings from this research will provide valuable insights into the barriers and facilitators of ECS delivery and contribute to developing sustainable, high-quality emergency care systems in low- and middle-income countries.

Availability of data and materials

No datasets were generated or analysed during the current study.

Abbreviations

All India Institute of Medical Sciences

Central coordinating team

Emergency care system

Emergency department

Focused group discussion

Global burden of disease

Healthcare facilities

Indian Council of Medical Research

In-depth interview

• Implementation research

Implementation support team

Low- and middle-income countries

Non-informal interactions

National Institute of Transforming India

Technical support unit

Reynolds TA, Sawe H, Rubiano AM, Shin SD, Wallis L, Mock CN, et al. Strengthening health systems to provide emergency care. In: Jamison DT, Gelband H, Horton S, Jha P, Laxminarayan R, Mock CN, et al., editors. Disease control priorities: improving health and reducing poverty. 3rd ed. Washington (DC): The International Bank for Reconstruction and Development/The World Bank; 2017.

Google Scholar  

World Health Organization: WHO. (2020, December 9). The top 10 causes of death. https://www.who.int/news-room/fact-sheets/detail/the-top-10-causes-of-death . Accessed 29 Jan 2024.

Chang CY, Abujaber S, Reynolds TA, Camargo CA, Obermeyer Z. Burden of emergency conditions and emergency care usage: new estimates from 40 countries. Emerg Med J. 2016;33:794–800.

Article   PubMed   Google Scholar  

Lang ES, Wyer PC, Eskin B. Executive summary: knowledge translation in emergency medicine: establishing a research agenda and guide map for evidence uptake. Acad Emerg Med. 2007;14:915–8.

UN DESA Policy Brief No. 153: India overtakes China as the world’s most populous country. (2023, April 24). https://www.un.org/development/desa/dpad/publication/un-desa-policy-brief-no-153-india-overtakes-china-as-the-worlds-most-populous-country/ .

Misra A, Yadav DC, Kole T. Emergency care in India beyond 75 years of independence – problems and solutions. J Glob Health. 2023;13:03015.

Article   PubMed   PubMed Central   Google Scholar  

Emergency and injury care at secondary and tertiary level centres in India. A report of current status on country level assessment. https://www.niti.gov.in/sites/default/files/2021-12/AIIMS_STUDY_1.pdf . Accessed 10 Apr 2024.

Wibring K, Magnusson C, Axelsson C, Lundgren P, Herlitz J, Andersson HM. Towards definitions of time-sensitive conditions in prehospital care. Scand J Trauma Resusc Emerg Med. 2020;28:7.

Obermeyer Z, Abujaber S, Makar M, Stoll S, Kayden SR, Wallis LA, et al. Emergency care in 59 low- and middle-income countries: a systematic review. Bull World Health Organ. 2015;93:577-586G.

Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50.

de Wit K, Curran J, Thoma B, Dowling S, Lang E, Kuljic N, et al. Review of implementation strategies to change healthcare provider behaviour in the emergency department. CJEM. 2018;20:453–60.

Asha S, Curtis KA, Grant N, Taylor C, Lo S, Smart R, et al. Comparison of radiation exposure of trauma patients from diagnostic radiology procedures before and after the introduction of a panscan protocol. Emerg Med Australas. 2012;24:43–51.

Baker K, Brierley S, Mitchell G, Roe J. Effecting change using careplans: experience from two fractured neck of femur pathways. Aust Health Rev. 2012;36:308–12.

Ballow SL, Kaups KL, Anderson S, Chang M. A standardized rapid sequence intubation protocol facilitates airway management in critically injured patients. J Trauma Acute Care Surg. 2012;73:1401–5.

Bressan S, Romanato S, Mion T, Zanconato S, Da Dalt L. Implementation of adapted PECARN decision rule for children with minor head injury in the pediatric emergency department. Acad Emerg Med. 2012;19:801–7.

Heskestad B, Waterloo K, Ingebrigtsen T, Romner B, Harr ME, Helseth E. An observational study of compliance with the Scandinavian guidelines for management of minimal, mild and moderate head injury. Scand J Trauma Resusc Emerg Med. 2012;20:32.

Puljula J, Cygnel H, Mäkinen E, Tuomivaara V, Karttunen V, Karttunen A, et al. Mild traumatic brain injury diagnosis frequently remains unrecorded in subjects with craniofacial fractures. Injury. 2012;43:2100–4.

Lim C, Bawden J, Wing A, Villa-Roel C, Meurer DP, Bullard MJ, et al. Febrile neutropenia in EDs: the role of an electronic clinical practice guideline. Am J Emerg Med. 2012;30:5-11.e5.

Na S, Kuan WS, Mahadevan M, Li C-H, Shrikhande P, Ray S, et al. Implementation of early goal-directed therapy and the surviving sepsis campaign resuscitation bundle in Asia. Int J Qual Health Care. 2012;24:452–62.

Nguyen HM, Schiavoni A, Scott KD, Tanios MA. Implementation of sepsis management guideline in a community-based teaching hospital – can education be potentially beneficial for septic patients? Int J Clin Pract. 2012;66:705–10.

Article   PubMed   CAS   Google Scholar  

Plambech MZ, Lurie AI, Ipsen HL. Initial, successful implementation of sepsis guidelines in an emergency department. Dan Med J. 2012;59:A4545.

PubMed   Google Scholar  

Georgiou A, Lam M, Allardice J, Hart GK, Westbrook JI. Troponin testing in the emergency department: a longitudinal study to assess the impact and sustainability of decision support strategies. J Clin Pathol. 2012;65:546–50.

Kinsman LD, Rotter T, Willis J, Snow PC, Buykx P, Humphreys JS. Do clinical pathways enhance access to evidence-based acute myocardial infarction treatment in rural emergency departments? Aust J Rural Health. 2012;20:59–66.

Asadi L, Eurich DT, Gamble J-M, Minhas-Sandhu JK, Marrie TJ, Majumdar SR. Guideline adherence and macrolides reduced mortality in outpatients with pneumonia. Respir Med. 2012;106:451–8.

Chokshi M, Patil B, Khanna R, Neogi SB, Sharma J, Paul VK, et al. Health systems in India. J Perinatol. 2016;36:S9-12.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Naghavi M, Ong KL, Aali A, Ababneh HS, Abate YH, Abbafati C, et al. Global burden of 288 causes of death and life expectancy decomposition in 204 countries and territories and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet. 2024;403:2100–32.

Article   Google Scholar  

Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44:177–94.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.

Moen, R. "Foundation and History of the PDSA Cycle." Asian network for quality conference. Tokyo. https://www.deming.org/sites/default/files/pdf/2015/PDSA_History_Ron_Moen.Pdf.2009 . 2009. Accessed 11 Apr 2024.

Naeem M, Ozuem W, Howell K, Ranfagni S. A step-by-step process of thematic analysis to develop a conceptual model in qualitative research. Int J Qual Methods. 2023;22:16094069231205788.

Download references

Acknowledgements

A note of thanks is devoted to the authors for their contribution in the manuscript. It is a multi-region collaboration with involvement of researchers at different study sites. This enable us to promote greater equity and coverage. Dr Rupak Mukhopadhyay helped in the framework of manuscript with respect to implementation research. Dr Dushyant helped in coordination from study sites while framing the manuscript.

This project is centrally funded from the Indian Council of Medical Research, India.

Author information

Authors and affiliations.

Department of Emergency Medicine, All India Institute of Medical Sciences, New Delhi, India

Tej Prakash Sinha, Sanjeev Bhoi, Dolly Sharma, Sushmita Chauhan, Radhika Magan & Ankit Kumar Sahu

Division of NCD, ICMR, New Delhi, India

Stuti Bhargava

WHO SEARO, New Delhi, India

Patanjali Dev Nayar

OSD, Health, NITI Aayog, New Delhi, India

Venkatnarayan Kannan

Department of Paediatrics, AIIMS, New Delhi, India

Rakesh Lodha

Department of Gynaecology, AIIMS, New Delhi, India

Garima Kacchawa

INCLEN, New Delhi, India

Narendra Kumar Arora

Tomo Riba Institute of Health & Medical Sciences, Naharlagun, Arunachal Pradesh, India

Medicine Department, Anugrah Narayan Magadh Medical College, Gaya, Bihar, India

Pramod Kumar Sinha

Rajarshi Dashrath Autonomous State Medical College, Ayodhya, Uttar Pradesh, India

Satyajeet Verma

Shaheed Hasan Khan Mewati Government Medical College Nalhar Hospital, Nuh, Haryana, India

Pawan Goyal

Emergency Department, Trivandrum Medical College, Trivandrum, Kerala, India

K. V. Viswanathan

Department of Emergency & Trauma, Tomo Riba Institute of Health & Medical Sciences, Naharlagun, Arunachal Pradesh, India

Department of Community Medicine, Tomo Riba Institute of Health & Medical Sciences, Naharlagun, Arunachal Pradesh, India

Pallavi Boro

Department of Opthalmology, Shaheed Hasan Khan Mewati Government Medical College Nalhar Hospital, Nuh, Haryana, India

Yogesh Kumar

Department of Community Medicine, Rajarshi Dashrath Autonomous State Medical College, Ayodhya, Uttar Pradesh, India

Pratibha Gupta

Emergency & Trauma Care, Trivandrum District Hospital, Thiruvananthapuram, Kerala, India

Srikanth Damodaran

Department of Emergency Medicine, Anugrah Narayan Magadh Medical College, Gaya, Bihar, India

Nasar Jubair

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: Dr Sanjeev Bhoi, Dr Tej Prakash Sinha, Dr Patanjali Nayar, Dr Venkatnarayan Kannan, Dr Rakesh Lodha, Dr Garima Kacchawa, Dr Narendra Kumar Arora, Ms Dolly Sharma, Dr Moji Jini, Dr Pramod Kumar Sinha, Dr Satyajeet Verma, Dr Pawan Goyal and Dr Viswanathan KV. Methodology: Dr Sanjeev Bhoi, Dr Tej Prakash Sinha, Dr Patanjali Nayar, Dr Ankit Sahu, and Ms Dolly Sharma. Original draft preparation: Dr Ankit Sahu, Ms Dolly Sharma, Ms Sushmita Chauhan and Dr Radhika Magan. Review and editing: Dr Ankit Sahu, Dr Sanjeev Bhoi, Dr Tej Prakash Sinha and Ms Dolly Sharma. Project administration: Dr Sanjeev Bhoi, Dr Tej Prakash Sinha, Ms Dolly Sharma, Dr Stuti Bhargava, Dr Kemba Padu, Dr Pallavi Boro, Dr Yogesh Kumar, Dr Pratibha Gupta, Dr Srikanth Damodaran and Dr Nasir Jubair.

Corresponding author

Correspondence to Sanjeev Bhoi .

Ethics declarations

Ethics approval and consent to participate.

The ethics approval was taken from the central institute coordinating the project (Institute Ethics Committee, All India Institute of Medical Sciences, New Delhi, India), and from all five medical colleges, which are the tertiary centres handling the chain of hospitals under them. These medical colleges are as follows: (1) Tomo Riba Institute of Health & Medical Sciences, Naharlagun, Arunachal Pradesh; (2) Anugrah Narayan Magadh Medical College, Gaya, Bihar; (3) Rajarshi Dashrath Autonomous State Medical College, Ayodhya, Uttar Pradesh; (4) Shaheed Hasan Khan Mewati Government Medical College Nalhar Hospital, Nuh, Haryana; and (5) Trivandrum Medical College, Kerala. Appropriate written permission will be taken from the administrators of district hospitals, sub-district hospitals, community health centres, primary health centres and health and wellness centres. Since this is a system-level intervention, individual consent from TSC patients will not be required. However, for qualitative interviews, each interviewee will be asked to provide consent, which will be recorded in the audio.

Consent for publications

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sinha, T.P., Bhoi, S., Sharma, D. et al. Strengthening facility-based integrated emergency care services for time sensitive emergencies at all levels of healthcare in India: An implementation research study protocol. Health Res Policy Sys 22 , 125 (2024). https://doi.org/10.1186/s12961-024-01183-x

Download citation

Received : 10 April 2024

Accepted : 15 July 2024

Published : 09 September 2024

DOI : https://doi.org/10.1186/s12961-024-01183-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emergency care services
• Time-sensitive conditions

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

Combining qualitative and quantitative research within mixed method research designs: A methodological review

Ulrika Östlund.

a Division of Nursing, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

b Institute for Applied Health Research/School of Health, Glasgow Caledonian University, United Kingdom

Yvonne Wengström

c Division of Nursing, Department or Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

Neneh Rowa-Dewar

d Public Health Sciences, University of Edinburgh, United Kingdom

It has been argued that mixed methods research can be useful in nursing and health science because of the complexity of the phenomena studied. However, the integration of qualitative and quantitative approaches continues to be one of much debate and there is a need for a rigorous framework for designing and interpreting mixed methods research. This paper explores the analytical approaches (i.e. parallel, concurrent or sequential) used in mixed methods studies within healthcare and exemplifies the use of triangulation as a methodological metaphor for drawing inferences from qualitative and quantitative findings originating from such analyses.

This review of the literature used systematic principles in searching CINAHL, Medline and PsycINFO for healthcare research studies which employed a mixed methods approach and were published in the English language between January 1999 and September 2009.

In total, 168 studies were included in the results. Most studies originated in the United States of America (USA), the United Kingdom (UK) and Canada. The analytic approach most widely used was parallel data analysis. A number of studies used sequential data analysis; far fewer studies employed concurrent data analysis. Very few of these studies clearly articulated the purpose for using a mixed methods design. The use of the methodological metaphor of triangulation on convergent, complementary, and divergent results from mixed methods studies is exemplified and an example of developing theory from such data is provided.

A trend for conducting parallel data analysis on quantitative and qualitative data in mixed methods healthcare research has been identified in the studies included in this review. Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings, help researchers to clarify their theoretical propositions and the basis of their results. This can offer a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and develop new theory.

What is already known about the topic?

• • Mixed methods research, where quantitative and qualitative methods are combined, is increasingly recognized as valuable, because it can potentially capitalize on the respective strengths of quantitative and qualitative approaches.
• • There is a lack of pragmatic guidance in the research literature as how to combine qualitative and quantitative approaches and how to integrate qualitative and quantitative findings.
• • Analytical approaches used in mixed-methods studies differ on the basis of the sequence in which the components occur and the emphasis given to each, e.g. parallel, sequential or concurrent.

What this paper adds

• • A trend for conducting parallel analysis on quantitative and qualitative data in healthcare research is apparent within the literature.
• • Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings and help researchers to clearly present both their theoretical propositions and the basis of their results.
• • Using triangulation as a methodological metaphor may also support a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and aid the development of new theory.

1. Introduction

Mixed methods research has been widely used within healthcare research for a variety of reasons. The integration of qualitative and quantitative approaches is an interesting issue and continues to be one of much debate ( Bryman, 2004 , Morgan, 2007 , Onwuegbuzie and Leech, 2005 ). In particular, the different epistemological and ontological assumptions and paradigms associated with qualitative and quantitative research have had a major influence on discussions on whether the integration of the two is feasible, let alone desirable ( Morgan, 2007 , Sale et al., 2002 ). Proponents of mixed methods research suggest that the purist view, that quantitative and qualitative approaches cannot be merged, poses a threat to the advancement of science ( Onwuegbuzie and Leech, 2005 ) and that while epistemological and ontological commitments may be associated with certain research methods, the connections are not necessary deterministic ( Bryman, 2004 ). Mixed methods research can be viewed as an approach which draws upon the strengths and perspectives of each method, recognising the existence and importance of the physical, natural world as well as the importance of reality and influence of human experience ( Johnson and Onquegbuzie, 2004 ). Rather than continue these debates in this paper, we aim to explore the approaches used to integrate qualitative and quantitative data within healthcare research to date. Accordingly, this paper focuses on the practical issues of conducting mixed methods studies and the need to develop a rigorous framework for designing and interpreting mixed methods studies to advance the field. In this paper, we will attempt to offer some guidance for those interested in mixed methods research on ways to combine qualitative and quantitative methods.

The concept of mixing methods was first introduced by Jick (1979) , as a means for seeking convergence across qualitative and quantitative methods within social science research ( Creswell, 2003 ). It has been argued that mixed methods research can be particularly useful in healthcare research as only a broader range of perspectives can do justice to the complexity of the phenomena studied ( Clarke and Yaros, 1988 , Foss and Ellefsen, 2002 , Steckler et al., 1992 ). By combining qualitative and quantitative findings, an overall or negotiated account of the findings can be forged, not possible by using a singular approach ( Bryman, 2007 ). Mixed methods can also help to highlight the similarities and differences between particular aspects of a phenomenon ( Bernardi et al., 2007 ). Interest in, and expansion of, the use of mixed methods designs have most recently been fuelled by pragmatic issues: the increasing demand for cost effective research and the move away from theoretically driven research to research which meets policymakers’ and practitioners’ needs and the growing competition for research funding ( Brannen, 2009 , O’Cathain et al., 2007 ).

Tashakkori and Creswell (2007) broadly define mixed methods research as “research in which the investigator collects and analyses data, integrates the findings and draws inferences using both qualitative and quantitative approaches” (2007:3). In any mixed methods study, the purpose of mixing qualitative and quantitative methods should be clear in order to determine how the analytic techniques relate to one another and how, if at all, the findings should be integrated ( O’Cathain et al., 2008 , Onwuegbuzie and Teddlie, 2003 ). It has been argued that a characteristic of truly mixed methods studies are those which involve integration of the qualitative and quantitative findings at some stage of the research process, be that during data collection, analysis or at the interpretative stage of the research ( Kroll and Neri, 2009 ). An example of this is found in mixed methods studies which use a concurrent data analysis approach, in which each data set is integrated during the analytic stage to provide a complete picture developed from both data sets after data has been qualitised or quantitised (i.e. where both forms of data have been converted into either qualitative or quantitative data so that it can be easily merged) ( Onwuegbuzie and Teddlie, 2003 ). Other analytic approaches have been identified including; parallel data analysis, in which collection and analysis of both data sets is carried out separately and the findings are not compared or consolidated until the interpretation stage, and finally sequential data analysis, in which data are analysed in a particular sequence with the purpose of informing, rather than being integrated with, the use of, or findings from, the other method ( Onwuegbuzie and Teddlie, 2003 ). An example of sequential data analysis might be where quantitative findings are intended to lead to theoretical sampling in an in depth qualitative investigation or where qualitative data is used to generate items for the development of quantitative measures.

When qualitative and quantitative methods are mixed in a single study, one method is usually given priority over the other. In such cases, the aim of the study, the rationale for employing mixed methods, and the weighting of each method determine whether, and how, the empirical findings will be integrated. This is less challenging in sequential mixed methods studies where one approach clearly informs the other, however, guidance on combining qualitative and quantitative data of equal weight, for example, in concurrent mixed methods studies, is rather less clear ( Foss and Ellefsen, 2002 ). This is made all the more challenging by a common flaw which is to insufficiently and inexplicitly identify the relationships between the epistemological and methodological concepts in a particular study and the theoretical propositions about the nature of the phenomena under investigation ( Kelle, 2001 ).

One approach to combining different data of equal weight and which facilitate clear identification of the links between the different levels of theory, epistemology, and methodology could be to frame triangulation as a ‘methodological metaphor’, as argued by Erzberger and Kelle (2003) . This can help to; describe the logical relations between the qualitative and quantitative findings and the theoretical concepts in a study; demonstrate the way in which both qualitative and quantitative data can be combined to facilitate an improved understanding of particular phenomena; and, can also be used to help generate new theory ( Erzberger and Kelle, 2003 ) (see Fig. 1 ). The points of the triangle represent theoretical propositions and empirical findings from qualitative and quantitative data while the sides of the triangle represent the logical relationships between these propositions and findings. The nature and use of the triangle depends upon the outcome from the analysis, whether that be convergent , where qualitative and quantitative findings lead to the same conclusion; complementary, where qualitative and quantitative results can be used to supplement each other or; divergent , where the combination of qualitative and quantitative results provides different (and at times contradictory) findings. Each of these outcomes requires a different way of using the triangulation metaphor to link theoretical propositions to empirical findings ( Erzberger and Kelle, 2003 ).

Illustrating the triangulation triangle ( Erzberger and Kelle, 2003 )

1.1. Purpose of this paper

In the following paper, we identify the analytical approaches used in mixed methods healthcare research and exemplify the use of triangulation ( Erzberger and Kelle, 2003 ) as a methodological metaphor for drawing inferences from qualitative and quantitative findings. Papers reporting on mixed methods studies within healthcare research were reviewed to (i) determine the type of analysis approach used, i.e. parallel, concurrent, or sequential data analysis and, (ii) identify studies which could be used to illustrate the use of the methodological metaphor of triangulation suggested by Erzberger and Kelle (2003) . Four papers were selected to illustrate the application of the triangulation metaphor on complementary, convergent and divergent outcomes and to develop theory.

This literature review has used systematic principles ( Cochrane, 2009 , Khan, 2001 ) to search for mixed methods studies within healthcare research. The first search was conducted in September 2009 in the data bases CINAHL, Medline and PsycINFO on papers published in English language between 1999 and 2009. To identify mixed methods studies, the search terms (used as keywords and where possible as MeSH terms) were: “mixed methods”, “mixed research methods”, “mixed research”, “triangulation”, “complementary methods”, “concurrent mixed analysis” and “multi-strategy research.” These terms were searched individually and then combined (with OR). This resulted in 1896 hits in CINAHL, 1177 in Medline and 1943 in PsycINFO.

To focus on studies within, or relevant to, a healthcare context the following search terms were used (as keywords or as MeSH terms and combined with OR): “health care research”; “health services research”; and “health”. These limits applied to the initial search (terms combined with AND) resulted in 205 hits in Medline and 100 hits in PsycINFO. Since this combination in CINAHL only limited the search results to 1017; a similar search was conducted but without using the search term triangulation to capture mixed methods papers; resulting in 237 hits. In CINAHL the search result on 1017 papers was further limited by using “interventions” as a keyword resulting in 160 papers also selected to be reviewed. Moreover; in Medline the mixed methods data set was limited by the MeSH term “research” resulting in 218 hits and in PsycINFO with “intervention” as keyword or MeSH term resulting in 178 hits.

When duplicates were removed the total numbers of papers identified were 843. The abstracts were then reviewed by each author and those identified as relevant to the review were selected to be retrieved and reviewed in full text. Papers were selected based on the following inclusion criteria: empirical studies; published in peer review journals; healthcare research (for the purpose of this paper defined as any study focussing on participants in receipt, or involved in the delivery, of healthcare or a study conducted within a healthcare setting, e.g. different kinds of care, health economics, decision making, and professionals’ role development); and using mixed methods (defined as a study in which both qualitative and quantitative data were collected and analysed ( Halcomb et al., 2009b ). To maintain rigour, a random sample (10%) of the full text papers was reviewed jointly by two authors. Any disagreements or uncertainties that arose between the reviewers regarding their inclusion or in determining the type of analytic approach used were resolved through discussion between the authors.

In addition to the criteria outlined above, papers were excluded if the qualitative element constituted a few open-ended questions in a questionnaire, as we would agree with previous authors who have argued such studies do not strictly constitute a mixed methods design ( Kroll and Neri, 2009 ). Papers were also excluded if they could not be retrieved in full text via the library services at the University of Edinburgh, Glasgow Caledonian University or the Karolinska Institutet, or did not adequately or clearly describe their analytic strategy, for example, failing to report how the qualitative and quantitative data sets were analysed individually and, where relevant, how these were integrated. See Table 1 for reasons for the exclusion of subsequent papers.

Reasons for exclusion.

A second search was conducted within the databases of Medline, PsychInfo and Cinahl to identify studies which have specifically used Erzberger and Kelle's (2003) triangulation metaphor to frame the description and interpretation of their findings. The term ‘triangulation metaphor’ (as keywords) and author searches on ‘Christian Erzberger’ and ‘Udo Kelle’ were conducted. Three papers, published by Christian Erzberger and Udo Kelle, were identified in the PsychInfo databases but none of these were relevant to the purpose of this review. There were no other relevant papers identified in the other two databases.

168 Papers were included in the final review and reviewed to determine the type of mixed analysis approach used, i.e. parallel, concurrent, or sequential mixed analysis. Four of these papers (identified from the first search on mixed methods studies and healthcare research) were also used to exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ). Data was extracted from included papers accordingly in relation to these purposes.

In total, 168 papers were included in our review. The majority of these studies originated in the USA ( n  = 63), the UK ( n  = 39) and Canada ( n  = 19), perhaps reflecting the considerable interest and expertise in mixed methods research within these countries. The focus of the studies included in the review varied significantly and the populations studied included both patients and healthcare professionals.

3.1. Analytic approaches

Table 2 illustrates the types of analytic approaches adopted in each of the studies included in the review. The most widely used analytic approach ( n  = 98) was parallel analysis ( Creswell and Plano Clark, 2007 ). However, very few of the studies employing parallel analysis clearly articulate their purpose for mixing qualitative and quantitative data, the weighting (or priority) given to the qualitative and quantitative data or the expected outcomes from doing so, mirroring previous research findings ( O’Cathain et al., 2008 ). The weighting, or priority, of the qualitative and quantitative data in a mixed methods study is dependent upon various factors including; the aims of the study and whether the purpose is, for example, to contextualise quantitative data using qualitative data or to use qualitative data to inform a larger quantitative approach such as a survey. Nonetheless, the omission of this statement makes it difficult to determine which data set the conclusions have been drawn from and the role of, or emphasis on, each approach. Therefore, is of importance for authors to clearly state this in their papers ( Creswell and Plano Clark, 2007 ). A number of studies had also used sequential data analysis ( n  = 46), where qualitative approaches were visibly used to inform the development of both clinical tools (e.g. Canales and Rakowski, 2006 ) and research measures and surveys (e.g. Beatty et al., 2004 ) or where quantitative surveys were supplemented by and issues further explored using qualitative approaches (e.g. Abadia and Oviedo, 2009 , Cheng, 2004 , Halcomb et al., 2008 ).

Included papers illustrating their analytical approach and country of origin.

Most notably, with only 20 included studies using a concurrent approach to data analysis, this was the least common design employed. Compared to the studies using a parallel or sequential approach, the authors of concurrent studies more commonly provided an explanation for their purpose of using a mixed methods design in their study, e.g. how it addressed a gap or would facilitate and advance the state of knowledge (e.g. Bussing et al., 2005 , Kartalova-O’Doherty and Tedstone Doherty, 2009 ). Despite this, there remained a lack of clarity within these studies about the weighting given to, and priority of, each method. Consequently, the importance and relevance of the findings produced by each approach and how these have informed their conclusions and interpretation is lacking. In four of the included papers a combination of approaches to data analysis (i.e. sequential and concurrent, parallel and concurrent, or sequential and parallel) were used. In the next section, we have selected papers to illustrate the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ).

3.2. Using the methodological metaphor of triangulation

We have selected four papers from our review ( Lukkarinen, 2005 , Midtgaard et al., 2006 , Shipman et al., 2008 , Skilbeck et al., 2005 ) to illustrate how the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) can be applied to mixed methods studies. Each of these studies has been used to illustrate how the metaphor of triangulation can be applied to studies producing: (i) complementary findings, (ii) convergent findings, and (iii) divergent findings. In the following section, we demonstrate how the application of the metaphor can be used as a framework both to develop theory and to facilitate the interpretation of the findings from mixed methods studies and their conclusions in each of these scenarios, and how using the metaphor can help to promote greater clarity of the study's purpose, its theoretical propositions, and the links between data sets.

3.2.1. Triangulating complementary results

To exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from complementary results, we have drawn on the results of a UK based study by Shipman et al. (2008) ( Fig. 2 ). In the UK, members of district nursing teams (DNs) provide most nursing care to people at home in the last year of life. Following concerns that inadequate education might limit the confidence of some DNs to support patients and their carers’ at home, and that low home death rates may in part be related to this, the Department of Health (DH) identified good examples of palliative care educational initiatives for DNs and invested in a 3-year national education and support programme in the principles and practice of palliative care. Shipman et al.’s study evaluates whether the programme had measurable effects on DN knowledge and confidence in competency in the principles and practice of palliative care. The study had two parts, a summative (concerned with outcomes) quantitative component which included ‘before and after’ postal questionnaires which measured effects on DNs’ ( n  = 1280) knowledge, confidence and perceived competence in the principles and practice of palliative care and a formative (concerned with process) qualitative component which included semi-structured focus groups and interviews with a sub-sample of DNs ( n  = 39).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on complementary results in the study by Shipman et al. (2008) .

While their theoretical proposition may not be explicitly stated by the authors, there is clearly an implicit theoretical proposition that the educational intervention would improve DNs knowledge and confidence (theoretical proposition 1, Fig. 2 ). This was supported by the quantitative findings which showed significant improvement in the district nurses confidence in their professional competence post intervention. Qualitative results supported and complemented the quantitative findings as the district nurses described several benefits from the program including greater confidence in tackling complex problems and better communication with patient and carers’ because of greater understanding of the reasons for symptoms. Thus, a complementary theoretical proposition (theoretical proposition 2, Fig. 2 ) can be deduced from the qualitative findings: the DN's better understanding of factors contributing to complex problems and underlying reasons for symptoms led to improved confidence in competence raised from district nurses increased understanding.

Fig. 2 illustrates the theoretical propositions, the empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 is supported by the quantitative findings. From qualitative findings, a complementary theoretical proposition (theoretical proposition 2) can be stated explaining the process that led to the DNs improved confidence in competence.

3.2.2. Triangulating convergent results

To illustrate how the methodological metaphor of triangulation can be used to draw inferences from convergent findings, we have drawn on the example of a Danish study by Midtgaard et al. (2006) ( Fig. 3 ). This study was conducted to explore experiences of group cohesion and changes in quality of life (QoL) among people ( n  = 55) who participated in a weekly physical exercise intervention (for six weeks) during treatment for cancer. The study, conducted in a Danish hospital, involved the use of structured QoL questionnaires, administered at baseline and post intervention (at six weeks) to determine changes in QoL and health status, and qualitative focus groups, conducted post intervention (at six weeks), to explore aspects of cohesion within the group. With regards to the theoretical proposition of the study ( Fig. 3 ), group cohesion was seen as essential to understand the processes within the group that facilitated the achievement of desired outcomes and the satisfaction of affective needs as well as promoting a sense of belonging to the group itself.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on convergent results in the study by Midtgaard et al. (2006) .

This proposition was deductively tested in an intervention where patients exercised in mixed gender groups of seven to nine members during a nine hour weekly session over a six week period and was supported by both the empirical quantitative and qualitative findings. The quantitative data showed significant improvements in peoples’ emotional functioning, social functioning and mental health. The qualitative data showed how the group setting motivated the individuals to pursue personal endeavors beyond physical limitations, that patients used each others as role models during ‘down periods’ and how exercising in a group made individuals feel a sense of obligation to train and to do their best. This subsequently helped to improve their social functioning which in turn satisfied their affective needs, improving their improved emotional functioning and mental health.

Fig. 3 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Both the quantitative and qualitative findings, demonstrating improvements in participants’ emotional and social functioning and their mental health, can be attributed to the nature of group cohesion within the programme as expected.

3.2.3. Triangulating divergent results

Qualitative and quantitative results that seem to contradict each other are often explained as resulting from methodological error. However, instead of a methodological flaw, a divergent result could be a consequence of the inadequacy of the theoretical concepts used. This may indicate the need for changing or developing the theoretical concepts involved ( Erzberger and Kelle, 2003 ). The following example of using the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from divergent results is intended as an example rather than an attempt to change the theoretical concept involved. In a study by Skilbeck et al. (2005) ( Fig. 4 ), some results were found to be divergent which was explained as resulting from the use of inadequate questionnaires. We do not wish to critique their conclusion; rather we intend to simply offer an alternative interpretation for their findings.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on divergent results using the study by Skilbeck et al. (2005) .

The study aimed to explore family carers’ expectations and experiences of respite services provided by one independent hospice in North England. This hospice provides inpatient respite beds specifically for planned respite admission for a two-week period. Referrals were predominated from general practitioners and patients and their carers were offered respite care twice a year, during the study this was reduced to once a year for each patient. Data was collected prior to respite admission and post respite care by semi-structured interviews and using the Relative Stress Scale inventory (RSSI), a validated scale to measure relative distress in relation to caring. Twenty-five carers were included but pre- and post-data were completed by 12 carers. Qualitative data was analysed by using a process of constant comparison and quantitative data by descriptive and comparative statistical analysis.

No clear theoretical proposition was stated by the authors, but from the definition of respite care it is possible to deduce that ‘respite care is expected to provide relief from care-giving to the primary care provider’ (theoretical proposition 1, Fig. 4 ). This proposition was tested quantitatively by pre- and post-test using the RSSI showing that the majority of carers experienced either a negative or no change in scores following the respite stay (no test of significance was stated). Accordingly, the theoretical proposition was not supported by the quantitative empirical data. The qualitative empirical results, however, were supportive in showing that most of the carers considered respite care to be important as it enabled them to have a break and a rest from ongoing care-responsibilities. From this divergent empirical data it could be suggested changing or developing the original theoretical proposition. It seems that respite care gave the carers relief from their care-responsibilities but not from the distress carers experienced in relation to caring (measured by the used scale). We would therefore suggest that in order to lessen distress related to caring, other types of support is also needed which would change the theoretical proposition as suggested (theoretical proposition 2).

Fig. 4 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 was not supported by the quantitative findings (indicated in Fig. 4 by the broken arrow), but the qualitative findings supported this proposition. From these divergent empirical findings, the theoretical proposition could accordingly be changed and developed. Respite care seemed to provide relief from carers’ on-going care-responsibilities, but other types of support need to be added to provide relief from distress experienced (theoretical proposition 2).

3.2.4. Triangulation to produce theoretical propositions

Methodological triangulation has also been applied to illustrate how theoretical propositions can be produced by drawing on the findings from a Finnish study by Lukkarinen (2005) ( Fig. 5 ). The purpose of this longitudinal study was to describe, explain and understand the subjective health related quality of life (QoL) and life course of people with coronary artery disease (CAD). A longitudinal quantitative study was undertaken during the year post treatment and 19 individuals also attended thematic interviews one year after treatment. This study is one of the few studies that clearly defines theoretical underpinnings for both the selected methods and their purpose, namely “to obtain quantitatively abundant average information about the QoL of CAD patients and the changes in it as well as the patients’ individual, unique experiences of their respective life situations” ( Lukkarinen, 2005 :622).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) to develop theory from the study by Lukkarinen (2005) .

The results of the quantitative analysis showed that the male and female CAD patients in the youngest age group had the poorest QoL. While patients’ QoL improved in the dimensions of pain, energy and mobility it deteriorated on dimensions of social isolation, sleep and emotional reactions. From the viewpoint of methodological triangulation used in the study the aim of the quantitative approach was to observe changes in QoL at the group level and also explore correlations of background factors to QoL. The qualitative approach generated information concerning both QoL in the individuals’ life situation and life course and the individuals’ rehabilitation. Both the quantitative and the qualitative analysis showed the youngest CAD patients to have the poorest psychosocial QoL. The results obtained using qualitative methods explained the quantitative findings and offered new insight into the factors related to poor psychosocial QoL, which could be used to help develop theoretical propositions around these. Patients at risk of poorer QoL were those with an acute onset of illness at a young age that led to an unexpected termination of career, resulting in financial problems, and worries about family. This group also experienced lack of emotional support (especially the females with CAD) and were concerned for the illness that was not alleviated by treatment. The interviews and the method of phenomenological psychology therefore helped to gain insight into the participants’ situational experience of QoL and life course, not detectable by the use of a questionnaire.

Fig. 5 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the relationships between these. The use of the mixed methods approach enabled a clearer understanding to emerge in relation to the lived experience of CAD patients and the factors that were related to poor QoL. This understanding allows new theoretical propositions about these issues to be developed and further explored, as depicted at the theoretical level.

4. Discussion

As the need for, and use of, mixed methods research continues to grow, the issue of quality within mixed methods studies is becoming increasingly important ( O’Cathain et al., 2008 , O’Cathain et al., 2007 ). Similarly, the need for guidance on the analysis and integration of qualitative and quantitative data is a prominant issue ( Bazeley, 2009 ). This paper firstly intended to review the types of analytic approaches (parallel, concurrent or sequential data analysis) that have been used in mixed methods studies within healthcare research. As identified in previous research ( O’Cathain et al., 2008 ), we found that the majority of studies included in our review employed parallel data analysis in which the different analyses are not compared or consolidated until the full analysis of both data sets have been completed. A trend to conduct separate analysis on quantitative and qualitative data is apparent in mixed methods healthcare studies, despite the fact that if the data were correlated, a more complete picture of a particular phenomenon may be produced ( Onwuegbuzie and Teddlie, 2003 ). If qualitative and quantitative data are not integrated during data collection or analysis, the findings may be integrated at the stage of interpretation and conclusion.

Although little pragmatic guidance exists within the wider literature, Erzberger and Kelle (2003) have published some practical advice, on the integration of mixed methods findings. For mixed methodologists, the ‘triangulation metaphor’ offers a framework to facilitate a description of the relationships between data sets and theoretical concepts and can also assist in the integration of qualitative and quantitative data ( Erzberger and Kelle, 2003 ). Yet despite the fact that the framework was published in 2003 within Tashakkori and Teddlie's (2003) seminal work, the Handbook for Mixed Methods in Social and Behavioural Research, our search revealed that it has received little application within the published body of work around mixed methods studies since its publication. This is surprising since mixed methodologists are acutely aware of the lack of guidance with regards to the pragmatics and practicalities of conducting mixed methods research ( Bryman, 2006 , Leech et al., 2010 ). Furthermore, there have been frequent calls to move the field of mixed methods away from the “should we or shouldn’t we” debate towards the practical application, analysis and integration of mixed methods and its’ findings and what we can learn from each other's work and advice. Consequently, we have a state of ambiguity and instability in the field of mixed methods in which mixed methodologists find themselves lacking appropriate sources or evidence to draw upon with which to facilitate the future design, conduct and interpretation of mixed methods studies. It is for these reasons that we, in this paper, also intended to identify and select studies that could be used as examples for the application of Erzberger and Kelle's (2003) triangulation metaphor.

When reviewing the studies it was clear that the majority of theoretical assumptions were implicit, rather than explicitly stated by authors. Wu and Volker (2009) previously acknowledged that while studies undoubtedly have a theoretical basis in their literature reviews and the nature of their research questions, they often fail to clearly articulate a particular theoretical framework. This is unfortunate as theory can help researchers to clarify their ideas and also help data collection, analysis and to improve the study's rigour ( Wu and Volker, 2009 ). When using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ), researchers are encouraged to articulate their theoretical propositions and to validate their conclusions in relation to the chosen theories. Theory can also guide researchers when defining outcome measures . Should the findings not support the chosen theory, as shown in our examples on complementary and divergent results, researchers can modify or expand their theory accordingly and new theory may be developed ( Wu and Volker, 2009 ). It is therefore our belief that using triangulation as a methodological metaphor in mixed methods research can also benefit the design of mixed method studies.

Like other researchers ( O’Cathain et al., 2008 ), we have also found that most of the papers reviewed lacked clarity in whether the reported results primarily stemmed from qualitative or quantitative findings. Many of the papers were even less clear when discussing their results and the basis of their conclusions. The reporting of mixed methods studies is notoriously challenging, but clarity and transparency are, at the very least, crucial in such reports ( O’Cathain, 2009 ). Using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) may be one way of addressing this lack of clarity by explicitly showing the types of data that researchers have based their interpretations on. It may even help address some of the issues raised in the debate on the feasibility of integrating research methods and results stemming from different epistemological and ontological assumptions and paradigms ( Morgan, 2007 , Sale et al., 2002 ). In order to carry out methodological triangulation researchers also need to identify and observe the consistency and adequacy of the two methods, positivistic and phenomenological regarding the research questions, data collection, methods of analysis and conclusions.

While we used systematic principles in our search for mixed methods studies in healthcare research, we cannot claim to have included all such studies. In many cases, reports of mixed methods studies are subjected to ‘salami slicing’ by researchers and hence the conduct of, and findings from, individual approaches are addressed in separate papers. Since these papers are often not indexed as a ‘mixed method’ study, they are undoubtedly more difficult to identify. Furthermore, different terminologies are used to describe and index mixed methods studies within the electronic databases ( Halcomb and Andrew, 2009a ), making it challenging to be certain that all relevant studies were captured in this review. However, the studies included in this review should give a sufficient overview of the use of mixed analysis in healthcare research and most importantly, they enable us to make suggestions for the future design, conduct, interpretation and reporting of mixed methods studies. It is also important to emphasise that we have based our triangulation examples on the data published but have no further knowledge of the analysis and findings undertaken by the authors. The examples should thus be taken as examples and not alternative explanations or interpretations.

Mixed methods research within healthcare remains an emerging field and its use is subject to much debate. It is therefore particularly important that researchers clearly describe their use of the approach and the conclusions made to improve transparency and quality within mixed methods research. The use of triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) can help researchers not only to present their theoretical propositions but also the origin of their results in an explicit way and to understand the links between theory, epistemology and methodology in relation to their topic area. Furthermore it has the potential to make valid inferences, challenge existing theoretical assumptions and to develop or create new ones.

Conflict of interest

None declared.

Ethical approval

Not required.

• Abadia C.E., Oviedo D.G. Bureaucratic itineraries in Colombia. A theoretical and methodological tool to assess managed-care health care systems. Social Science & Medicine. 2009; 68 (6):1153–1160. [ PubMed ] [ Google Scholar ]
• Bazeley P. Analysing mixed methods data. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 84–118. [ Google Scholar ]
• Beatty P.W., Neri M.T., Bell K., DeJong G. Use of outcomes information in acute inpatient rehabilitation. American Journal of Physical Medicine & Rehabilitation. 2004; 83 (6):468–478. [ PubMed ] [ Google Scholar ]
• Bernardi L., Kleim S., von der Lippe H. Social influences on fertility: a comparative mixed methods study in Eastern and Western Germany. Journal of Mixed Methods Research. 2007; 1 (1):23–47. [ Google Scholar ]
• Brannen J. Prologue: mixed methods for novice researchers: reflections and themes. International Journal of Multiple Research Approaches. 2009; 3 (1):8–12. [ Google Scholar ]
• Bryman A. University Press; Oxford: 2004. Social Research Methods. [ Google Scholar ]
• Bryman A. Integrating quantitative and qualitative research: how is it done? Qualitative Research. 2006; 6 (1):97–113. [ Google Scholar ]
• Bryman A. Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research. 2007; 1 (1):8–22. [ Google Scholar ]
• Bussing R., Koro-Ljungberg M.E., Gary F., Mason D.M., Garvan C.W. Exploring help-seeking for ADHD symptoms: a mixed-methods approach. Harvard Review of Psychiatry. 2005; 13 (2):85–101. [ PubMed ] [ Google Scholar ]
• Canales M.K., Rakowski W. Development of a culturally specific instrument for mammography screening: an example with American Indian women in Vermont. Journal of Nursing Measurement. 2006; 14 (2):99–115. [ PubMed ] [ Google Scholar ]
• Cheng G.Y. A study of clinical questions posed by hospital clinicians. Journal of the Medical Library Association. 2004; 92 (4):445–458. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Clarke P.N., Yaros P.S. Research blenders: commentary and response. Transitions to new methodologies in nursing sciences. Nursing Science Quarterly. 1988; 1 (4):147–151. [ PubMed ] [ Google Scholar ]
• Cochrane, 2009. Cochrane Handbook for Systematic Reviews of Interventions. http://www.cochrane-handbook.org/ . The Cochrane Collaboration.
• Creswell J.W. Sage; Thousand Oaks: 2003. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. [ Google Scholar ]
• Creswell J.W., Plano Clark V.L. Sage; Thousand Oaks: 2007. Designing and Conducting Mixed Methods Research. [ Google Scholar ]
• Erzberger C., Kelle U. Making inferences in mixed methods: The rules of integration. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in Social & Behavioural Research. Sage; Thousand Oaks: 2003. pp. 457–488. [ Google Scholar ]
• Foss C., Ellefsen B. The value of combining qualitative and quantitative approaches in nursing research by means of method triangulation. Journal of Advanced Nursing. 2002; 40 (2):242–248. [ PubMed ] [ Google Scholar ]
• Halcomb E.J., Andrew A. Managing mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 50–64. [ Google Scholar ]
• Halcomb E.J., Andrew A., Brannen J. Introduction to mixed methods research for nursing and the health sciences. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 3–12. [ Google Scholar ]
• Halcomb E.J., Davidson P.M., Griffiths R., Daly J. Cardiovascular disease management: time to advance the practice nurse role? Australian Health Review. 2008; 32 (1):44–53. [ PubMed ] [ Google Scholar ]
• Jick T.D. Mixing qualitative and quantitative methods: triangulation in action. Administrative Science Quarterly. 1979; 24 :602–611. [ Google Scholar ]
• Johnson R.B., Onquegbuzie A.J. Mixed methods research: a paradigm whose time has come. Educational Researcher. 2004; 33 (7):14–26. [ Google Scholar ]
• Kartalova-O’Doherty Y., Tedstone Doherty D. Satisfied carers of persons with enduring mental illness: who and why? The International Journal of Social Psychiatry. 2009; 55 (3):257–271. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kelle U. Sociological explanations between micro and macro and the integration of qualitative and quantitative methods. Forum: Qualitative Social Research. 2001; 2 (1):5. Art. [ Google Scholar ]
• Khan K.S. Centre for Review and Dissemination; York: 2001. Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for those Carrying Out or Commissioning Reviews. [ Google Scholar ]
• Kroll T., Neri M. Designs for mixed methods research. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 31–49. [ Google Scholar ]
• Leech N.L., Dellinger A.M., Brannagan K.B., Tanaka H. Evaluating mixed research studies: a mixed methods approach. Journal of Mixed Methods Research. 2010; 4 (1):17–31. [ Google Scholar ]
• Lukkarinen H. Methodological triangulation showed the poorest quality of life in the youngest people following treatment of coronary artery disease: a longitudinal study. International Journal of Nursing Studies. 2005; 42 (6):619–627. [ PubMed ] [ Google Scholar ]
• Midtgaard J., Rorth M., Stelter R., Adamsen L. The group matters: an explorative study of group cohesion and quality of life in cancer patients participating in physical exercise intervention during treatment. European Journal of Cancer Care. 2006; 15 (1):25–33. [ PubMed ] [ Google Scholar ]
• Morgan D.L. Paradigms lost and pragmatism regained: methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research. 2007; 1 (1):48–76. [ Google Scholar ]
• O’Cathain A. Reporting mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 135–158. [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. Why, and how, mixed methods research is undertaken in health services research in England: a mixed methods study. BMC Health Services Research. 2007; 7 :85. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy. 2008; 13 (2):92–98. [ PubMed ] [ Google Scholar ]
• Onwuegbuzie A., Teddlie C. A framework for analysing data in mixed methods research. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in socIal & Behavioural Research. Sage; Thousands Oak: 2003. pp. 351–383. [ Google Scholar ]
• Onwuegbuzie A.J., Leech N.L. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative methodologies. International Journal of Social Research Methodology. 2005; 8 (5):375–387. [ Google Scholar ]
• Sale J.E.M., Lohfeld L.H., Brazil K. Revisiting the quantitative-qualitative debate: implications for mixed methods research. Quality and Quantity. 2002; 36 :43–53. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shipman C., Burt J., Ream E., Beynon T., Richardson A., Addington-Hall J. Improving district nurses’ confidence and knowledge in the principles and practice of palliative care. Journal of Advanced Nursing. 2008; 63 (5):494–505. [ PubMed ] [ Google Scholar ]
• Skilbeck J.K., Payne S.A., Ingleton M.C., Nolan M., Carey I., Hanson A. An exploration of family carers’ experience of respite services in one specialist palliative care unit. Palliative Medicine. 2005; 19 (8):610–618. [ PubMed ] [ Google Scholar ]
• Steckler A., McLeroy K.R., Goodman R.M., Bird S.T., McCormick L. Toward integrating qualitative and quantitative methods: an introduction. Health Education Quarterly. 1992; 19 (1):1–8. [ PubMed ] [ Google Scholar ]
• Tashakkori A., Creswell J.W. Editorial: the new era of mixed methods. Journal of Mixed Methods Research. 2007; 1 (1):3–7. [ Google Scholar ]
• Tashakkori A., Teddlie C. Sage; Thousands Oak: 2003. Handbook of Mixed Methods in Social & Behavioural Research. [ Google Scholar ]
• Wu H.L., Volker D.L. The use of theory in qualitative approaches to research: application in end-of-life studies. Journal of Advanced Nursing. 2009; 65 (12):2719–2732. [ PubMed ] [ Google Scholar ]