⊕⊕⊕⊕
⊕⊕⊕◯
⊕⊕◯◯
⊕◯◯◯
We now describe in more detail the five reasons (or domains) for downgrading the certainty of a body of evidence for a specific outcome. In each case, if no reason is found for downgrading the evidence, it should be classified as 'no limitation or not serious' (not important enough to warrant downgrading). If a reason is found for downgrading the evidence, it should be classified as 'serious' (downgrading the certainty rating by one level) or 'very serious' (downgrading the certainty grade by two levels). For non-randomized studies assessed with ROBINS-I, rating down by three levels should be classified as 'extremely' serious.
(1) Risk of bias or limitations in the detailed design and implementation
Our confidence in an estimate of effect decreases if studies suffer from major limitations that are likely to result in a biased assessment of the intervention effect. For randomized trials, these methodological limitations include failure to generate a random sequence, lack of allocation sequence concealment, lack of blinding (particularly with subjective outcomes that are highly susceptible to biased assessment), a large loss to follow-up or selective reporting of outcomes. Chapter 8 provides a discussion of study-level assessments of risk of bias in the context of a Cochrane Review, and proposes an approach to assessing the risk of bias for an outcome across studies as ‘Low’ risk of bias, ‘Some concerns’ and ‘High’ risk of bias for randomized trials. Levels of ‘Low’. ‘Moderate’, ‘Serious’ and ‘Critical’ risk of bias arise for non-randomized studies assessed with ROBINS-I ( Chapter 25 ). These assessments should feed directly into this GRADE domain. In particular, ‘Low’ risk of bias would indicate ‘no limitation’; ‘Some concerns’ would indicate either ‘no limitation’ or ‘serious limitation’; and ‘High’ risk of bias would indicate either ‘serious limitation’ or ‘very serious limitation’. ‘Critical’ risk of bias on ROBINS-I would indicate extremely serious limitations in GRADE. Review authors should use their judgement to decide between alternative categories, depending on the likely magnitude of the potential biases.
Every study addressing a particular outcome will differ, to some degree, in the risk of bias. Review authors should make an overall judgement on whether the certainty of evidence for an outcome warrants downgrading on the basis of study limitations. The assessment of study limitations should apply to the studies contributing to the results in the ‘Summary of findings’ table, rather than to all studies that could potentially be included in the analysis. We have argued in Chapter 7, Section 7.6.2 , that the primary analysis should be restricted to studies at low (or low and unclear) risk of bias where possible.
Table 14.2.a presents the judgements that must be made in going from assessments of the risk of bias to judgements about study limitations for each outcome included in a ‘Summary of findings’ table. A rating of high certainty evidence can be achieved only when most evidence comes from studies that met the criteria for low risk of bias. For example, of the 22 studies addressing the impact of beta-blockers on mortality in patients with heart failure, most probably or certainly used concealed allocation of the sequence, all blinded at least some key groups and follow-up of randomized patients was almost complete (Brophy et al 2001). The certainty of evidence might be downgraded by one level when most of the evidence comes from individual studies either with a crucial limitation for one item, or with some limitations for multiple items. An example of very serious limitations, warranting downgrading by two levels, is provided by evidence on surgery versus conservative treatment in the management of patients with lumbar disc prolapse (Gibson and Waddell 2007). We are uncertain of the benefit of surgery in reducing symptoms after one year or longer, because the one study included in the analysis had inadequate concealment of the allocation sequence and the outcome was assessed using a crude rating by the surgeon without blinding.
(2) Unexplained heterogeneity or inconsistency of results
When studies yield widely differing estimates of effect (heterogeneity or variability in results), investigators should look for robust explanations for that heterogeneity. For instance, drugs may have larger relative effects in sicker populations or when given in larger doses. A detailed discussion of heterogeneity and its investigation is provided in Chapter 10, Section 10.10 and Section 10.11 . If an important modifier exists, with good evidence that important outcomes are different in different subgroups (which would ideally be pre-specified), then a separate ‘Summary of findings’ table may be considered for a separate population. For instance, a separate ‘Summary of findings’ table would be used for carotid endarterectomy in symptomatic patients with high grade stenosis (70% to 99%) in which the intervention is, in the hands of the right surgeons, beneficial, and another (if review authors considered it relevant) for asymptomatic patients with low grade stenosis (less than 30%) in which surgery appears harmful (Orrapin and Rerkasem 2017). When heterogeneity exists and affects the interpretation of results, but review authors are unable to identify a plausible explanation with the data available, the certainty of the evidence decreases.
(3) Indirectness of evidence
Two types of indirectness are relevant. First, a review comparing the effectiveness of alternative interventions (say A and B) may find that randomized trials are available, but they have compared A with placebo and B with placebo. Thus, the evidence is restricted to indirect comparisons between A and B. Where indirect comparisons are undertaken within a network meta-analysis context, GRADE for network meta-analysis should be used (see Chapter 11, Section 11.5 ).
Second, a review may find randomized trials that meet eligibility criteria but address a restricted version of the main review question in terms of population, intervention, comparator or outcomes. For example, suppose that in a review addressing an intervention for secondary prevention of coronary heart disease, most identified studies happened to be in people who also had diabetes. Then the evidence may be regarded as indirect in relation to the broader question of interest because the population is primarily related to people with diabetes. The opposite scenario can equally apply: a review addressing the effect of a preventive strategy for coronary heart disease in people with diabetes may consider studies in people without diabetes to provide relevant, albeit indirect, evidence. This would be particularly likely if investigators had conducted few if any randomized trials in the target population (e.g. people with diabetes). Other sources of indirectness may arise from interventions studied (e.g. if in all included studies a technical intervention was implemented by expert, highly trained specialists in specialist centres, then evidence on the effects of the intervention outside these centres may be indirect), comparators used (e.g. if the comparator groups received an intervention that is less effective than standard treatment in most settings) and outcomes assessed (e.g. indirectness due to surrogate outcomes when data on patient-important outcomes are not available, or when investigators seek data on quality of life but only symptoms are reported). Review authors should make judgements transparent when they believe downgrading is justified, based on differences in anticipated effects in the group of primary interest. Review authors may be aided and increase transparency of their judgements about indirectness if they use Table 14.2.b available in the GRADEpro GDT software (Schünemann et al 2013).
(4) Imprecision of results
When studies include few participants or few events, and thus have wide confidence intervals, review authors can lower their rating of the certainty of the evidence. The confidence intervals included in the ‘Summary of findings’ table will provide readers with information that allows them to make, to some extent, their own rating of precision. Review authors can use a calculation of the optimal information size (OIS) or review information size (RIS), similar to sample size calculations, to make judgements about imprecision (Guyatt et al 2011b, Schünemann 2016). The OIS or RIS is calculated on the basis of the number of participants required for an adequately powered individual study. If the 95% confidence interval excludes a risk ratio (RR) of 1.0, and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (an RR of under 0.75 or over 1.25 is often suggested as a very rough guide) downgrading for imprecision may be appropriate even if OIS criteria are met (Guyatt et al 2011b, Schünemann 2016).
(5) High probability of publication bias
The certainty of evidence level may be downgraded if investigators fail to report studies on the basis of results (typically those that show no effect: publication bias) or outcomes (typically those that may be harmful or for which no effect was observed: selective outcome non-reporting bias). Selective reporting of outcomes from among multiple outcomes measured is assessed at the study level as part of the assessment of risk of bias (see Chapter 8, Section 8.7 ), so for the studies contributing to the outcome in the ‘Summary of findings’ table this is addressed by domain 1 above (limitations in the design and implementation). If a large number of studies included in the review do not contribute to an outcome, or if there is evidence of publication bias, the certainty of the evidence may be downgraded. Chapter 13 provides a detailed discussion of reporting biases, including publication bias, and how it may be tackled in a Cochrane Review. A prototypical situation that may elicit suspicion of publication bias is when published evidence includes a number of small studies, all of which are industry-funded (Bhandari et al 2004). For example, 14 studies of flavanoids in patients with haemorrhoids have shown apparent large benefits, but enrolled a total of only 1432 patients (i.e. each study enrolled relatively few patients) (Alonso-Coello et al 2006). The heavy involvement of sponsors in most of these studies raises questions of whether unpublished studies that suggest no benefit exist (publication bias).
A particular body of evidence can suffer from problems associated with more than one of the five factors listed here, and the greater the problems, the lower the certainty of evidence rating that should result. One could imagine a situation in which randomized trials were available, but all or virtually all of these limitations would be present, and in serious form. A very low certainty of evidence rating would result.
Table 14.2.a Further guidelines for domain 1 (of 5) in a GRADE assessment: going from assessments of risk of bias in studies to judgements about study limitations for main outcomes across studies
|
|
|
|
|
Low risk of bias | Most information is from results at low risk of bias. | Plausible bias unlikely to seriously alter the results. | No apparent limitations. | No serious limitations, do not downgrade. |
Some concerns | Most information is from results at low risk of bias or with some concerns. | Plausible bias that raises some doubt about the results. | Potential limitations are unlikely to lower confidence in the estimate of effect. | No serious limitations, do not downgrade. |
Potential limitations are likely to lower confidence in the estimate of effect. | Serious limitations, downgrade one level. | |||
High risk of bias | The proportion of information from results at high risk of bias is sufficient to affect the interpretation of results. | Plausible bias that seriously weakens confidence in the results. | Crucial limitation for one criterion, or some limitations for multiple criteria, sufficient to lower confidence in the estimate of effect. | Serious limitations, downgrade one level. |
Crucial limitation for one or more criteria sufficient to substantially lower confidence in the estimate of effect. | Very serious limitations, downgrade two levels. |
Table 14.2.b Judgements about indirectness by outcome (available in GRADEpro GDT)
| |||||
|
|
| |||
| Probably yes | Probably no | No | ||
|
|
|
|
Intervention:
Yes | Probably yes | Probably no | No |
|
|
|
|
Comparator:
Direct comparison:
Final judgement about indirectness across domains:
|
|
|
Although NRSI and downgraded randomized trials will generally yield a low rating for certainty of evidence, there will be unusual circumstances in which review authors could ‘upgrade’ such evidence to moderate or even high certainty ( Table 14.3.a ).
Review authors should report the grading of the certainty of evidence in the Results section for each outcome for which this has been performed, providing the rationale for downgrading or upgrading the evidence, and referring to the ‘Summary of findings’ table where applicable.
Table 14.3.a provides a framework and examples for how review authors can justify their judgements about the certainty of evidence in each domain. These justifications should also be included in explanatory notes to the ‘Summary of Findings’ table (see Section 14.1.6.10 ).
Chapter 15, Section 15.6 , describes in more detail how the overall GRADE assessment across all domains can be used to draw conclusions about the effects of the intervention, as well as providing implications for future research.
Table 14.3.a Framework for describing the certainty of evidence and justifying downgrading or upgrading
|
|
|
| Describe the risk of bias based on the criteria used in the risk-of-bias table. | Downgraded because of 10 randomized trials, five did not blind patients and caretakers. |
| Describe the degree of inconsistency by outcome using one or more indicators (e.g. I and P value), confidence interval overlap, difference in point estimate, between-study variance. | Not downgraded because the proportion of the variability in effect estimates that is due to true heterogeneity rather than chance is not important (I = 0%). |
| Describe if the majority of studies address the PICO – were they similar to the question posed? | Downgraded because the included studies were restricted to patients with advanced cancer. |
| Describe the number of events, and width of the confidence intervals. | The confidence intervals for the effect on mortality are consistent with both an appreciable benefit and appreciable harm and we lowered the certainty. |
| Describe the possible degree of publication bias. | 1. The funnel plot of 14 randomized trials indicated that there were several small studies that showed a small positive effect, but small studies that showed no effect or harm may have been unpublished. The certainty of the evidence was lowered. 2. There are only three small positive studies, it appears that studies showing no effect or harm have not been published. There also is for-profit interest in the intervention. The certainty of the evidence was lowered. |
| Describe the magnitude of the effect and the widths of the associate confidence intervals. | Upgraded because the RR is large: 0.3 (95% CI 0.2 to 0.4), with a sufficient number of events to be precise. |
| The studies show a clear relation with increases in the outcome of an outcome (e.g. lung cancer) with higher exposure levels. | Upgraded because the dose-response relation shows a relative risk increase of 10% in never smokers, 15% in smokers of 10 pack years and 20% in smokers of 15 pack years. |
| Describe which opposing plausible biases and confounders may have not been considered. | The estimate of effect is not controlled for the following possible confounders: smoking, degree of education, but the distribution of these factors in the studies is likely to lead to an under-estimate of the true effect. The certainty of the evidence was increased. |
Authors: Holger J Schünemann, Julian PT Higgins, Gunn E Vist, Paul Glasziou, Elie A Akl, Nicole Skoetz, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group (formerly Applicability and Recommendations Methods Group) and the Cochrane Statistical Methods Group
Acknowledgements: Andrew D Oxman contributed to earlier versions. Professor Penny Hawe contributed to the text on adverse effects in earlier versions. Jon Deeks provided helpful contributions on an earlier version of this chapter. For details of previous authors and editors of the Handbook , please refer to the Preface.
Funding: This work was in part supported by funding from the Michael G DeGroote Cochrane Canada Centre and the Ontario Ministry of Health.
Alonso-Coello P, Zhou Q, Martinez-Zapata MJ, Mills E, Heels-Ansdell D, Johanson JF, Guyatt G. Meta-analysis of flavonoids for the treatment of haemorrhoids. British Journal of Surgery 2006; 93 : 909-920.
Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Jr., Zaza S. Grading quality of evidence and strength of recommendations. BMJ 2004; 328 : 1490.
Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH. GRADE guidelines: 3. Rating the quality of evidence. Journal of Clinical Epidemiology 2011; 64 : 401-406.
Bhandari M, Busse JW, Jackowski D, Montori VM, Schünemann H, Sprague S, Mears D, Schemitsch EH, Heels-Ansdell D, Devereaux PJ. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. Canadian Medical Association Journal 2004; 170 : 477-480.
Brophy JM, Joseph L, Rouleau JL. Beta-blockers in congestive heart failure. A Bayesian meta-analysis. Annals of Internal Medicine 2001; 134 : 550-560.
Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P, Meerpohl JJ, Vandvik PO, Brozek JL, Akl EA, Bossuyt P, Churchill R, Glenton C, Rosenbaum S, Tugwell P, Welch V, Garner P, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format. Journal of Clinical Epidemiology 2016; 74 : 7-18.
Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In: Egger M, Davey Smith G, Altman DG, editors. Systematic Reviews in Health Care: Meta-analysis in Context . 2nd ed. London (UK): BMJ Publication Group; 2001. p. 313-335.
Devereaux PJ, Choi PT, Lacchetti C, Weaver B, Schünemann HJ, Haines T, Lavis JN, Grant BJ, Haslam DR, Bhandari M, Sullivan T, Cook DJ, Walter SD, Meade M, Khan H, Bhatnagar N, Guyatt GH. A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. Canadian Medical Association Journal 2002; 166 : 1399-1406.
Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Statistics in Medicine 2000; 19 : 1707-1728.
Furukawa TA, Guyatt GH, Griffith LE. Can we individualize the 'number needed to treat'? An empirical study of summary effect measures in meta-analyses. International Journal of Epidemiology 2002; 31 : 72-76.
Gibson JN, Waddell G. Surgical interventions for lumbar disc prolapse: updated Cochrane Review. Spine 2007; 32 : 1735-1747.
Guyatt G, Oxman A, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann H. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336 : 3.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology 2011a; 64 : 383-394.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM, Freyschuss B, Vist G, Jaeschke R, Williams JW, Jr., Murad MH, Sinclair D, Falck-Ytter Y, Meerpohl J, Whittington C, Thorlund K, Andrews J, Schünemann HJ. GRADE guidelines 6. Rating the quality of evidence--imprecision. Journal of Clinical Epidemiology 2011b; 64 : 1283-1293.
Iorio A, Spencer FA, Falavigna M, Alba C, Lang E, Burnand B, McGinn T, Hayden J, Williams K, Shea B, Wolff R, Kujpers T, Perel P, Vandvik PO, Glasziou P, Schünemann H, Guyatt G. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ 2015; 350 : h870.
Langendam M, Carrasco-Labra A, Santesso N, Mustafa RA, Brignardello-Petersen R, Ventresca M, Heus P, Lasserson T, Moustgaard R, Brozek J, Schünemann HJ. Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed. Journal of Clinical Epidemiology 2016; 74 : 19-27.
Levine MN, Raskob G, Landefeld S, Kearon C, Schulman S. Hemorrhagic complications of anticoagulant treatment: the Seventh ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest 2004; 126 : 287S-310S.
Orrapin S, Rerkasem K. Carotid endarterectomy for symptomatic carotid stenosis. Cochrane Database of Systematic Reviews 2017; 6 : CD001081.
Salpeter S, Greyber E, Pasternak G, Salpeter E. Risk of fatal and nonfatal lactic acidosis with metformin use in type 2 diabetes mellitus. Cochrane Database of Systematic Reviews 2007; 4 : CD002967.
Santesso N, Carrasco-Labra A, Langendam M, Brignardello-Petersen R, Mustafa RA, Heus P, Lasserson T, Opiyo N, Kunnamo I, Sinclair D, Garner P, Treweek S, Tovey D, Akl EA, Tugwell P, Brozek JL, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. Journal of Clinical Epidemiology 2016; 74 : 28-39.
Schünemann HJ, Best D, Vist G, Oxman AD, Group GW. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. Canadian Medical Association Journal 2003; 169 : 677-680.
Schünemann HJ, Jaeschke R, Cook DJ, Bria WF, El-Solh AA, Ernst A, Fahy BF, Gould MK, Horan KL, Krishnan JA, Manthous CA, Maurer JR, McNicholas WT, Oxman AD, Rubenfeld G, Turino GM, Guyatt G. An official ATS statement: grading the quality of evidence and strength of recommendations in ATS guidelines and recommendations. American Journal of Respiratory and Critical Care Medicine 2006; 174 : 605-614.
Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW, Jr., Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 2008a; 336 : 1106-1110.
Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Bossuyt P, Chang S, Muti P, Jaeschke R, Guyatt GH. GRADE: assessing the quality of evidence for diagnostic recommendations. ACP Journal Club 2008b; 149 : 2.
Schünemann HJ, Mustafa R, Brozek J. [Diagnostic accuracy and linked evidence--testing the chain]. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 2012; 106 : 153-160.
Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, Shea B, Wells G, Helfand M. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 49-62.
Schünemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? Journal of Clinical Epidemiology 2016; 75 : 6-15.
Schünemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, Morgan RL, Gartlehner G, Kunz R, Katikireddi SV, Sterne J, Higgins JPT, Guyatt G, Group GW. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. Journal of Clinical Epidemiology 2018.
Spencer-Bonilla G, Quinones AR, Montori VM, International Minimally Disruptive Medicine W. Assessing the Burden of Treatment. Journal of General Internal Medicine 2017; 32 : 1141-1145.
Spencer FA, Iorio A, You J, Murad MH, Schünemann HJ, Vandvik PO, Crowther MA, Pottie K, Lang ES, Meerpohl JJ, Falck-Ytter Y, Alonso-Coello P, Guyatt GH. Uncertainties in baseline risk estimates and confidence in treatment effects. BMJ 2012; 345 : e7401.
Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016; 355 : i4919.
Thompson DC, Rivara FP, Thompson R. Helmets for preventing head and facial injuries in bicyclists. Cochrane Database of Systematic Reviews 2000; 2 : CD001855.
Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR. Practical methods for incorporating summary time-to-event data into meta-analysis. Trials 2007; 8 .
van Dalen EC, Tierney JF, Kremer LCM. Tips and tricks for understanding and using SR results. No. 7: time‐to‐event data. Evidence-Based Child Health 2007; 2 : 1089-1090.
For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Edward barroga.
1 Department of General Education, Graduate School of Nursing Science, St. Luke’s International University, Tokyo, Japan.
2 Department of Biological Sciences, Messiah University, Mechanicsburg, PA, USA.
The development of research questions and the subsequent hypotheses are prerequisites to defining the main research purpose and specific objectives of a study. Consequently, these objectives determine the study design and research outcome. The development of research questions is a process based on knowledge of current trends, cutting-edge studies, and technological advances in the research field. Excellent research questions are focused and require a comprehensive literature search and in-depth understanding of the problem being investigated. Initially, research questions may be written as descriptive questions which could be developed into inferential questions. These questions must be specific and concise to provide a clear foundation for developing hypotheses. Hypotheses are more formal predictions about the research outcomes. These specify the possible results that may or may not be expected regarding the relationship between groups. Thus, research questions and hypotheses clarify the main purpose and specific objectives of the study, which in turn dictate the design of the study, its direction, and outcome. Studies developed from good research questions and hypotheses will have trustworthy outcomes with wide-ranging social and health implications.
Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses. 1 , 2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results. 3 , 4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the inception of novel studies and the ethical testing of ideas. 5 , 6
It is crucial to have knowledge of both quantitative and qualitative research 2 as both types of research involve writing research questions and hypotheses. 7 However, these crucial elements of research are sometimes overlooked; if not overlooked, then framed without the forethought and meticulous attention it needs. Planning and careful consideration are needed when developing quantitative or qualitative research, particularly when conceptualizing research questions and hypotheses. 4
There is a continuing need to support researchers in the creation of innovative research questions and hypotheses, as well as for journal articles that carefully review these elements. 1 When research questions and hypotheses are not carefully thought of, unethical studies and poor outcomes usually ensue. Carefully formulated research questions and hypotheses define well-founded objectives, which in turn determine the appropriate design, course, and outcome of the study. This article then aims to discuss in detail the various aspects of crafting research questions and hypotheses, with the goal of guiding researchers as they develop their own. Examples from the authors and peer-reviewed scientific articles in the healthcare field are provided to illustrate key points.
A research question is what a study aims to answer after data analysis and interpretation. The answer is written in length in the discussion section of the paper. Thus, the research question gives a preview of the different parts and variables of the study meant to address the problem posed in the research question. 1 An excellent research question clarifies the research writing while facilitating understanding of the research topic, objective, scope, and limitations of the study. 5
On the other hand, a research hypothesis is an educated statement of an expected outcome. This statement is based on background research and current knowledge. 8 , 9 The research hypothesis makes a specific prediction about a new phenomenon 10 or a formal statement on the expected relationship between an independent variable and a dependent variable. 3 , 11 It provides a tentative answer to the research question to be tested or explored. 4
Hypotheses employ reasoning to predict a theory-based outcome. 10 These can also be developed from theories by focusing on components of theories that have not yet been observed. 10 The validity of hypotheses is often based on the testability of the prediction made in a reproducible experiment. 8
Conversely, hypotheses can also be rephrased as research questions. Several hypotheses based on existing theories and knowledge may be needed to answer a research question. Developing ethical research questions and hypotheses creates a research design that has logical relationships among variables. These relationships serve as a solid foundation for the conduct of the study. 4 , 11 Haphazardly constructed research questions can result in poorly formulated hypotheses and improper study designs, leading to unreliable results. Thus, the formulations of relevant research questions and verifiable hypotheses are crucial when beginning research. 12
Excellent research questions are specific and focused. These integrate collective data and observations to confirm or refute the subsequent hypotheses. Well-constructed hypotheses are based on previous reports and verify the research context. These are realistic, in-depth, sufficiently complex, and reproducible. More importantly, these hypotheses can be addressed and tested. 13
There are several characteristics of well-developed hypotheses. Good hypotheses are 1) empirically testable 7 , 10 , 11 , 13 ; 2) backed by preliminary evidence 9 ; 3) testable by ethical research 7 , 9 ; 4) based on original ideas 9 ; 5) have evidenced-based logical reasoning 10 ; and 6) can be predicted. 11 Good hypotheses can infer ethical and positive implications, indicating the presence of a relationship or effect relevant to the research theme. 7 , 11 These are initially developed from a general theory and branch into specific hypotheses by deductive reasoning. In the absence of a theory to base the hypotheses, inductive reasoning based on specific observations or findings form more general hypotheses. 10
Research questions and hypotheses are developed according to the type of research, which can be broadly classified into quantitative and qualitative research. We provide a summary of the types of research questions and hypotheses under quantitative and qualitative research categories in Table 1 .
Quantitative research questions | Quantitative research hypotheses |
---|---|
Descriptive research questions | Simple hypothesis |
Comparative research questions | Complex hypothesis |
Relationship research questions | Directional hypothesis |
Non-directional hypothesis | |
Associative hypothesis | |
Causal hypothesis | |
Null hypothesis | |
Alternative hypothesis | |
Working hypothesis | |
Statistical hypothesis | |
Logical hypothesis | |
Hypothesis-testing | |
Qualitative research questions | Qualitative research hypotheses |
Contextual research questions | Hypothesis-generating |
Descriptive research questions | |
Evaluation research questions | |
Explanatory research questions | |
Exploratory research questions | |
Generative research questions | |
Ideological research questions | |
Ethnographic research questions | |
Phenomenological research questions | |
Grounded theory questions | |
Qualitative case study questions |
In quantitative research, research questions inquire about the relationships among variables being investigated and are usually framed at the start of the study. These are precise and typically linked to the subject population, dependent and independent variables, and research design. 1 Research questions may also attempt to describe the behavior of a population in relation to one or more variables, or describe the characteristics of variables to be measured ( descriptive research questions ). 1 , 5 , 14 These questions may also aim to discover differences between groups within the context of an outcome variable ( comparative research questions ), 1 , 5 , 14 or elucidate trends and interactions among variables ( relationship research questions ). 1 , 5 We provide examples of descriptive, comparative, and relationship research questions in quantitative research in Table 2 .
Quantitative research questions | |
---|---|
Descriptive research question | |
- Measures responses of subjects to variables | |
- Presents variables to measure, analyze, or assess | |
What is the proportion of resident doctors in the hospital who have mastered ultrasonography (response of subjects to a variable) as a diagnostic technique in their clinical training? | |
Comparative research question | |
- Clarifies difference between one group with outcome variable and another group without outcome variable | |
Is there a difference in the reduction of lung metastasis in osteosarcoma patients who received the vitamin D adjunctive therapy (group with outcome variable) compared with osteosarcoma patients who did not receive the vitamin D adjunctive therapy (group without outcome variable)? | |
- Compares the effects of variables | |
How does the vitamin D analogue 22-Oxacalcitriol (variable 1) mimic the antiproliferative activity of 1,25-Dihydroxyvitamin D (variable 2) in osteosarcoma cells? | |
Relationship research question | |
- Defines trends, association, relationships, or interactions between dependent variable and independent variable | |
Is there a relationship between the number of medical student suicide (dependent variable) and the level of medical student stress (independent variable) in Japan during the first wave of the COVID-19 pandemic? |
In quantitative research, hypotheses predict the expected relationships among variables. 15 Relationships among variables that can be predicted include 1) between a single dependent variable and a single independent variable ( simple hypothesis ) or 2) between two or more independent and dependent variables ( complex hypothesis ). 4 , 11 Hypotheses may also specify the expected direction to be followed and imply an intellectual commitment to a particular outcome ( directional hypothesis ) 4 . On the other hand, hypotheses may not predict the exact direction and are used in the absence of a theory, or when findings contradict previous studies ( non-directional hypothesis ). 4 In addition, hypotheses can 1) define interdependency between variables ( associative hypothesis ), 4 2) propose an effect on the dependent variable from manipulation of the independent variable ( causal hypothesis ), 4 3) state a negative relationship between two variables ( null hypothesis ), 4 , 11 , 15 4) replace the working hypothesis if rejected ( alternative hypothesis ), 15 explain the relationship of phenomena to possibly generate a theory ( working hypothesis ), 11 5) involve quantifiable variables that can be tested statistically ( statistical hypothesis ), 11 6) or express a relationship whose interlinks can be verified logically ( logical hypothesis ). 11 We provide examples of simple, complex, directional, non-directional, associative, causal, null, alternative, working, statistical, and logical hypotheses in quantitative research, as well as the definition of quantitative hypothesis-testing research in Table 3 .
Quantitative research hypotheses | |
---|---|
Simple hypothesis | |
- Predicts relationship between single dependent variable and single independent variable | |
If the dose of the new medication (single independent variable) is high, blood pressure (single dependent variable) is lowered. | |
Complex hypothesis | |
- Foretells relationship between two or more independent and dependent variables | |
The higher the use of anticancer drugs, radiation therapy, and adjunctive agents (3 independent variables), the higher would be the survival rate (1 dependent variable). | |
Directional hypothesis | |
- Identifies study direction based on theory towards particular outcome to clarify relationship between variables | |
Privately funded research projects will have a larger international scope (study direction) than publicly funded research projects. | |
Non-directional hypothesis | |
- Nature of relationship between two variables or exact study direction is not identified | |
- Does not involve a theory | |
Women and men are different in terms of helpfulness. (Exact study direction is not identified) | |
Associative hypothesis | |
- Describes variable interdependency | |
- Change in one variable causes change in another variable | |
A larger number of people vaccinated against COVID-19 in the region (change in independent variable) will reduce the region’s incidence of COVID-19 infection (change in dependent variable). | |
Causal hypothesis | |
- An effect on dependent variable is predicted from manipulation of independent variable | |
A change into a high-fiber diet (independent variable) will reduce the blood sugar level (dependent variable) of the patient. | |
Null hypothesis | |
- A negative statement indicating no relationship or difference between 2 variables | |
There is no significant difference in the severity of pulmonary metastases between the new drug (variable 1) and the current drug (variable 2). | |
Alternative hypothesis | |
- Following a null hypothesis, an alternative hypothesis predicts a relationship between 2 study variables | |
The new drug (variable 1) is better on average in reducing the level of pain from pulmonary metastasis than the current drug (variable 2). | |
Working hypothesis | |
- A hypothesis that is initially accepted for further research to produce a feasible theory | |
Dairy cows fed with concentrates of different formulations will produce different amounts of milk. | |
Statistical hypothesis | |
- Assumption about the value of population parameter or relationship among several population characteristics | |
- Validity tested by a statistical experiment or analysis | |
The mean recovery rate from COVID-19 infection (value of population parameter) is not significantly different between population 1 and population 2. | |
There is a positive correlation between the level of stress at the workplace and the number of suicides (population characteristics) among working people in Japan. | |
Logical hypothesis | |
- Offers or proposes an explanation with limited or no extensive evidence | |
If healthcare workers provide more educational programs about contraception methods, the number of adolescent pregnancies will be less. | |
Hypothesis-testing (Quantitative hypothesis-testing research) | |
- Quantitative research uses deductive reasoning. | |
- This involves the formation of a hypothesis, collection of data in the investigation of the problem, analysis and use of the data from the investigation, and drawing of conclusions to validate or nullify the hypotheses. |
Unlike research questions in quantitative research, research questions in qualitative research are usually continuously reviewed and reformulated. The central question and associated subquestions are stated more than the hypotheses. 15 The central question broadly explores a complex set of factors surrounding the central phenomenon, aiming to present the varied perspectives of participants. 15
There are varied goals for which qualitative research questions are developed. These questions can function in several ways, such as to 1) identify and describe existing conditions ( contextual research question s); 2) describe a phenomenon ( descriptive research questions ); 3) assess the effectiveness of existing methods, protocols, theories, or procedures ( evaluation research questions ); 4) examine a phenomenon or analyze the reasons or relationships between subjects or phenomena ( explanatory research questions ); or 5) focus on unknown aspects of a particular topic ( exploratory research questions ). 5 In addition, some qualitative research questions provide new ideas for the development of theories and actions ( generative research questions ) or advance specific ideologies of a position ( ideological research questions ). 1 Other qualitative research questions may build on a body of existing literature and become working guidelines ( ethnographic research questions ). Research questions may also be broadly stated without specific reference to the existing literature or a typology of questions ( phenomenological research questions ), may be directed towards generating a theory of some process ( grounded theory questions ), or may address a description of the case and the emerging themes ( qualitative case study questions ). 15 We provide examples of contextual, descriptive, evaluation, explanatory, exploratory, generative, ideological, ethnographic, phenomenological, grounded theory, and qualitative case study research questions in qualitative research in Table 4 , and the definition of qualitative hypothesis-generating research in Table 5 .
Qualitative research questions | |
---|---|
Contextual research question | |
- Ask the nature of what already exists | |
- Individuals or groups function to further clarify and understand the natural context of real-world problems | |
What are the experiences of nurses working night shifts in healthcare during the COVID-19 pandemic? (natural context of real-world problems) | |
Descriptive research question | |
- Aims to describe a phenomenon | |
What are the different forms of disrespect and abuse (phenomenon) experienced by Tanzanian women when giving birth in healthcare facilities? | |
Evaluation research question | |
- Examines the effectiveness of existing practice or accepted frameworks | |
How effective are decision aids (effectiveness of existing practice) in helping decide whether to give birth at home or in a healthcare facility? | |
Explanatory research question | |
- Clarifies a previously studied phenomenon and explains why it occurs | |
Why is there an increase in teenage pregnancy (phenomenon) in Tanzania? | |
Exploratory research question | |
- Explores areas that have not been fully investigated to have a deeper understanding of the research problem | |
What factors affect the mental health of medical students (areas that have not yet been fully investigated) during the COVID-19 pandemic? | |
Generative research question | |
- Develops an in-depth understanding of people’s behavior by asking ‘how would’ or ‘what if’ to identify problems and find solutions | |
How would the extensive research experience of the behavior of new staff impact the success of the novel drug initiative? | |
Ideological research question | |
- Aims to advance specific ideas or ideologies of a position | |
Are Japanese nurses who volunteer in remote African hospitals able to promote humanized care of patients (specific ideas or ideologies) in the areas of safe patient environment, respect of patient privacy, and provision of accurate information related to health and care? | |
Ethnographic research question | |
- Clarifies peoples’ nature, activities, their interactions, and the outcomes of their actions in specific settings | |
What are the demographic characteristics, rehabilitative treatments, community interactions, and disease outcomes (nature, activities, their interactions, and the outcomes) of people in China who are suffering from pneumoconiosis? | |
Phenomenological research question | |
- Knows more about the phenomena that have impacted an individual | |
What are the lived experiences of parents who have been living with and caring for children with a diagnosis of autism? (phenomena that have impacted an individual) | |
Grounded theory question | |
- Focuses on social processes asking about what happens and how people interact, or uncovering social relationships and behaviors of groups | |
What are the problems that pregnant adolescents face in terms of social and cultural norms (social processes), and how can these be addressed? | |
Qualitative case study question | |
- Assesses a phenomenon using different sources of data to answer “why” and “how” questions | |
- Considers how the phenomenon is influenced by its contextual situation. | |
How does quitting work and assuming the role of a full-time mother (phenomenon assessed) change the lives of women in Japan? |
Qualitative research hypotheses | |
---|---|
Hypothesis-generating (Qualitative hypothesis-generating research) | |
- Qualitative research uses inductive reasoning. | |
- This involves data collection from study participants or the literature regarding a phenomenon of interest, using the collected data to develop a formal hypothesis, and using the formal hypothesis as a framework for testing the hypothesis. | |
- Qualitative exploratory studies explore areas deeper, clarifying subjective experience and allowing formulation of a formal hypothesis potentially testable in a future quantitative approach. |
Qualitative studies usually pose at least one central research question and several subquestions starting with How or What . These research questions use exploratory verbs such as explore or describe . These also focus on one central phenomenon of interest, and may mention the participants and research site. 15
Hypotheses in qualitative research are stated in the form of a clear statement concerning the problem to be investigated. Unlike in quantitative research where hypotheses are usually developed to be tested, qualitative research can lead to both hypothesis-testing and hypothesis-generating outcomes. 2 When studies require both quantitative and qualitative research questions, this suggests an integrative process between both research methods wherein a single mixed-methods research question can be developed. 1
Research questions followed by hypotheses should be developed before the start of the study. 1 , 12 , 14 It is crucial to develop feasible research questions on a topic that is interesting to both the researcher and the scientific community. This can be achieved by a meticulous review of previous and current studies to establish a novel topic. Specific areas are subsequently focused on to generate ethical research questions. The relevance of the research questions is evaluated in terms of clarity of the resulting data, specificity of the methodology, objectivity of the outcome, depth of the research, and impact of the study. 1 , 5 These aspects constitute the FINER criteria (i.e., Feasible, Interesting, Novel, Ethical, and Relevant). 1 Clarity and effectiveness are achieved if research questions meet the FINER criteria. In addition to the FINER criteria, Ratan et al. described focus, complexity, novelty, feasibility, and measurability for evaluating the effectiveness of research questions. 14
The PICOT and PEO frameworks are also used when developing research questions. 1 The following elements are addressed in these frameworks, PICOT: P-population/patients/problem, I-intervention or indicator being studied, C-comparison group, O-outcome of interest, and T-timeframe of the study; PEO: P-population being studied, E-exposure to preexisting conditions, and O-outcome of interest. 1 Research questions are also considered good if these meet the “FINERMAPS” framework: Feasible, Interesting, Novel, Ethical, Relevant, Manageable, Appropriate, Potential value/publishable, and Systematic. 14
As we indicated earlier, research questions and hypotheses that are not carefully formulated result in unethical studies or poor outcomes. To illustrate this, we provide some examples of ambiguous research question and hypotheses that result in unclear and weak research objectives in quantitative research ( Table 6 ) 16 and qualitative research ( Table 7 ) 17 , and how to transform these ambiguous research question(s) and hypothesis(es) into clear and good statements.
Variables | Unclear and weak statement (Statement 1) | Clear and good statement (Statement 2) | Points to avoid |
---|---|---|---|
Research question | Which is more effective between smoke moxibustion and smokeless moxibustion? | “Moreover, regarding smoke moxibustion versus smokeless moxibustion, it remains unclear which is more effective, safe, and acceptable to pregnant women, and whether there is any difference in the amount of heat generated.” | 1) Vague and unfocused questions |
2) Closed questions simply answerable by yes or no | |||
3) Questions requiring a simple choice | |||
Hypothesis | The smoke moxibustion group will have higher cephalic presentation. | “Hypothesis 1. The smoke moxibustion stick group (SM group) and smokeless moxibustion stick group (-SLM group) will have higher rates of cephalic presentation after treatment than the control group. | 1) Unverifiable hypotheses |
Hypothesis 2. The SM group and SLM group will have higher rates of cephalic presentation at birth than the control group. | 2) Incompletely stated groups of comparison | ||
Hypothesis 3. There will be no significant differences in the well-being of the mother and child among the three groups in terms of the following outcomes: premature birth, premature rupture of membranes (PROM) at < 37 weeks, Apgar score < 7 at 5 min, umbilical cord blood pH < 7.1, admission to neonatal intensive care unit (NICU), and intrauterine fetal death.” | 3) Insufficiently described variables or outcomes | ||
Research objective | To determine which is more effective between smoke moxibustion and smokeless moxibustion. | “The specific aims of this pilot study were (a) to compare the effects of smoke moxibustion and smokeless moxibustion treatments with the control group as a possible supplement to ECV for converting breech presentation to cephalic presentation and increasing adherence to the newly obtained cephalic position, and (b) to assess the effects of these treatments on the well-being of the mother and child.” | 1) Poor understanding of the research question and hypotheses |
2) Insufficient description of population, variables, or study outcomes |
a These statements were composed for comparison and illustrative purposes only.
b These statements are direct quotes from Higashihara and Horiuchi. 16
Variables | Unclear and weak statement (Statement 1) | Clear and good statement (Statement 2) | Points to avoid |
---|---|---|---|
Research question | Does disrespect and abuse (D&A) occur in childbirth in Tanzania? | How does disrespect and abuse (D&A) occur and what are the types of physical and psychological abuses observed in midwives’ actual care during facility-based childbirth in urban Tanzania? | 1) Ambiguous or oversimplistic questions |
2) Questions unverifiable by data collection and analysis | |||
Hypothesis | Disrespect and abuse (D&A) occur in childbirth in Tanzania. | Hypothesis 1: Several types of physical and psychological abuse by midwives in actual care occur during facility-based childbirth in urban Tanzania. | 1) Statements simply expressing facts |
Hypothesis 2: Weak nursing and midwifery management contribute to the D&A of women during facility-based childbirth in urban Tanzania. | 2) Insufficiently described concepts or variables | ||
Research objective | To describe disrespect and abuse (D&A) in childbirth in Tanzania. | “This study aimed to describe from actual observations the respectful and disrespectful care received by women from midwives during their labor period in two hospitals in urban Tanzania.” | 1) Statements unrelated to the research question and hypotheses |
2) Unattainable or unexplorable objectives |
a This statement is a direct quote from Shimoda et al. 17
The other statements were composed for comparison and illustrative purposes only.
To construct effective research questions and hypotheses, it is very important to 1) clarify the background and 2) identify the research problem at the outset of the research, within a specific timeframe. 9 Then, 3) review or conduct preliminary research to collect all available knowledge about the possible research questions by studying theories and previous studies. 18 Afterwards, 4) construct research questions to investigate the research problem. Identify variables to be accessed from the research questions 4 and make operational definitions of constructs from the research problem and questions. Thereafter, 5) construct specific deductive or inductive predictions in the form of hypotheses. 4 Finally, 6) state the study aims . This general flow for constructing effective research questions and hypotheses prior to conducting research is shown in Fig. 1 .
Research questions are used more frequently in qualitative research than objectives or hypotheses. 3 These questions seek to discover, understand, explore or describe experiences by asking “What” or “How.” The questions are open-ended to elicit a description rather than to relate variables or compare groups. The questions are continually reviewed, reformulated, and changed during the qualitative study. 3 Research questions are also used more frequently in survey projects than hypotheses in experiments in quantitative research to compare variables and their relationships.
Hypotheses are constructed based on the variables identified and as an if-then statement, following the template, ‘If a specific action is taken, then a certain outcome is expected.’ At this stage, some ideas regarding expectations from the research to be conducted must be drawn. 18 Then, the variables to be manipulated (independent) and influenced (dependent) are defined. 4 Thereafter, the hypothesis is stated and refined, and reproducible data tailored to the hypothesis are identified, collected, and analyzed. 4 The hypotheses must be testable and specific, 18 and should describe the variables and their relationships, the specific group being studied, and the predicted research outcome. 18 Hypotheses construction involves a testable proposition to be deduced from theory, and independent and dependent variables to be separated and measured separately. 3 Therefore, good hypotheses must be based on good research questions constructed at the start of a study or trial. 12
In summary, research questions are constructed after establishing the background of the study. Hypotheses are then developed based on the research questions. Thus, it is crucial to have excellent research questions to generate superior hypotheses. In turn, these would determine the research objectives and the design of the study, and ultimately, the outcome of the research. 12 Algorithms for building research questions and hypotheses are shown in Fig. 2 for quantitative research and in Fig. 3 for qualitative research.
Research questions and hypotheses are crucial components to any type of research, whether quantitative or qualitative. These questions should be developed at the very beginning of the study. Excellent research questions lead to superior hypotheses, which, like a compass, set the direction of research, and can often determine the successful conduct of the study. Many research studies have floundered because the development of research questions and subsequent hypotheses was not given the thought and meticulous attention needed. The development of research questions and hypotheses is an iterative process based on extensive knowledge of the literature and insightful grasp of the knowledge gap. Focused, concise, and specific research questions provide a strong foundation for constructing hypotheses which serve as formal predictions about the research outcomes. Research questions and hypotheses are crucial elements of research that should not be overlooked. They should be carefully thought of and constructed when planning research. This avoids unethical studies and poor outcomes by defining well-founded objectives that determine the design, course, and outcome of the study.
Disclosure: The authors have no potential conflicts of interest to disclose.
Author Contributions:
Home » Research Summary – Structure, Examples and Writing Guide
Table of Contents
Definition:
A research summary is a brief and concise overview of a research project or study that highlights its key findings, main points, and conclusions. It typically includes a description of the research problem, the research methods used, the results obtained, and the implications or significance of the findings. It is often used as a tool to quickly communicate the main findings of a study to other researchers, stakeholders, or decision-makers.
The Structure of a Research Summary typically include:
Here are the steps you can follow to write a research summary:
Here is an example of a research summary:
Title: The Effects of Yoga on Mental Health: A Meta-Analysis
Introduction: This meta-analysis examines the effects of yoga on mental health. The study aimed to investigate whether yoga practice can improve mental health outcomes such as anxiety, depression, stress, and quality of life.
Methodology : The study analyzed data from 14 randomized controlled trials that investigated the effects of yoga on mental health outcomes. The sample included a total of 862 participants. The yoga interventions varied in length and frequency, ranging from four to twelve weeks, with sessions lasting from 45 to 90 minutes.
Results : The meta-analysis found that yoga practice significantly improved mental health outcomes. Participants who practiced yoga showed a significant reduction in anxiety and depression symptoms, as well as stress levels. Quality of life also improved in those who practiced yoga.
Discussion : The findings of this study suggest that yoga can be an effective intervention for improving mental health outcomes. The study supports the growing body of evidence that suggests that yoga can have a positive impact on mental health. Limitations of the study include the variability of the yoga interventions, which may affect the generalizability of the findings.
Conclusion : Overall, the findings of this meta-analysis support the use of yoga as an effective intervention for improving mental health outcomes. Further research is needed to determine the optimal length and frequency of yoga interventions for different populations.
References :
The purpose of a research summary is to provide a brief overview of a research project or study, including its main points, findings, and conclusions. The summary allows readers to quickly understand the essential aspects of the research without having to read the entire article or study.
Research summaries serve several purposes, including:
The following are some of the key characteristics of a research summary:
Here are some situations when it may be appropriate to write a research summary:
Research summaries offer several advantages, including:
Limitations of the Research Summary are as follows:
Researcher, Academic Writer, Web developer
The discussion section of a research paper analyzes and interprets the findings, provides context, compares them with previous studies, identifies limitations, and suggests future research directions.
Updated on September 15, 2023
Structure your discussion section right, and you’ll be cited more often while doing a greater service to the scientific community. So, what actually goes into the discussion section? And how do you write it?
The discussion section of your research paper is where you let the reader know how your study is positioned in the literature, what to take away from your paper, and how your work helps them. It can also include your conclusions and suggestions for future studies.
First, we’ll define all the parts of your discussion paper, and then look into how to write a strong, effective discussion section for your paper or manuscript.
The discussion section comes later in your paper, following the introduction, methods, and results. The discussion sets up your study’s conclusions. Its main goals are to present, interpret, and provide a context for your results.
The discussion section provides an analysis and interpretation of the findings, compares them with previous studies, identifies limitations, and suggests future directions for research.
This section combines information from the preceding parts of your paper into a coherent story. By this point, the reader already knows why you did your study (introduction), how you did it (methods), and what happened (results). In the discussion, you’ll help the reader connect the ideas from these sections.
The discussion provides context and interpretations for the results. It also answers the questions posed in the introduction. While the results section describes your findings, the discussion explains what they say. This is also where you can describe the impact or implications of your research.
Most research studies aim to answer a question, replicate a finding, or address limitations in the literature. These goals are first described in the introduction. However, in the discussion section, the author can refer back to them to explain how the study's objective was achieved.
The discussion can also describe the effect of your findings on research or practice. How are your results significant for readers, other researchers, or policymakers?
A complete and effective discussion section should at least touch on the points described below.
The discussion should begin with a brief factual summary of the results. Concisely overview the main results you obtained.
Your results section described a list of findings, but what message do they send when you look at them all together?
Your findings were detailed in the results section, so there’s no need to repeat them here, but do provide at least a few highlights. This will help refresh the reader’s memory and help them focus on the big picture.
Read the first paragraph of the discussion section in this article (PDF) for an example of how to start this part of your paper. Notice how the authors break down their results and follow each description sentence with an explanation of why each finding is relevant.
Following a clear and direct writing style is especially important in the discussion section. After all, this is where you will make some of the most impactful points in your paper. While the results section often contains technical vocabulary, such as statistical terms, the discussion section lets you describe your findings more clearly.
Once you’ve given your reader an overview of your results, you need to interpret those results. In other words, what do your results mean? Discuss the findings’ implications and significance in relation to your research question or hypothesis.
Look into your findings and explore what’s behind them or what may have caused them. If your introduction cited theories or studies that could explain your findings, use these sources as a basis to discuss your results.
For example, look at the second paragraph in the discussion section of this article on waggling honey bees. Here, the authors explore their results based on information from the literature.
Sometimes, your findings are not what you expect. Here’s where you describe this and try to find a reason for it. Could it be because of the method you used? Does it have something to do with the variables analyzed? Comparing your methods with those of other similar studies can help with this task.
Refer to related studies to place your research in a larger context and the literature. Compare and contrast your findings with existing literature, highlighting similarities, differences, and/or contradictions.
Studies with similar findings to yours can be cited to show the strength of your findings. Information from these studies can also be used to help explain your results. Differences between your findings and others in the literature can also be discussed here.
If you have more than one objective in your study or many key findings, you can dedicate a separate section to each of these. Here’s an example of this approach. You can see that the discussion section is divided into topics and even has a separate heading for each of them.
Many journals require you to include the limitations of your study in the discussion. Even if they don’t, there are good reasons to mention these in your paper.
A study’s limitations are points to be improved upon in future research. While some of these may be flaws in your method, many may be due to factors you couldn’t predict.
Examples include time constraints or small sample sizes. Pointing this out will help future researchers avoid or address these issues. This part of the discussion can also include any attempts you have made to reduce the impact of these limitations, as in this study .
Pointing out the limitations of your study demonstrates transparency. It also shows that you know your methods well and can conduct a critical assessment of them.
The final paragraph of the discussion section should contain the take-home messages for your study. It can also cite the “strong points” of your study, to contrast with the limitations section.
Remind the reader what your hypothesis was before you conducted the study.
Identify your main findings and describe how they relate to your hypothesis.
Were you able to answer your research question? Or address a gap in the literature?
Describe the impact that your results may have on the topic of study. Your results may show, for instance, that there are still limitations in the literature for future studies to address. There may be a need for studies that extend your findings in a specific way. You also may need additional research to corroborate your findings.
This fictitious example covers all the aspects discussed above. Your actual discussion section will probably be much longer, but you can read this to get an idea of everything your discussion should cover.
Our results showed that the presence of cats in a household is associated with higher levels of perceived happiness by its human occupants. These findings support our hypothesis and demonstrate the association between pet ownership and well-being.
The present findings align with those of Bao and Schreer (2016) and Hardie et al. (2023), who observed greater life satisfaction in pet owners relative to non-owners. Although the present study did not directly evaluate life satisfaction, this factor may explain the association between happiness and cat ownership observed in our sample.
Our findings must be interpreted in light of some limitations, such as the focus on cat ownership only rather than pets as a whole. This may limit the generalizability of our results.
Nevertheless, this study had several strengths. These include its strict exclusion criteria and use of a standardized assessment instrument to investigate the relationships between pets and owners. These attributes bolster the accuracy of our results and reduce the influence of confounding factors, increasing the strength of our conclusions. Future studies may examine the factors that mediate the association between pet ownership and happiness to better comprehend this phenomenon.
This brief discussion begins with a quick summary of the results and hypothesis. The next paragraph cites previous research and compares its findings to those of this study. Information from previous studies is also used to help interpret the findings. After discussing the results of the study, some limitations are pointed out. The paper also explains why these limitations may influence the interpretation of results. Then, final conclusions are drawn based on the study, and directions for future research are suggested.
If you find writing in scientific English challenging, the discussion and conclusions are often the hardest parts of the paper to write. That’s because you’re not just listing up studies, methods, and outcomes. You’re actually expressing your thoughts and interpretations in words.
Always give it your best, but sometimes a helping hand can, well, help. Getting a professional edit can help clarify your work’s importance while improving the English used to explain it. When readers know the value of your work, they’ll cite it. We’ll assign your study to an expert editor knowledgeable in your area of research. Their work will clarify your discussion, helping it to tell your story. Find out more about AJE Editing.
Adam Goulston, PsyD, MS, MBA, MISD, ELS
Science Marketing Consultant
See our "Privacy Policy"
Pair your Premium Editing with our add-on service Presubmission Review for an overall assessment of your manuscript.
Medical Editing Services
Select Page
Posted by Rene Tetzner | Sep 2, 2021 | Paper Writing Advice | 0 |
How To Write the Findings Section of a Research Paper Each research project is unique, so it is natural for one researcher to make use of somewhat different strategies than another when it comes to designing and writing the section of a research paper dedicated to findings. The academic or scientific discipline of the research, the field of specialisation, the particular author or authors, the targeted journal or other publisher and the editor making the decisions about publication can all have a significant impact. The practical steps outlined below can be effectively applied to writing about the findings of most advanced research, however, and will prove especially helpful for early-career scholars who are preparing a research paper for a first publication.
Step 1 : Consult the guidelines or instructions that the targeted journal (or other publisher) provides for authors and read research papers it has already published, particularly ones similar in topic, methods or results to your own. The guidelines will generally outline specific requirements for the results or findings section, and the published articles will provide sound examples of successful approaches. Watch particularly for length limitations and restrictions on content. Interpretation, for instance, is usually reserved for a later discussion section, though not always – qualitative research papers often combine findings and interpretation. Background information and descriptions of methods, on the other hand, almost always appear in earlier sections of a research paper. In most cases it is appropriate in a findings section to offer basic comparisons between the results of your study and those of other studies, but knowing exactly what the journal wants in the report of research findings is essential. Learning as much as you can about the journal’s aims and scope as well as the interests of its readers is invaluable as well.
Step 2 : Reflect at some length on your research results in relation to the journal’s requirements while planning the findings section of your paper. Choose for particular focus experimental results and other research discoveries that are particularly relevant to your research questions and objectives, and include them even if they are unexpected or do not support your ideas and hypotheses. Streamline and clarify your report, especially if it is long and complex, by using subheadings that will help you avoid excessive and peripheral details as you write and also help your reader understand and remember your findings. Consider appendices for raw data that might interest specialists but prove too long or distracting for other readers. The opening paragraph of a findings section often restates research questions or aims to refocus the reader’s attention, and it is always wise to summarise key findings at the end of the section, providing a smooth intellectual transition to the interpretation and discussion that follows in most research papers. There are many effective ways in which to organise research findings. The structure of your findings section might be determined by your research questions and hypotheses or match the arrangement of your methods section. A chronological order or hierarchy of importance or meaningful grouping of main themes or categories might prove effective. It may be best to present all the relevant findings and then explain them and your analysis of them, or explaining the results of each trial or test immediately after reporting it may render the material clearer and more comprehensible for your readers. Keep your audience, your most important evidence and your research goals in mind.
Step 3 : Design effective visual presentations of your research results to enhance the textual report of your findings. Tables of various styles and figures of all kinds such as graphs, maps and photos are used in reporting research findings, but do check the journal guidelines for instructions on the number of visual aids allowed, any required design elements and the preferred formats for numbering, labelling and placement in the manuscript. As a general rule, tables and figures should be numbered according to first mention in the main text of the paper, and each one should be clearly introduced and explained at least briefly in that text so that readers know what is presented and what they are expected to see in a particular visual element. Tables and figures should also be self-explanatory, however, so their design should include all definitions and other information necessary for a reader to understand the findings you intend to show without returning to your text. If you construct your tables and figures before drafting your findings section, they can serve as focal points to help you tell a clear and informative story about your findings and avoid unnecessary repetition. Some authors will even work on tables and figures before organising the findings section (Step 2), which can be an extremely effective approach, but it is important to remember that the textual report of findings remains primary. Visual aids can clarify and enrich the text, but they cannot take its place.
Step 4 : Write your findings section in a factual and objective manner. The goal is to communicate information – in some cases a great deal of complex information – as clearly, accurately and precisely as possible, so well-constructed sentences that maintain a simple structure will be far more effective than convoluted phrasing and expressions. The active voice is often recommended by publishers and the authors of writing manuals, and the past tense is appropriate because the research has already been done. Make sure your grammar, spelling and punctuation are correct and effective so that you are conveying the meaning you intend. Statements that are vague, imprecise or ambiguous will often confuse and mislead readers, and a verbose style will add little more than padding while wasting valuable words that might be put to far better use in clear and logical explanations. Some specialised terminology may be required when reporting findings, but anything potentially unclear or confusing that has not already been defined earlier in the paper should be clarified for readers, and the same principle applies to unusual or nonstandard abbreviations. Your readers will want to understand what you are reporting about your results, not waste time looking up terms simply to understand what you are saying. A logical approach to organising your findings section (Step 2) will help you tell a logical story about your research results as you explain, highlight, offer analysis and summarise the information necessary for readers to understand the discussion section that follows.
Step 5 : Review the draft of your findings section and edit and revise until it reports your key findings exactly as you would have them presented to your readers. Check for accuracy and consistency in data across the section as a whole and all its visual elements. Read your prose aloud to catch language errors, awkward phrases and abrupt transitions. Ensure that the order in which you have presented results is the best order for focussing readers on your research objectives and preparing them for the interpretations, speculations, recommendations and other elements of the discussion that you are planning. This will involve looking back over the paper’s introductory and background material as well as anticipating the discussion and conclusion sections, and this is precisely the right point in the process for reviewing and reflecting. Your research results have taken considerable time to obtain and analyse, so a little more time to stand back and take in the wider view from the research door you have opened is a wise investment. The opinions of any additional readers you can recruit, whether they are professional mentors and colleagues or family and friends, will often prove invaluable as well.
Journal editing.
Journal article editing services
PhD thesis editing services
Manuscript editing.
Manuscript editing services
Expert editing for all papers
Research paper editing services
Professional book editing services
How To Write the Findings Section of a Research Paper These five steps will help you write a clear & interesting findings section for a research paper
September 6, 2021
August 30, 2021
September 7, 2021
August 27, 2021
Run a free plagiarism check in 10 minutes, generate accurate citations for free.
Published on October 30, 2022 by Shona McCombes . Revised on October 19, 2023.
The research question is one of the most important parts of your research paper , thesis or dissertation . It’s important to spend some time assessing and refining your question before you get started.
The exact form of your question will depend on a few things, such as the length of your project, the type of research you’re conducting, the topic , and the research problem . However, all research questions should be focused, specific, and relevant to a timely social or scholarly issue.
Once you’ve read our guide on how to write a research question , you can use these examples to craft your own.
Research question | Explanation |
---|---|
The first question is not enough. The second question is more , using . | |
Starting with “why” often means that your question is not enough: there are too many possible answers. By targeting just one aspect of the problem, the second question offers a clear path for research. | |
The first question is too broad and subjective: there’s no clear criteria for what counts as “better.” The second question is much more . It uses clearly defined terms and narrows its focus to a specific population. | |
It is generally not for academic research to answer broad normative questions. The second question is more specific, aiming to gain an understanding of possible solutions in order to make informed recommendations. | |
The first question is too simple: it can be answered with a simple yes or no. The second question is , requiring in-depth investigation and the development of an original argument. | |
The first question is too broad and not very . The second question identifies an underexplored aspect of the topic that requires investigation of various to answer. | |
The first question is not enough: it tries to address two different (the quality of sexual health services and LGBT support services). Even though the two issues are related, it’s not clear how the research will bring them together. The second integrates the two problems into one focused, specific question. | |
The first question is too simple, asking for a straightforward fact that can be easily found online. The second is a more question that requires and detailed discussion to answer. | |
? dealt with the theme of racism through casting, staging, and allusion to contemporary events? | The first question is not — it would be very difficult to contribute anything new. The second question takes a specific angle to make an original argument, and has more relevance to current social concerns and debates. |
The first question asks for a ready-made solution, and is not . The second question is a clearer comparative question, but note that it may not be practically . For a smaller research project or thesis, it could be narrowed down further to focus on the effectiveness of drunk driving laws in just one or two countries. |
Note that the design of your research question can depend on what method you are pursuing. Here are a few options for qualitative, quantitative, and statistical research questions.
Type of research | Example question |
---|---|
Qualitative research question | |
Quantitative research question | |
Statistical research question |
If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.
Methodology
Statistics
Research bias
If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.
McCombes, S. (2023, October 19). 10 Research Question Examples to Guide your Research Project. Scribbr. Retrieved August 19, 2024, from https://www.scribbr.com/research-process/research-question-examples/
Other students also liked, writing strong research questions | criteria & examples, how to choose a dissertation topic | 8 steps to follow, evaluating sources | methods & examples, what is your plagiarism score.
Table of Contents
Before conducting a study, a research proposal should be created that outlines researchers’ plans and methodology and is submitted to the concerned evaluating organization or person. Creating a research proposal is an important step to ensure that researchers are on track and are moving forward as intended. A research proposal can be defined as a detailed plan or blueprint for the proposed research that you intend to undertake. It provides readers with a snapshot of your project by describing what you will investigate, why it is needed, and how you will conduct the research.
Your research proposal should aim to explain to the readers why your research is relevant and original, that you understand the context and current scenario in the field, have the appropriate resources to conduct the research, and that the research is feasible given the usual constraints.
This article will describe in detail the purpose and typical structure of a research proposal , along with examples and templates to help you ace this step in your research journey.
A research proposal¹ ,² can be defined as a formal report that describes your proposed research, its objectives, methodology, implications, and other important details. Research proposals are the framework of your research and are used to obtain approvals or grants to conduct the study from various committees or organizations. Consequently, research proposals should convince readers of your study’s credibility, accuracy, achievability, practicality, and reproducibility.
With research proposals , researchers usually aim to persuade the readers, funding agencies, educational institutions, and supervisors to approve the proposal. To achieve this, the report should be well structured with the objectives written in clear, understandable language devoid of jargon. A well-organized research proposal conveys to the readers or evaluators that the writer has thought out the research plan meticulously and has the resources to ensure timely completion.
A research proposal is a sales pitch and therefore should be detailed enough to convince your readers, who could be supervisors, ethics committees, universities, etc., that what you’re proposing has merit and is feasible . Research proposals can help students discuss their dissertation with their faculty or fulfill course requirements and also help researchers obtain funding. A well-structured proposal instills confidence among readers about your ability to conduct and complete the study as proposed.
Research proposals can be written for several reasons:³
Research proposals should aim to answer the three basic questions—what, why, and how.
The What question should be answered by describing the specific subject being researched. It should typically include the objectives, the cohort details, and the location or setting.
The Why question should be answered by describing the existing scenario of the subject, listing unanswered questions, identifying gaps in the existing research, and describing how your study can address these gaps, along with the implications and significance.
The How question should be answered by describing the proposed research methodology, data analysis tools expected to be used, and other details to describe your proposed methodology.
Here is a research proposal sample template (with examples) from the University of Rochester Medical Center. 4 The sections in all research proposals are essentially the same although different terminology and other specific sections may be used depending on the subject.
If you want to know how to make a research proposal impactful, include the following components:¹
1. Introduction
This section provides a background of the study, including the research topic, what is already known about it and the gaps, and the significance of the proposed research.
2. Literature review
This section contains descriptions of all the previous relevant studies pertaining to the research topic. Every study cited should be described in a few sentences, starting with the general studies to the more specific ones. This section builds on the understanding gained by readers in the Introduction section and supports it by citing relevant prior literature, indicating to readers that you have thoroughly researched your subject.
3. Objectives
Once the background and gaps in the research topic have been established, authors must now state the aims of the research clearly. Hypotheses should be mentioned here. This section further helps readers understand what your study’s specific goals are.
4. Research design and methodology
Here, authors should clearly describe the methods they intend to use to achieve their proposed objectives. Important components of this section include the population and sample size, data collection and analysis methods and duration, statistical analysis software, measures to avoid bias (randomization, blinding), etc.
5. Ethical considerations
This refers to the protection of participants’ rights, such as the right to privacy, right to confidentiality, etc. Researchers need to obtain informed consent and institutional review approval by the required authorities and mention this clearly for transparency.
6. Budget/funding
Researchers should prepare their budget and include all expected expenditures. An additional allowance for contingencies such as delays should also be factored in.
7. Appendices
This section typically includes information that supports the research proposal and may include informed consent forms, questionnaires, participant information, measurement tools, etc.
8. Citations
Writing a research proposal begins much before the actual task of writing. Planning the research proposal structure and content is an important stage, which if done efficiently, can help you seamlessly transition into the writing stage. 3,5
Key Takeaways
Here’s a summary of the main points about research proposals discussed in the previous sections:
Q1. How is a research proposal evaluated?
A1. In general, most evaluators, including universities, broadly use the following criteria to evaluate research proposals . 6
Q2. What is the difference between the Introduction and Literature Review sections in a research proposal ?
A2. The Introduction or Background section in a research proposal sets the context of the study by describing the current scenario of the subject and identifying the gaps and need for the research. A Literature Review, on the other hand, provides references to all prior relevant literature to help corroborate the gaps identified and the research need.
Q3. How long should a research proposal be?
A3. Research proposal lengths vary with the evaluating authority like universities or committees and also the subject. Here’s a table that lists the typical research proposal lengths for a few universities.
Arts programs | 1,000-1,500 | |
University of Birmingham | Law School programs | 2,500 |
PhD | 2,500 | |
2,000 | ||
Research degrees | 2,000-3,500 |
Q4. What are the common mistakes to avoid in a research proposal ?
A4. Here are a few common mistakes that you must avoid while writing a research proposal . 7
Thus, a research proposal is an essential document that can help you promote your research and secure funds and grants for conducting your research. Consequently, it should be well written in clear language and include all essential details to convince the evaluators of your ability to conduct the research as proposed.
This article has described all the important components of a research proposal and has also provided tips to improve your writing style. We hope all these tips will help you write a well-structured research proposal to ensure receipt of grants or any other purpose.
References
Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.
Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.
Experience the future of academic writing – Sign up to Paperpal and start writing for free!
How to write a phd research proposal.
The future of academia: how ai tools are changing the way we do research, you may also like, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , how to write your research paper in apa..., how to choose a dissertation topic, how to write an academic paragraph (step-by-step guide), maintaining academic integrity with paperpal’s generative ai writing..., research funding basics: what should a grant proposal..., how to write an abstract in research papers..., how to write dissertation acknowledgements.
When considering internal data or the results of a study, often business leaders either take the evidence presented as gospel or dismiss it altogether. Both approaches are misguided. What leaders need to do instead is conduct rigorous discussions that assess any findings and whether they apply to the situation in question.
Such conversations should explore the internal validity of any analysis (whether it accurately answers the question) as well as its external validity (the extent to which results can be generalized from one context to another). To avoid missteps, you need to separate causation from correlation and control for confounding factors. You should examine the sample size and setting of the research and the period over which it was conducted. You must ensure that you’re measuring an outcome that really matters instead of one that is simply easy to measure. And you need to look for—or undertake—other research that might confirm or contradict the evidence.
By employing a systematic approach to the collection and interpretation of information, you can more effectively reap the benefits of the ever-increasing mountain of external and internal data and make better decisions.
Five pitfalls to avoid
The problem.
When managers are presented with internal data or an external study, all too often they either automatically accept its accuracy and relevance to their business or dismiss it out of hand.
Leaders mistakenly conflate causation with correlation, underestimate the importance of sample size, focus on the wrong outcomes, misjudge generalizability, or overweight a specific result.
Leaders should ask probing questions about the evidence in a rigorous discussion about its usefulness. They should create a psychologically safe environment so that participants will feel comfortable offering diverse points of view.
Let’s say you’re leading a meeting about the hourly pay of your company’s warehouse employees. For several years it has automatically been increased by small amounts to keep up with inflation. Citing a study of a large company that found that higher pay improved productivity so much that it boosted profits, someone on your team advocates for a different approach: a substantial raise of $2 an hour for all workers in the warehouse. What would you do?
BMC Health Services Research volume 24 , Article number: 904 ( 2024 ) Cite this article
163 Accesses
Metrics details
Different professionals working in healthcare organizations (e.g., physicians, veterinarians, pharmacists, biologists, engineers, etc.) must be able to properly manage scarce resources to meet increasingly complex needs and demands. Due to the lack of specific courses in curricular university education, particularly in the field of medicine, management training programs have become an essential element in preparing health professionals to cope with global challenges. This study aims to examine factors influencing the effectiveness of management training programs and their outcomes in healthcare settings, at middle-management level, in general and by different groups of participants: physicians and non-physicians, participants with or without management positions.
A survey was used for gathering information from a purposive sample of professionals in the healthcare field attending management training programs in Italy. Factor analysis, a set of ordinal logistic regressions and an unpaired two-sample t-test were used for data elaboration.
The findings show the importance of diversity of pedagogical approaches and tools and debate, and class homogeneity, as effectiveness factors. Lower competencies held before the training programs and problems of dialogue and discussion during the course are conducive to innovative practice introduction. Interpersonal and career outcomes are greater for those holding management positions.
The study reveals four profiles of participants with different gaps and needs. Training programs should be tailored based on participants’ profiles, in terms of pedagogical approaches and tools, and preserve class homogeneity in terms of professional backgrounds and management levels to facilitate constructive dialogue and solution finding approach.
Peer Review reports
Several healthcare systems worldwide have identified management training as a precondition for developing appropriate strategies to address global challenges such as, on one hand, poor health service outcomes in front of increased health expenditure, particularly for pharmaceuticals, personnel shortages and low productivity, and on the other hand in terms of unbalanced quality and equal access to healthcare across the population [ 1 ]. The sustainability of health systems itself seems to be associated with the presence of leaders, at all levels of health organizations, who are able to correctly manage scarce resources to meet increasingly complex health needs and demands, at the same time motivating health personnel under an increasing amount of stress and steering their behaviors towards the system’s goals, in order to drive the transition towards more decentralized, interorganizational and patient-centered care models [ 2 ].
Recently, professional training as an activity aimed at increasing learning of new capabilities (reskilling) and improving existing ones (upskilling) during the lifetime of individuals (lifelong learning) has been identified by the European Commission as one of the seven flagship programs to be developed in the National Recovery and Resilience Plans (NRRP) to support the achievement of European Union’s goals, such as green and digital transitions, innovation, economic and social inclusion and occupation [ 3 ]. As a consequence, many member states have implemented training programs to face current and future challenges in health, which often represents a core mission in their NRRPs.
The increased importance of developing management training programs is also related to the rigidity and focalization of university degree courses in medicine, which do not provide physicians with the basic tools for fulfilling managerial roles [ 4 ]. Furthermore, taking on these roles does not automatically mean filling existing gaps in management capabilities and skills [ 5 ]. Several studies have demonstrated that, in the health setting, management competencies are influenced by positions and management levels as well as by organization and system’s features [ 6 , 7 ]. Hence, training programs aimed at increasing management competencies cannot be developed without considering these differences.
To date, few studies have focused on investigating management training programs in healthcare [ 8 ]. In particular, much more investigation is required on methods, contents, processes and challenges determining the effectiveness of training programs addressed to health managers by taking into account different environments, positions and management levels [ 1 ]. A gap also exists in the assessment of management training programs’ outcomes [ 9 ]. This study aims to examine factors influencing the effectiveness and outcomes of management training, at the middle-management level, in healthcare. It intends to answer the following research questions: which factors influence the management training process? Which relationships exist between management competencies held before the program, factors of effectiveness, critical issues encountered, and results achieved or prefigured at the end of the program? Are there differences, in terms of factors of effectiveness, challenges and outcomes, between the following groups of management training programs’ participants: physicians and non-physicians, participants with or without management positions?
Currently, there is a wide debate about the added value of management to health organizations [ 10 ] and thus about the importance of spreading management competencies within health organizations to improve their performance. Through a systematic review, Lega et al. [ 11 ] highlighted four approaches to examine the impact of management on healthcare performance, focusing on management practices, managers’ characteristics, engagement of professionals in performance management and organizational features and management styles.
Although findings have not always been univocal, several studies suggest a positive relationship between management competencies and practices and outcomes in healthcare organizations, both from a clinical and financial point of view [ 12 ]. Among others, Vainieri et al. [ 13 ] found, in the Italian setting, a positive association between top management’s competencies and organizational performance, assessed through a multidimensional perspective. This study also reveals the mediating effect of information sharing, in terms of strategy, results and organization structure, in the relationship between managerial competencies and performance.
The key role of management competencies clearly emerges for health executives, who have to turn system policies into a vision, and then articulate it into effective strategies and actions within their organizations to steer and engage professionals [ 14 , 15 , 16 , 17 , 18 , 19 ]. However, health systems are increasingly complex and continually changing across contexts and health service levels. This means the role of health executives is evolving as well and identifying the capacities they need to address current and emerging issues becomes more difficult. For instance, a literature review conducted by Figueroa et al. [ 20 ] sheds light on priorities and challenges for health leadership at three structural levels: macro context (international and national), meso context (organizations) and micro context (individual healthcare managers).
Doctor-managers are requested to carry both clinical tasks and tasks related to budgeting, goal setting and performance evaluation. As a consequence, a growing stream of research has speculated whether managers with a clinical background actually affect healthcare performance outcomes, but studies have produced inconclusive findings. In relation to this topic, Sarto and Veronesi [ 21 ] carried out a literature review showing a generally positive impact of clinical leadership on different types of outcome measures, with only a few studies reporting negative impacts on financial and social performance. Morandi et al. [ 22 ] focused on doctor-managers who have become middle managers and investigated the potential bias in performance appraisal due to the mismatch between self-reported and official performance data. At the individual level, the role played by managerial behavior, training, engagement, and perceived organizational support was analyzed. Among others indications they suggested that training programs should be revised to reduce bias in performance appraisal. Tasi et al. [ 23 ] conducted a cross-sectional analysis of the 115 largest U.S. hospitals, divided into physician-led and non-physician-led, which revealed that physician-led hospital systems have higher quality ratings across all specialities and more inpatient days per hospital bed than non-physician-led hospitals. No differences between the groups were found in total revenue and profit margins. The main implication of their study is that hospital systems may benefit from the presence of physician leadership to improve the quality and efficiency of care delivered to patients as long as education and training are able to adequately prepare them. The main issue, as also observed by others [ 4 , 24 ], is that university education in medicine still includes little focus on aspects such as collaborative management, communication and coordination, and leadership skills. Such a circumstance motivates the call for further training. Regarding the implementation of training programs, Liang et al. [ 1 ] have recently shown how it is hindered, among others, by a lack of sufficient knowledge about needed competencies and existing gaps. Their analysis, which focuses on senior managers from three categories in Chinese hospitals, shows that before commencing the programs senior managers had not acquired adequate management competencies either through formal or informal training. It is worth noticing that significant differences exist between hospital categories and management levels. For this reason, they recommend using a systemic approach to design training programs, which considers different hospital types, management levels and positions. Yarbrough et al. [ 6 ] examined how competence training worked in healthcare organizations and the competencies needed for leaders at different points of their careers at various organizational levels. They carried out a cross-sectional survey of 492 US hospital executives, whose most significant result was that competence training is effective in healthcare organizations.
Walston and Khaliq [ 25 ], from a survey of 2,001 hospital CEOs across the US concluded that the greatest contribution of continuing education is to keep CEOs updated on technological and market changes that impact their current job responsibilities. Conversely, it does not seem to be valued for career or succession planning. About the methods of continuing education, an increasing use of some internet-based tools was found. Walston et al. [ 26 ] identified the factors affecting continuing education, finding, among others, that CEOs from for-profit and larger hospitals tend to take less continuing education, whereas senior managers' commitment to continuing education is influenced by region, gender, the CEO's personal continuing education hours and the focus on change.
Furthermore, the principles that inspire modern healthcare models, such as dehospitalization, horizontal coordination and patient-centeredness, imply the increased importance of middle managers, within single structures but also along clinical pathways and projects, to create and sustain high performances [ 27 , 28 , 29 ].
Whaley and Gillis [ 8 ] investigated the development of training programs aimed at increasing managerial competencies and leadership of middle managers, both from clinical and nonclinical backgrounds, in the US context. By adopting the top managers’ perspective, they found a widespread difficulty in aligning training needs and program contents. A 360° assessment of the competencies of Australian middle-level health service managers from two public hospitals was then conducted by Liang et al. [ 7 ] to identify managerial competence levels and training and development needs. The assessment found competence gaps and confirmed that managerial strengths and weaknesses varied across management groups from different organizations. In general, several studies have shown that leading at various organizational levels, in healthcare, does not necessarily require the same levels and types of competencies.
Liang et al. [ 30 ] explored the core competencies required for middle to senior-level managers in Victorian public hospitals. By adopting mixed methods, they confirmed six core competencies and provided guidance to the development of the competence-based educational approach for training the current and future management workforce. Liang et al. [ 31 ] then focused on the poorly investigated area of community health services, which are one of the main solutions to reducing the increasing demand for hospital care in general, and, in particular, in the reforms of the Australian health system. Their study advanced the understanding of the key competencies required by senior and mid-level managers for effective and efficient community health service delivery. A following cross-sectional study by AbuDagga et al. [ 32 ] highlighted that some community health services, such as home healthcare and hospice agencies, also need specific cultural competence training to be effective, in terms of reducing health disparities.
Using both qualitative and quantitative methods, Liang et al. [ 33 ] developed a management competence framework. Such a framework was then validated on a sample of 117 senior and middle managers working in two public hospitals and five community services in Victoria, Australia [ 34 ]. Fanelli et al. [ 35 ] used mixed methods to identify the following specific managerial competencies, which healthcare professionals perceive as crucial to improve their performance: quality evaluation based on outcomes, enhancement of professional competencies, programming based on process management, project cost assessment, informal communication style and participatory leadership.
Loh [ 5 ], through a qualitative analysis conducted in Australian hospitals, examined the motivation behind the choice of medically trained managers to undertake postgraduate management training. Interesting results stemming from the analysis include the fact that doctors often move into management positions without first undertaking training, but also that clinical experience alone does not lead to required management competencies. It is also interesting to remark that effective postgraduate management training for doctors requires a combination of theory and practice, and that doctors choose to undertake training mostly to gain credibility.
Ravaghi et al. [ 36 ] conducted a literature review to assess the evidence on the effectiveness of different types of training and educational programs delivered to hospital managers. The analysis identifies a set of aspects that are impacted by training programs. Training programs focus on technical, interpersonal and conceptual skills, and positive effects are mainly reported for technical skills. Numerous challenges are involved in designing and delivering training programs, including lack of time, difficulty in employing competencies in the workplace, also due to position instability, continuous changes in the health system environment, and lack of support by policymakers. One of the more common flaws concerns the fact that managers are mainly trained as individuals, but they work in teams. The implications of the study are that increased investments and large-scale planning are required to develop the knowledge and competencies of hospital managers. Another shortage concerns the outcome measurement of training programs, which is a usually neglected issue in the literature [ 9 ]. It also emerges that the training programs performing best are specific, structured and comprehensive.
Kakemam and Liang [ 2 ] conducted a literature review to shed light on the methods used to assess management competencies, and, thus, professional development needs in healthcare. Their analysis confirms that most studies focus on middle and senior managers and demonstrate great variability in methods and processes of assessment. As a consequence, they elaborate a framework to guide the design and implementation of management competence studies in different contexts and countries.
In the end, the literature has long pointed out that developing and strengthening the competencies and skills of health managers represent a core goal for increasing the efficiency and effectiveness of health systems, and management training is crucial for achieving such a goal [ 37 ]. The reasons can be summarized as follows: university education has scarcely been able to provide physicians and, in general, health operators, with adequate, or at least basic, managerial competencies and skills; over time, professionals have been involved in increasingly complex and rapidly changing working environments, requiring increased management responsibilities as well as new competencies and skills; in many settings, for instance in Italy, delays in the enforcement of law requiring the attendance of specific management training courses to take up a leadership position, hindered the acquisition of new competencies and the improvement of existing ones by those already managing health organizations, structures and services.
For the purposes of this study, management competencies refer to the possession and ability to use skills and tools for service organization and service planning, control and evaluation, evidence-informed decision-making and human resource management in the healthcare field.
The reform of the Italian National Health System (INHS), implemented by Legislative Decree No. 502/1992 and inspired by neo-managerial theories, introduced the role of the general manager and assigned new responsibilities to managers.
However, the inadequate performance achieved in the first years of the application of the reform highlighted the cultural gap that made the normative adoption of managerial approach and tools unproductive on the operational level. Legislation evolved accordingly, and in order to hold management positions, management training became mandatory. Decree-Law No. 583/1996 (converted into Law No. 4/1997) provided that the requirements and criteria for access to the top management level were to be determined. Therefore, Presidential Decree No. 484/1997 determined these requirements and also the requirements and criteria to access the middle-management level of INHS’ healthcare authorities. This regulation also imposed the acquisition of a specific management training certificate, dictated rules concerning the duration, contents, and teaching methods of management training courses issuing this certificate, and indicated the requirements for attendance. Immediately afterwards, Legislative Decree No. 229/1999 amended the discipline of medical management and health professions and promoted continuous training in healthcare. It also regulated management training, which became an essential requirement for the appointments of health directors and directors of complex structures in the healthcare authorities, for the categories of physicians, dentists, veterinarians, pharmacists, biologists, chemists, physicists and psychologists.
The second pillar of the INHS reform was the regionalization of the INHS. Therefore, the Regions had to organize the courses to achieve management training certificates on the basis of specific agreements with the State, which regulated the contents, the methodology, the duration and the procedures for obtaining certification. The State-Regions Conference approved the first interregional agreement on management training in July 2003, whereas the State-Regions Agreement of 16 May 2019 regulated the training courses. The mandatory contents of the management training outlined the skills and behaviors expected from general managers and other top management key players (Health Director, Administrative Director and Social and Health Director), but also for all middle managers.
A survey was used to gather information from a purposive sample of professionals in the healthcare field taking part in management training programs. In particular, a structured questionnaire was submitted to 140 participants enrolled in two management programs organized by an Italian university: a second-level specializing master course and a training program carried out in collaboration with the Region. The programs awarded participants the title needed to be appointed as a director of a ward or administrative unit in a public healthcare organization, and share the same scientific committee, teaching staff, administrative staff and venue. The respondents’ profile is shown in Table 1 .
It is worth pointing out that the teaching staff is characterized by diversity: teachers have different educational and professional backgrounds, are practitioners or academics, and come from different Italian regions.
The questionnaire was submitted and completed in presence and online between November 2022 and February 2023. All participants decided to take part in the analysis spontaneously and gave their consent, being granted total anonymity.
The questionnaire, which was developed for this study and based on the literature, consisted of 64 questions shared in the following five sections: participant profile (10 items), management competencies held by participants before the training program (4 items), effectiveness factors of the training program (23 items), challenges to effectiveness (10 items), and outcomes of the training program (17 items) (an English language version of the questionnaire is attached to this paper as a supplementary file). In particular, the second section aimed to shed light on the participants’ situation regarding management competencies held before the start of the training program and how they were acquired; the third section aimed to collect participants’ opinions regarding how the program was conducted and the factors influencing its effectiveness; the fourth section aimed to collect participants’ opinions regarding the main obstacles encountered during the program; and the fifth section aimed to reveal the main outcomes of the program in terms of knowledge, skills, practices and career.
Except for those of the first section, which collected personal information, all the items of the next four categories – management competencies, effectiveness factors, challenges and outcome — were measured through a 5-point Likert scale. To ensure that the content of the questionnaire was appropriate, clear and relevant, a pre-testing was conducted in October 2022 by asking four academics and four practitioners, both physicians and not, with and without management positions, to fill it out. The aim was to understand whether the questionnaire really addressed the information needs behind the study and was easily and correctly understood by respondents. Therefore, the four individuals involved in the pre-testing were asked to fill it out simultaneously but independently, and at the end of the compilation, a focus group that included them and the three authors was used to collect their opinions and suggestions. After this phase, the following changes were made: in the ‘Participant profile’ section, ‘Veterinary medicine’ was added to the fields accounting for the ‘Educational background’ (item 3); in Sect. 2, it was decided to modify the explanation given to ‘basic management competencies’ and align it to what required by Presidential Decree No. 484/1997; in Sect. 3, item 25 was added to catch a missing aspect that respondents considered important, and brackets were added to the description of items 15, 16 and 29 to clarify the concepts of mixed and homogenous class and pedagogical approaches and tools; in Sect. 4, in the description of item 40, the words ‘find the energy required’ were added to avoid confusion with items 38 and 39, whereas brackets were added to items 41 and 45 to provide more explanation; in Sect. 5, brackets were added to the description of item 51 to increase clarity, and the last item was divided into two (now items 63 and 64) to distinguish the training program’s impact on career at different times.
With reference to the methods, first, a factor analysis based on the principal component method was conducted within each section of the questionnaire (except for the first again), in order to reduce the number of variables and shed light on the factors influencing the management training process. Bartlett's sphericity test and the Kaiser–Meyer–Olkin (KMO) value were performed to assess sampling adequacy, whereas factors were extracted following the Kaiser criterion, i.e., eigenvalues greater than unity, and total variance explained. The rotation method used was the Varimax method with Kaiser normalization, except for the second section (i.e., management competencies held by participants before the training program) that), which did not require rotation since a single factor emerged from the analysis. Bartlett's sphericity test was statistically significant ( p < 0.001) in all sections, KMO values were all greater than 0.65 (average value 0.765), and the total variances explained were all greater than 65% (average value of approximately 70.89%), which are acceptable values for such analysis.
Second, a set of ordinal logistic regressions were performed to assess the relationships existing between management competencies held before the start of the course, effectiveness factors, challenges, and outcomes of the training program.
The factors that emerged from the factor analysis were used as independent variables, whereas some significant outcome items accounting for different performance aspects were selected as dependent variables: improved management competencies, innovation practices, professional relationships, and career prospects. Ordered logit regressions were used because the dependent variables (outcomes) were measured on ordinal scales. Some control variables for the respondent profiles were included in the regression models: age, gender, educational background, management position, and working in the healthcare field.
With the aim of understanding which explanatory variables could exert an influence, a backward elimination method was used, adopting a threshold level of significance values below 0.20 ( p < 0.20). Table 4 shows the results of regressions with independent variables obtained following the criterion mentioned above. All four models respected the null hypothesis, which means that the proportional odds assumption behind the ordered logit regressions had not been rejected ( p > 0.05). Third and last, an unpaired two-sample t-test was used to examine the differences between groups of participants in the management training programs selected based on two criteria: physicians and non-physicians, and participants with or without management positions.
First, descriptive statistics is useful for understanding the aspects participants considered the most and least important by category. This can be done by focusing on the items of the four sections of the questionnaire (except for the first one depicting participant profiles) that were given the highest and lowest scores at the sample level and by different groups of participants (physicians and non-physicians, participants with or without management positions). Table 2 summarizes the mean values and standard deviations by group of these higher and lower scores. Focusing on management competencies, all groups reported having mainly acquired them through professional experience, except for non-physicians who attributed major significance to postgraduate training programs, with a mean value of 3.05 out of 5. All groups agreed on the poor role of university education in providing management competencies, with mean values for the sample and all four groups below 2.5. It is worth noting that this item exhibits the lowest value for physicians (1.67) and the highest for non-physicians (2.37). In addition, physicians are the group attributing the lowest values to postgraduate education and professional experience for acquiring management competencies. In reference to factors of effectiveness, all groups also agree on the necessity of mixing theoretical and practical lessons during the training program with mean values of well above 4.5, whereas exclusive use of self-assessment is generally viewed as the most ineffective practice, except for non-physician, who attribute the lowest value to remote lessons (mean 1.82). Among the challenges, the whole sample and physicians and participants without management positions see the lack of financial support from their organization as the main problem (mean 4.10), while non-physicians and participants with management positions believe this is represented by a lack of time, with mean values, respectively, of 3.75 and 4. All agree that dialogue and discussion during the course have been the least relevant of the problems, with mean values below 1.5. Outcomes show generally high values, as revealed by the fact that the lowest values exhibit mean values around 3.5. It is worth noting that an increased understanding of the healthcare systems has been the main benefit gained from the program, with mean values equal to or higher than 4.50. The lowest positive impact is attributed by all attendees to improved relationships with superiors and top management, with mean values between 3.44 and 3.74, with the exception of participants without management positions who mention improved career prospects.
To shed light on the factors influencing the management training process, the findings of the factor analyses conducted by category are reported. Starting from the management competencies held before the training program, the following single factor was extracted from the four items, named and interpreted as follows:
Basic management competencies, which measures the level of management competencies acquired by participants through higher education, post-graduate training and professional experience.
The effectiveness factors are then grouped into six factors, named and explained as follows:
Diversity and debate, which aggregates five items assessing the importance of diversity in participants’ and teachers’ educational and professional backgrounds and pedagogical approaches and tools, as well as level of participant engagement and discussion during lessons and in carrying out the project work required to complete the program.
Specialization, which includes three items accounting for a robust knowledge of healthcare systems by focusing on teachers’ profiles and lessons’ theoretical approaches.
Lessons in presence, which groups three items explaining that in-presence lessons increase learning outcomes and discussion among participants.
Final self-assessment, made up of three items asserting that learning outcomes should be assessed by participants themselves at the end of the course.
Written intermediate assessment, composed of two items explaining that mid-terms assessment should only be written.
Homogeneous class, which is made up of a single component accounting for participants’ similarity in terms of professional backgrounds and management levels, tasks and responsibilities.
The challenges are aggregated into the following four factors:
Lack of time, which includes three items reporting scarce time and energy for lessons and study.
Problems of dialogue and discussion, which groups three items focusing on difficulties in relating to and debating with other participants and teachers.
Low support from organization, which is made up of two items reporting poor financial support and low value given to the initiative from participants’ own organizations.
Organizational issues, which aggregates two items demonstrating scarce flexibility and collaboration by superiors and colleagues of participants’ own organizations and unfamiliarity to study.
Table 3 shows the component matrix with saturation coefficients and factors obtained for the management competencies held before the training program (unrotated), effectiveness factors (rotated), and challenges (rotated).
A set of ordinal logistic regressions was performed to examine the relationships between management competencies held before the start of the course, effectiveness factors, challenges and outcomes of the training program. The results, shown in Table 4 , are articulated into four models, one for each selected outcome. In relation to model 1, the factors ‘diversity and debate’ ( p < 0.001), ‘written intermediate assessment’ ( p < 0.05) and ‘homogeneous class’ ( p < 0.001) have a significant positive impact on the improvement of management competencies, which is also increased by low values attributed to ‘problems of dialogue and discussion’ ( p < 0.01). In model 2, the change of professional practices in light of lessons learned during the program, selected as an innovation outcome, is then positively affected by ‘diversity and debate’ ( p < 0.001), ‘homogeneous class’ ( p < 0.05) and ‘organizational issues’ ( p < 0.01), while it was negatively influenced by a high value of ‘basic management competencies’ held before the course ( p < 0.05). Regarding model 3, ‘Diversity and debate’ ( p < 0.001) and ‘homogeneous class’ ( p < 0.01) have a significant positive effect on the improvement of professional relationships as well, whereas the same is negatively affected by ‘lessons in presence’ ( p < 0.05). Finally, concerning model 4, the outcome career prospects benefit from ‘diversity and debate’ ( p < 0.05) and ‘homogeneous class’ ( p < 0.01), since both factors exert a positive effect. ‘Low support from organization’ negatively influences career prospects ( p < 0.001). Table 4 also shows that the LR test of proportionality of odds across the response categories cannot be rejected (all four p > 0.05).
Finally, it is worth noting that none of the control variables reflecting the respondent profiles (age, gender, management position, working in the healthcare field, and educational background) was found to be statistically significant. These variables are not reported in Table 4 because regression models were obtained following a backward elimination method, as explained in the method section.
In the end, the t-test reveals significant differences between physicians and non-physicians, as well as between participants with or without management positions. Table 5 shows only figures of t-test statistically significant with regards to competencies held before the attendance of the course, the factors of effectiveness, challenges of the training program, and outcomes achieved. In the first comparison, non-physicians show higher management competencies at the start of the program, with a mean value of 0.31, while physicians suffer from less support from their own organization with a mean value of 0.13 compared to -0.18, the mean value of the non-physicians. Concerning the second comparison, participants with management positions have higher management competencies at the start of the program (0.19 versus -0.13) and suffer more from lack of time, with higher mean values compared to participants without managerial positions, respectively 0.23 and -0.16. For what concerns the factors related to the effectiveness of the training program, participants with management positions exhibit a lower mean value in relation to written mid-term assessments, -0.24 versus 0.17, reported by participants with management positions. Differently, the final self-assessment at the end of the program is higher for participants with management positions, 0.24 compared to -0.17, the mean value of the participants without management positions. This latter category feels more the problem of low support from their organizations, with a mean value of 0.16 compared to -0.23, and is slightly less motivated by possible career improvement, with a mean value of 3.31 compared to 3.73 reported by participants with management positions.
The results stemming from the different analyses are now considered and interpreted in the light of the extant literature. Personal characteristics such as gender and age, differently from what was found by Walston et al. [ 26 ] for executives’ continuing education, and professional characteristics such as seniority and working in public or private sectors, do not seem to affect participation in management training programs.
The findings clearly show the outstanding importance of ‘diversity and debate’ and ‘class homogeneity’ as factors of effectiveness, since they positively impact all outcomes: competencies, innovation, professional relationships and career. These factors capture two key aspects complementing each other: on the one hand, participants and teachers’ different backgrounds provide the class with a wider pool of resources and expertise, whereas the use of pedagogical tools fostering discussion enriches the educational experience and stimulates creativity. On the other hand, due to the high level of professionalism in the setting, sharing common management levels means similar tasks and responsibilities, as well as facing similar problems. Consequently, speaking the same language leads to deeper knowledge and effective technical solutions.
In relation to the improvement of management competencies, it also emerges the critical role of a good class atmosphere, that is, the absence of problems of dialogue and discussion. ‘Diversity and debate’ and ‘class homogeneity’, as explained before, seem to contribute to this, since they enhance freedom of expression and fair confrontation, leading to improved learning outcomes. It is interesting to notice that the problems of dialogue and discussion turned out to be the least relevant challenge across the sample.
Two interesting points come from the factors affecting innovation. First, it seems that lower competencies before the training programs lead to the development of more innovative practices. The reason is that holding fewer basic competencies means a greater scope for action once new capabilities are learned: the spirit of openness is conducive to breaking down routines, and innovative practices hindered by a lack of knowledge and tools can thus be introduced. The reason is that holding fewer basic competencies means greater scope for action once new capabilities are learned: the spirit of openness is conducive to breaking down routines, and innovative practices hindered by a lack of knowledge and tools can thus be introduced. This extends the findings of previous studies since the employment of competencies in the workplace is influenced by the starting competence equipment of professionals [ 36 ], and those showing gaps have more room to recover, also in terms of motivation to change, that is, understanding the importance of meeting current and future challenges [ 26 ]. Second, more innovative practices are introduced by participants perceiving more organizational issues. This may reveal, on the one side, a stronger individual motivation towards professional growth of participants who suffer from lack of flexibility and collaboration from their own superiors and colleagues. In this regard, poor tolerance, flexibility and permissions in their workplace act as a stimulus to innovation, which can be viewed as a way of challenging the status quo. On the other side, in line with the above-mentioned concept, this confirms that unfamiliarity with the study increases the innovative potential of participants. Since this study reveals that physicians are neither adequately educated from a management point of view nor incentivized to attend post-graduation training programs, it points out how important is extending continuing education to all health professional categories [ 25 , 26 ].
The topic of competencies held by different categories needs more attention. The study reveals that physicians and participants without management positions start the program with less basic competencies. At the sample level, higher education is viewed as the most ineffective tool to provide such competencies, whereas professional experience is seen as the best way to gather them. Actually, non-physicians give the highest value to postgraduate education, which suggests they are those more interested or incentivized to take part in continuing education. Although holding managerial positions does not automatically mean having higher competencies [ 5 ], it is evident that such a professional experience contributes to filling existing gaps. Physicians stand out as the category for which university education, postgraduate education and professional experience exert the lowest impact on management competence improvement. Considering the relationship between competence held before the course and innovation, as described above, engaging physicians in training programs, even more if they do not have management responsibilities, has a major impact on health organizations’ development prospects. The findings also point out that effective management training requires a combination of theory and practice for all categories of professionals, not just for physicians, as observed by Loh [ 5 ].
The main outcome, in general and for all participant categories, is an increased understanding of how healthcare systems work, which anticipates increased competencies. This confirms the importance of knowledge on the healthcare environment [ 31 ], and clarifies the order of aspects impacted by training programs as reported by Ravaghi et al. [ 36 ]: first conceptual, then technical, and finally interpersonal. However, interpersonal outcomes are by far greater for those holding management positions, which extends the findings by Liang et al. [ 31 ]. In particular, participants already managing units report the greatest impacts in terms of ability to understand colleagues’ problems, improvement of professional relationships and collaboration with colleagues from other units. Obviously, participants with management positions, more than others, feel the lack of collaborative and communication skills, which represents one of the main flaws of university education in the field of medicine [ 4 ] and is also often neglected in management training [ 36 ]. This also confirms that different management levels show specific competence requirements and education needs [ 6 , 7 ].
It is then important to discuss the negative effect of lessons in presence on the improvement of professional relationships. At first glance, it may sound strange, but its real meaning emerges from a comprehensive interpretation of all the findings. First, it does not mean that remote lessons are more effective, as revealed by the fact that they, as a factor of effectiveness, are attributed very low values and, for all categories of participants, lower values than those attributed to lessons in presence and hybrid lessons. Non-physicians, in particular, attribute them the lowest value at all. At most, remote lessons are viewed as convenient rather than effective. The negative influence of lessons in presence can be explained by the fact that a specific category, i.e., those with management positions, rate this aspect much more important than other participants and, as reported above, find much more benefits in terms of improved relationships from management training. Participants with management positions, due to their tasks and responsibilities, suffer more than others from lack of time to be devoted to course participation. For them, as for the category of non-physicians, lack of time represents the main challenge to effectively attending the course. In the literature, such a problem is well considered, and lack of time is also viewed as a challenge to apply the skills learned during the course [ 36 ]. Considering that class discussion and homogeneity contribute to fostering relationships, a comprehensive reading of the findings reveals that due to workload, participants with management positions see particularly convenient and still effective remote lessons. Furthermore, if the class is formed by participants sharing similar professional backgrounds and management levels, debate is not precluded and interpersonal relationships improved as a consequence. From the observation of single items, it can be concluded that participants with management positions and in general those with higher basic management competencies at the start of the program, prefer more flexible and leaner training programs: intermediate assessment through conversation, self-assessment at the end of the course, more concentrated scheduled lessons and greater use of remote lessons.
Differently from what was found by Walston and Khaliq [ 25 ], the findings highlight that participants with management positions value the impact of management training on career prospects positively. These participants are also those more supported by their own organizations. Conversely, the lack of support, especially in terms of inadequate funds devoted to these initiatives, strongly affects physicians and participants without management positions, which clarifies what this challenge is about and who is mainly affected by it [ 36 ]. Low incentives mean having attended fewer training programs in the past, which, together with less management experience, explains why they have developed less competencies. Among the outcomes of the training program, the little attention paid by organizations is also testified by the lowest values attributed by all categories, except for participants without management positions, to the improvement of relationships with superiors and top management.
In general, the study contributes to a better understanding of the outcomes of management training programs in healthcare and their determinants [ 9 ]. In particular, it sheds light on gaps and education needs [ 1 ] by category of health professionals [ 2 ]. The research findings have major implications for practice, which can be drawn after identifying the four profiles of participants revealed by the study. All profiles share common characteristics, such as value given to debate, diversity of pedagogical approaches and tools and class homogeneity, rather than the need for a deeper comprehension of healthcare systems. However, they present characteristics that determine specific issues and education gaps, which are summarized as follows:
Physicians without management positions: low competencies at the start of the program and scarce incentives for attending the course from their own organization;
Physicians with management positions: they partially compensate for competence gaps through professional experience, suffer from lack of time, and are motivated by the chance to improve their career prospects;
Non-physicians without management positions: they partially fill competence gaps through postgraduate education, suffer from lack of time, and have scarce incentives for attending the course from their own organization;
Non-physicians with management positions: they partially bridge competence gaps through postgraduate education and professional experience, are the most affected by a lack of time, and are motivated by the chance to improve their career prospects.
Recommendations are outlined for different levels of action:
For policymakers, it is suggested to strengthen the ability of higher education courses in medicine and related fields to advance the understanding of healthcare systems’ structure and operation, as well as their current and future challenges. Such a new approach in the design curricula should then have as a main goal the provision of adequate management competencies.
For healthcare organizations, it is suggested to incentivize the acquisition of management competencies by all categories of professionals through postgraduate education and training programs. This means supporting them from both financial and organizational point of view, for instance, in terms of more flexible working conditions. Special attention should be paid to physicians who, even without executive roles, manage resources and directly impact the organization's effectiveness and efficiency levels through their day-by-day activity, and are the players holding the greatest innovative potential within the organization. Concerning the executives, especially in the current changing context of healthcare systems, much higher attention should be paid to fostering interpersonal skills, in terms of communication and cooperation.
For those designing training programs, it is suggested to tailor courses on the basis of participants’ profiles, using different pedagogical approaches and tools, for instance, in terms of teacher composition, lesson delivery methods and learning assessment methods, while preserving class homogeneity in terms of professional backgrounds and management levels to facilitate constructive dialogue and solution finding approaches. Designing ad hoc training programs would give the possibility to meet the needs of participants from an organizational point of view as well as, for instance, in terms of program length and lesson concentration.
This study has some limitations, which pave the way for future research. First, it is context-specific by country, since it is carried out within the INHS, which mandatorily requires health professionals to attend management training programs to hold certain positions. It is then context-specific by training program, since it focuses on management training programs providing participants with the title to be appointed as a director of a ward or administrative unit in a public healthcare organization. This determines the kind of management competencies included in the study, which are those mandatorily required for such a middle-management category. Therefore, there is a need to extend research and test these findings on different types of management training programs, participants and countries. Second, this study is based on a survey of participants’ perceptions, which causes two kinds of unavoidable issues: although based on the literature and pre-tested, the questionnaire could not be able to measure what it intends to or capture detailed and nuanced insights from respondents, and responses may be affected by biases due to reactive effects. Third, a backward elimination method was adopted to select variables in model building. Providing a balance between simplicity and fit of models, this variable selection technique is not consequences-free. Despite advantages such as starting the process with all variables included, removing the least important early, and leaving the most important in, it also has some disadvantages. The major is that once a variable is deleted from the model, it is not included anymore, although it may become significant later [ 38 ]. For these reasons, it is intended to reinforce research with new data sources, such as teachers’ perspectives and official assessments, and different variable selection strategies. A combination of qualitative and quantitative methods for data elaboration could then be used to deepen the analysis of the relationships between motivations, effectiveness factors and outcomes. Furthermore, since the investigation of competence development, acquisition of new competencies and the transfer of acquired competencies was beyond the purpose of this study, a longitudinal approach will be used to collect data from participants attending future training programs to track changes and identify patterns.
An English-language version of the questionnaire used in this study is attached to this paper as a supplementary file. The raw data collected via the questionnaire are not publicly available due to privacy and other restrictions. However, datasets generated and analyzed during the current study may be available from the corresponding author upon reasonable request.
Italian National Health System
Kaiser–Meyer–Olkin
National Recovery and Resilience Plan
Liang Z, Howard PF, Wang J, Xu M, Zhao M. Developing senior hospital managers: does ‘one size fit all’? – evidence from the evolving Chinese health system. BMC Health Serv Res. 2020;20(281):1–14. https://doi.org/10.1186/s12913-020-05116-6 .
Article Google Scholar
Kakemam E, Liang Z. Guidance for management competency identification and development in the health context: a systematic scoping review. BMC Health Serv Res. 2023;23(421):1–13. https://doi.org/10.1186/s12913-023-09404-9 .
European Commission. Annual Sustainable Growth Strategy. 2020. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52020DC0575&from=en
Blakeney EAR, Ali HN, Summerside N. Sustaining improvements in relational coordination following team training and practice change: a longitudinal analysis. Health Care Manag Rev. 2021;46(4):349–57. https://doi.org/10.1097/HMR.0000000000000288 .
Loh E. How and why medically-trained managers undertake postgraduate management training - a qualitative study from Victoria. J Health Organ Manag. 2015;29(4):438–54. https://doi.org/10.1108/jhom-10-2013-0233 .
Article PubMed Google Scholar
Yarbrough LA, Stowe M, Haefner J. Competency assessment and development among health-care leaders: results of a cross-sectional survey. Health Serv Manag Res. 2012;25(2):78–86. https://doi.org/10.1258/hsmr.2012.012012 .
Liang Z, Blackstock FC, Howard PF, Briggs DS, Leggat SG, Wollersheim D, Edvardsson D, Rahman A. An evidence-based approach to understanding the competency development needs of the health service management workforce in Australia. BMC Health Serv Res. 2018;18(976):1–12. https://doi.org/10.1186/s12913-018-3760-z .
Whaley A, Gillis WE. Leadership development programs for health care middle managers: an exploration of the top management team member perspective. Health Care Manag Rev. 2018;43(1):79–89. https://doi.org/10.1097/HMR.0000000000000131 .
Campbell C, Lomperis A, Gillespie K, Arrington B. Competency-based healthcare management education: the Saint Louise University experience. J Health Adm Educ. 2006;23:135–68.
PubMed Google Scholar
Issel ML. Value Added of Management to Health Care Organizations. Health Care Manag Rev. 2020;45(2):95. https://doi.org/10.1097/HMR.0000000000000280 .
Lega F, Prenestini A, Spurgeon P. Is management essential to improving the performance and sustainability of health care systems and organizations? a systematic review and a roadmap for future studies. Value Health. 2013;16(1 Suppl.):S46–51. https://doi.org/10.1016/j.jval.2012.10.004 .
Bloom N, Propper C, Seiler S, Van Reenen J. Management practices in hospitals. Health, Econometrics and Data Group (HEDG) working papers 09/23, HEDG, c/o department of economics, University of York. 2009.
Vainieri M, Ferrè F, Giacomelli G, Nuti S. Explaining performance in health care: how and when top management competencies make the difference. Health Care Manag Rev. 2019;44(4):306–17. https://doi.org/10.1097/HMR.0000000000000164 .
Del Vecchio M, Carbone C. Stabilità dei Direttori Generali nelle aziende sanitarie. In: Anessi Pessina E, Cantù E, editors. Rapporto OASI 2002 L’aziendalizzazione della sanità in Italia. Milano, Italy: Egea; 2002. p. 268–301.
Google Scholar
McAlearney AS. Leadership development in healthcare: a qualitative study. J Organ Behav. 2006;27:967–82.
McAlearney AS. Using leadership development pro- grams to improve quality and efficiency in healthcare. J Healthcare Manag. 2008;53:319–31.
McAlearney AS. Executive leadership development in U.S. health systems. J Healthcare Manag. 2010;55:207–22.
McAlearney AS, Fisher D, Heiser K, Robbins D, Kelleher K. Developing effective physician leaders: changing cultures and transforming organizations. Hosp Top. 2005;83(2):11–8.
Thompson JM, Kim TH. A profile of hospitals with leadership development programs. Health Care Manag. 2013;32(2):179–88. https://doi.org/10.1097/HCM.0b013e31828ef677 .
Figueroa C, Harrison R, Chauhan A, Meyer L. Priorities and challenges for health leadership and workforce management globally: a rapid review. BMC Health Serv Res. 2019;19(239):1–11. https://doi.org/10.1186/s12913-019-4080-7 .
Sarto F, Veronesi G. Clinical leadership and hospital performance: assessing the evidence base. BMC Health Serv Res. 2016;16(169):85–109. https://doi.org/10.1186/s12913-016-1395-5 .
Morandi F, Angelozzi D, Di Vincenzo F. Individual and job-related determinants of bias in performance appraisal: the case of middle management in health care organizations. Health Care Manag Rev. 2021;46(4):299–307. https://doi.org/10.1097/HMR.0000000000000268 .
Tasi MC, Keswani A, Bozic KJ. Does physician leadership affect hospital quality, operational efficiency, and financial performance? Health Care Manag Rev. 2019;44(3):256–62. https://doi.org/10.1097/hmr.0000000000000173 .
Hopkins J, Fassiotto M, Ku MC. Designing a physician leadership development program based on effective models of physician education. Health Care Manag Rev. 2018;43(4):293–302. https://doi.org/10.1097/HMR.0000000000000146 .
Walston SL, Khaliq AA. The importance and use of continuing education: findings of a national survey of hospital executives. J Health Admin Educ. 2010;27(2):113–25.
Walston SL, Chou AF, Khaliq AA. Factors affecting the continuing education of hospital CEOs and their senior managers. J Healthcare Manag. 2010;55(6):413–27. https://doi.org/10.1097/00115514-201011000-00008 .
Garman AN, McAlearney AS, Harrison MI, Song PH, McHugh M. High-performance work systems in health- care management, part 1: development of an evidence-informed model. Health Care Manag Rev. 2011;36(3):201–13. https://doi.org/10.1097/HMR.0b013e318201d1bf .
MacDavitt K, Chou S, Stone P. Organizational climate and healthcare outcomes. Joint Comm J Qual Patient Saf. 2007;33(S11):45–56. https://doi.org/10.1016/s1553-7250(07)33112-7 .
Singer SJ, Hayes J, Cooper JB, Vogt JW, Sales M, Aristidou A, Gray GC, Kiang MV, Meyer GS. A case for safety leadership training of hospital manager. Health Care Manag Rev. 2011;36(2):188–200. https://doi.org/10.1097/HMR.0b013e318208cd1d .
Liang Z, Leggat SG, Howard PF, Lee K. What makes a hospital manager competent at the middle and senior levels? Aust Health Rev. 2013;37(5):566–73. https://doi.org/10.1071/AH12004 .
Liang Z, Howard PF, Koh L, Leggat SG. Competency requirements for middle and senior managers in community health services. Aust J Prim Health. 2013;19(3):256–63. https://doi.org/10.1071/PY12041 .
AbuDagga A, Weech-Maldonado R, Tian F. Organizational characteristics associated with the provision of cultural competency training in home and hospice care agencies. Health Care Manag Rev. 2018;43(4):328–37. https://doi.org/10.1097/HMR.0000000000000144 .
Liang Z, Howard PF, Leggat SG, Bartram T. Development and validation of health service management competencies. J Health Organ Manag. 2018;32(2):157–75. https://doi.org/10.1108/JHOM-06-2017-0120 . (Epub 2018 Feb 8).
Howard PF, Liang Z, Leggat SG, Karimi L. Validation of a management competency assessment tool for health service managers. J Health Organ Manag. 2018;32(1):113–34. https://doi.org/10.1108/JHOM-08-2017-0223 .
Fanelli S, Lanza G, Enna C, Zangrandi A. Managerial competences in public organisations: the healthcare professionals’ perspective. BMC Health Serv Res. 2020;20(303):1–9. https://doi.org/10.1186/s12913-020-05179-5 .
Ravaghi H, Beyranvand T, Mannion R, Alijanzadeh M, Aryankhesal A, Belorgeot VD. Effectiveness of training and educational programs for hospital managers: a systematic review. Health Serv Manag Res. 2020;34(2):1–14. https://doi.org/10.1177/0951484820971460 .
Woltring C, Constantine W, Schwarte L. Does leadership training make a difference? J Public Health Manag Prac. 2003;9(2):103–22.
Chowdhury MZI, Turin TC. Variable selection strategies and its importance in clinical prediction modelling. Fam Med Comm Health. 2020;8(1):1–7. https://doi.org/10.1136/fmch-2019-000262 .
Download references
Not applicable.
DM 737/2021 risorse 2022–2023. Funded by the European Union - NextGenerationEU.
Authors and affiliations.
Department of Economics and Business, University of Sassari (Italy), Via Muroni 25, Sassari, 07100, Italy
Lucia Giovanelli & Nicoletta Fadda
Department of Humanities and Social Sciences, University of Sassari (Italy), Via Roma 151, 07100, Sassari, Italy
Federico Rotondo
You can also search for this author in PubMed Google Scholar
Although all the authors have made substantial contributions to the design and drafting of the manuscript: LG and FR conceptualized the study, FR and NF conducted the analysis and investigation and wrote the original draft; LG, FR and NF reviewed and edited the original draft, and LG supervised the whole process. All the authors read and approved the final manuscript.
Correspondence to Federico Rotondo .
Ethics approval and consent to participate.
The research involved human participants. All authors certify that participants decided to take part in the analysis voluntarily and provided informed consent to participate. Participants were granted total anonymity and were adequately informed of the aims, methods, institutional affiliations of the researchers and any other relevant aspects of the study. In line with the Helsinki Declaration and the Italian legislation (acknowledgement of EU Regulation no. 536/2014 on January 31st, 2022 and Ministerial Decree of November 30th, 2021), ethical approval by a committee was not required since the study was non-medical and non-interventional.
Competing interests.
The authors declare no competing interests.
Publisher’s note.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary material 1., rights and permissions.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Reprints and permissions
Cite this article.
Giovanelli, L., Rotondo, F. & Fadda, N. Management training programs in healthcare: effectiveness factors, challenges and outcomes. BMC Health Serv Res 24 , 904 (2024). https://doi.org/10.1186/s12913-024-11229-z
Download citation
Received : 15 January 2024
Accepted : 20 June 2024
Published : 07 August 2024
DOI : https://doi.org/10.1186/s12913-024-11229-z
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
ISSN: 1472-6963
Research has shown that the unilateral accumulation of international reserves by a country can improve its own macro-financial stability. However, we show that when many countries accumulate reserves, the induced general equilibrium effects weaken financial and macroeconomic stability, especially for countries that do not accumulate reserves. The issuance of public debt by advanced economies has the opposite effect. We derive these results from a two-region model where private defaultable debt has a productive use. Quantitative counterfactuals show that the surge in reserves (public debt) contributed to reduce (increase) world interest rates but also to increase (reduce) private leverage. This in turn increased (decreased) volatility in both emerging and advanced economies.
We thank participants at the Impulse and Propagation Mechanisms Workshop of the 2024 NBER Summer Institute, the 2024 China International Conference in Macroeconomics at ChineseUniversity of Hong Kong, the Eighth CCER Summer Institute at Peking University, the Macroeconomics in Emerging Markets Conference at Columbia University, the conference on Emerging Markets: Capital Flows, Debt Overhang, Inflation, and Growth organized by the NBER, FLAR, and Banco Central de Reserva del Peru, and presentations at the IMF and the Federal Reserve Bank of Minneapolis. We also thank our discussants, Mark Aguiar and Luis Gonzalo Llosa Velásquez, as well as Cristina Arellano, Javier Bianchi, and Illenin Kondo for helpful comments and suggestions. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
MARC RIS BibTeΧ
Download Citation Data
In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship — as well as online conference reports , video lectures , and interviews .
Published: August 08, 2024
One of the most underrated skills you can have as a marketer is marketing research — which is great news for this unapologetic cyber sleuth.
From brand design and product development to buyer personas and competitive analysis, I’ve researched a number of initiatives in my decade-long marketing career.
And let me tell you: having the right marketing research methods in your toolbox is a must.
Market research is the secret to crafting a strategy that will truly help you accomplish your goals. The good news is there is no shortage of options.
Thanks to the Internet, we have more marketing research (or market research) methods at our fingertips than ever, but they’re not all created equal. Let’s quickly go over how to choose the right one.
5 Research and Planning Templates + a Free Guide on How to Use Them in Your Market Research
All fields are required.
Click this link to access this resource at any time.
What are you researching? Do you need to understand your audience better? How about your competition? Or maybe you want to know more about your customer’s feelings about a specific product.
Before starting your research, take some time to identify precisely what you’re looking for. This could be a goal you want to reach, a problem you need to solve, or a question you need to answer.
For example, an objective may be as foundational as understanding your ideal customer better to create new buyer personas for your marketing agency (pause for flashbacks to my former life).
Or if you’re an organic sode company, it could be trying to learn what flavors people are craving.
Next, determine what data type will best answer the problems or questions you identified. There are primarily two types: qualitative and quantitative. (Sound familiar, right?)
Understanding the differences between qualitative and quantitative data will help you pinpoint which research methods will yield the desired results.
For instance, thinking of our earlier examples, qualitative data would usually be best suited for buyer personas, while quantitative data is more useful for the soda flavors.
However, truth be told, the two really work together.
Qualitative conclusions are usually drawn from quantitative, numerical data. So, you’ll likely need both to get the complete picture of your subject.
For example, if your quantitative data says 70% of people are Team Black and only 30% are Team Green — Shout out to my fellow House of the Dragon fans — your qualitative data will say people support Black more than Green.
(As they should.)
You’ll also want to understand the difference between primary and secondary research.
Primary research involves collecting new, original data directly from the source (say, your target market). In other words, it’s information gathered first-hand that wasn’t found elsewhere.
Some examples include conducting experiments, surveys, interviews, observations, or focus groups.
Meanwhile, secondary research is the analysis and interpretation of existing data collected from others. Think of this like what we used to do for school projects: We would read a book, scour the internet, or pull insights from others to work from.
So, which is better?
Personally, I say any research is good research, but if you have the time and resources, primary research is hard to top. With it, you don’t have to worry about your source's credibility or how relevant it is to your specific objective.
You are in full control and best equipped to get the reliable information you need.
Once you know your objective and what kind of data you want, you’re ready to select your marketing research method.
For instance, let’s say you’re a restaurant trying to see how attendees felt about the Speed Dating event you hosted last week.
You shouldn’t run a field experiment or download a third-party report on speed dating events; those would be useless to you. You need to conduct a survey that allows you to ask pointed questions about the event.
This would yield both qualitative and quantitative data you can use to improve and bring together more love birds next time around.
Now that you know what you’re looking for in a marketing research method, let’s dive into the best options.
Note: According to HubSpot’s 2024 State of Marketing report, understanding customers and their needs is one of the biggest challenges facing marketers today. The options we discuss are great consumer research methodologies , but they can also be used for other areas.
1. interviews.
Interviews are a form of primary research where you ask people specific questions about a topic or theme. They typically deliver qualitative information.
I’ve conducted many interviews for marketing purposes, but I’ve also done many for journalistic purposes, like this profile on comedian Zarna Garg . There’s no better way to gather candid, open-ended insights in my book, but that doesn’t mean they’re a cure-all.
What I like: Real-time conversations allow you to ask different questions if you’re not getting the information you need. They also push interviewees to respond quickly, which can result in more authentic answers.
What I dislike: They can be time-consuming and harder to measure (read: get quantitative data) unless you ask pointed yes or no questions.
Best for: Creating buyer personas or getting feedback on customer experience, a product, or content.
Focus groups are similar to conducting interviews but on a larger scale.
In marketing and business, this typically means getting a small group together in a room (or Zoom), asking them questions about various topics you are researching. You record and/or observe their responses to then take action.
They are ideal for collecting long-form, open-ended feedback, and subjective opinions.
One well-known focus group you may remember was run by Domino’s Pizza in 2009 .
After poor ratings and dropping over $100 million in revenue, the brand conducted focus groups with real customers to learn where they could have done better.
It was met with comments like “worst excuse for pizza I’ve ever had” and “the crust tastes like cardboard.” But rather than running from the tough love, it took the hit and completely overhauled its recipes.
The team admitted their missteps and returned to the market with better food and a campaign detailing their “Pizza Turn Around.”
The result? The brand won a ton of praise for its willingness to take feedback, efforts to do right by its consumers, and clever campaign. But, most importantly, revenue for Domino’s rose by 14.3% over the previous year.
The brand continues to conduct focus groups and share real footage from them in its promotion:
What I like: Similar to interviewing, you can dig deeper and pivot as needed due to the real-time nature. They’re personal and detailed.
What I dislike: Once again, they can be time-consuming and make it difficult to get quantitative data. There is also a chance some participants may overshadow others.
Best for: Product research or development
Pro tip: Need help planning your focus group? Our free Market Research Kit includes a handy template to start organizing your thoughts in addition to a SWOT Analysis Template, Survey Template, Focus Group Template, Presentation Template, Five Forces Industry Analysis Template, and an instructional guide for all of them. Download yours here now.
Surveys are a form of primary research where individuals are asked a collection of questions. It can take many different forms.
They could be in person, over the phone or video call, by email, via an online form, or even on social media. Questions can be also open-ended or closed to deliver qualitative or quantitative information.
A great example of a close-ended survey is HubSpot’s annual State of Marketing .
In the State of Marketing, HubSpot asks marketing professionals from around the world a series of multiple-choice questions to gather data on the state of the marketing industry and to identify trends.
The survey covers various topics related to marketing strategies, tactics, tools, and challenges that marketers face. It aims to provide benchmarks to help you make informed decisions about your marketing.
It also helps us understand where our customers’ heads are so we can better evolve our products to meet their needs.
Apple is no stranger to surveys, either.
In 2011, the tech giant launched Apple Customer Pulse , which it described as “an online community of Apple product users who provide input on a variety of subjects and issues concerning Apple.”
"For example, we did a large voluntary survey of email subscribers and top readers a few years back."
While these readers gave us a long list of topics, formats, or content types they wanted to see, they sometimes engaged more with content types they didn’t select or favor as much on the surveys when we ran follow-up ‘in the wild’ tests, like A/B testing.”
Pepsi saw similar results when it ran its iconic field experiment, “The Pepsi Challenge” for the first time in 1975.
The beverage brand set up tables at malls, beaches, and other public locations and ran a blindfolded taste test. Shoppers were given two cups of soda, one containing Pepsi, the other Coca-Cola (Pepsi’s biggest competitor). They were then asked to taste both and report which they preferred.
People overwhelmingly preferred Pepsi, and the brand has repeated the experiment multiple times over the years to the same results.
What I like: It yields qualitative and quantitative data and can make for engaging marketing content, especially in the digital age.
What I dislike: It can be very time-consuming. And, if you’re not careful, there is a high risk for scientific error.
Best for: Product testing and competitive analysis
Pro tip: " Don’t make critical business decisions off of just one data set," advises Pamela Bump. "Use the survey, competitive intelligence, external data, or even a focus group to give you one layer of ideas or a short-list for improvements or solutions to test. Then gather your own fresh data to test in an experiment or trial and better refine your data-backed strategy."
8. public domain or third-party research.
While original data is always a plus, there are plenty of external resources you can access online and even at a library when you’re limited on time or resources.
Some reputable resources you can use include:
It’s also smart to turn to reputable organizations that are specific to your industry or field. For instance, if you’re a gardening or landscaping company, you may want to pull statistics from the Environmental Protection Agency (EPA).
If you’re a digital marketing agency, you could look to Google Research or HubSpot Research . (Hey, I know them!)
What I like: You can save time on gathering data and spend more time on analyzing. You can also rest assured the data is from a source you trust.
What I dislike: You may not find data specific to your needs.
Best for: Companies under a time or resource crunch, adding factual support to content
Pro tip: Fellow HubSpotter Iskiev suggests using third-party data to inspire your original research. “Sometimes, I use public third-party data for ideas and inspiration. Once I have written my survey and gotten all my ideas out, I read similar reports from other sources and usually end up with useful additions for my own research.”
If the data you need isn’t available publicly and you can’t do your own market research, you can also buy some. There are many reputable analytics companies that offer subscriptions to access their data. Statista is one of my favorites, but there’s also Euromonitor , Mintel , and BCC Research .
What I like: Same as public domain research
What I dislike: You may not find data specific to your needs. It also adds to your expenses.
Best for: Companies under a time or resource crunch or adding factual support to content
You’re not going to like my answer, but “it depends.” The best marketing research method for you will depend on your objective and data needs, but also your budget and timeline.
My advice? Aim for a mix of quantitative and qualitative data. If you can do your own original research, awesome. But if not, don’t beat yourself up. Lean into free or low-cost tools . You could do primary research for qualitative data, then tap public sources for quantitative data. Or perhaps the reverse is best for you.
Whatever your marketing research method mix, take the time to think it through and ensure you’re left with information that will truly help you achieve your goals.
Related articles.
Free Guide & Templates to Help Your Market Research
Marketing software that helps you drive revenue, save time and resources, and measure and optimize your investments — all on one easy-to-use platform
IMAGES
COMMENTS
Following is a Research Findings Example sample for students: Title: The Effects of Exercise on Mental Health. Sample: 500 participants, both men and women, between the ages of 18-45. Methodology: Participants were divided into two groups. The first group engaged in 30 minutes of moderate intensity exercise five times a week for eight weeks.
Step 1: Consult the guidelines or instructions that the target journal or publisher provides authors and read research papers it has published, especially those with similar topics, methods, or results to your study. The guidelines will generally outline specific requirements for the results or findings section, and the published articles will ...
The Results (also sometimes called Findings) section in an empirical research paper describes what the researcher(s) found when they analyzed their data. Its primary purpose is to use the data collected to answer the ... • Make sure to review examples of Results sections from sample papers or journal articles in your discipline, as ...
3). Research Questions as Headings . You can also present your findings using your research questions as the headings in the findings section. This is a useful strategy that ensures you're answering your research questions and also allows the reader to quickly ascertain where the answers to your research questions are.
A two-sample t test was used to test the hypothesis that higher social distance from environmental problems would reduce the intent to donate to environmental organizations, with donation intention (recorded as a score from 1 to 10) as the outcome variable and social distance (categorized as either a low or high level of social distance) as the predictor variable.Social distance was found to ...
If not, we can help. Our panel of experts makes sure to keep the 3 pillars of the Dissertation strong. 1. Reporting Quantitative Findings. The best way to present your quantitative findings is to structure them around the research hypothesis or questions you intend to address as part of your dissertation project.
Reporting Research Results in APA Style | Tips & Examples. Published on December 21, 2020 by Pritha Bhandari.Revised on January 17, 2024. The results section of a quantitative research paper is where you summarize your data and report the findings of any relevant statistical analyses.. The APA manual provides rigorous guidelines for what to report in quantitative research papers in the fields ...
Provide a brief description of your study: Enter details about your research topic and findings. This information helps Paperpal generate a tailored outline that aligns with your paper's content. Generate the conclusion outline: After entering all necessary details, click on 'generate'.
As a general guide, your results chapter will typically include the following: Some demographic data about your sample; Reliability tests (if you used measurement scales); Descriptive statistics; Inferential statistics (if your research objectives and questions require these); Hypothesis tests (again, if your research objectives and questions require these); We'll discuss each of these ...
The results chapter in a dissertation or thesis (or any formal academic research piece) is where you objectively and neutrally present the findings of your qualitative analysis (or analyses if you used multiple qualitative analysis methods ). This chapter can sometimes be combined with the discussion chapter (where you interpret the data and ...
The results section of a research paper tells the reader what you found, while the discussion section tells the reader what your findings mean. The results section should present the facts in an academic and unbiased manner, avoiding any attempt at analyzing or interpreting the data. Think of the results section as setting the stage for the ...
Sample: a subset of individuals selected from a larger population for study or investigation. Those included in the sample are termed "participants." Generalizability: the ability to apply research findings from a sample to the broader target population, contingent on the sample being representative of that population.
The discussion section is one of the final parts of a research paper, in which an author describes, analyzes, and interprets their findings. They explain the significance of those results and tie everything back to the research question(s). In this handout, you will find a description of what a discussion section does, explanations of how to ...
These formats are supported by research that focused on improved understanding of the information they intend to convey (Carrasco-Labra et al 2016, Langendam et al 2016, Santesso et al 2016). ... Summary of findings ... Indicate where the sample size or number of events does not meet the optimal information size as calculated, or the 'rules ...
Taking time to reflect on your findings and what these might possibly mean requires some serious mind work—so do not try and rush this phase. Spend a few days away from your research, giving careful thought to the findings, trying to put them in perspective, and trying to gain some deeper insights. To begin facilitating the kind of thinking ...
INTRODUCTION. Scientific research is usually initiated by posing evidenced-based research questions which are then explicitly restated as hypotheses.1,2 The hypotheses provide directions to guide the study, solutions, explanations, and expected results.3,4 Both research questions and hypotheses are essentially formulated based on conventional theories and real-world processes, which allow the ...
Research Summary. Definition: A research summary is a brief and concise overview of a research project or study that highlights its key findings, main points, and conclusions. It typically includes a description of the research problem, the research methods used, the results obtained, and the implications or significance of the findings.
The discussion section provides an analysis and interpretation of the findings, compares them with previous studies, identifies limitations, and suggests future directions for research. This section combines information from the preceding parts of your paper into a coherent story. By this point, the reader already knows why you did your study ...
Begin with a clear statement of the principal findings. This will reinforce the main take-away for the reader and set up the rest of the discussion. Explain why the outcomes of your study are important to the reader. Discuss the implications of your findings realistically based on previous literature, highlighting both the strengths and ...
The conclusions are as stated below: i. Students' use of language in the oral sessions depicted their beliefs and values. based on their intentions. The oral sessions prompted the students to be ...
Step 4: Write your findings section in a factual and objective manner. The goal is to communicate information - in some cases a great deal of complex information - as clearly, accurately and precisely as possible, so well-constructed sentences that maintain a simple structure will be far more effective than convoluted phrasing and expressions.
The first question asks for a ready-made solution, and is not focused or researchable. The second question is a clearer comparative question, but note that it may not be practically feasible. For a smaller research project or thesis, it could be narrowed down further to focus on the effectiveness of drunk driving laws in just one or two countries.
6.2.1 Findings based on the questionnaires completed by teachers (quantitative data) In the sampled schools most of the variables that could have been expected, according to the literature, to have an effect on Grade 12 results, in fact had had no effect, at least among the sampled schools. The most significant variables which emerged from the ...
Research Proposal Example Here is a research proposal sample template (with examples) from the University of Rochester Medical Center. 4 The sections in all research proposals are essentially the same although different terminology and other specific sections may be used depending on the subject. Structure of a Research Proposal
Leaders mistakenly conflate causation with correlation, underestimate the importance of sample size, focus on the wrong outcomes, misjudge generalizability, or overweight a specific result. The ...
A survey was used for gathering information from a purposive sample of professionals in the healthcare field attending management training programs in Italy. ... there is a need to extend research and test these findings on different types of management training programs, participants and countries. Second, this study is based on a survey of ...
Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, and business professionals.
Personally, I say any research is good research, but if you have the time and resources, primary research is hard to top. With it, you don't have to worry about your source's credibility or how relevant it is to your specific objective. You are in full control and best equipped to get the reliable information you need. 3. Put it all together.