in mixed methods research quantitative and qualitative findings should be

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Three techniques for...

Three techniques for integrating data in mixed methods studies

• Related content
• Peer review
• Alicia O’Cathain , professor 1 ,
• Elizabeth Murphy , professor 2 ,
• Jon Nicholl , professor 1
• 1 Medical Care Research Unit, School of Health and Related Research, University of Sheffield, Sheffield S1 4DA, UK
• 2 University of Leicester, Leicester, UK
• Correspondence to: A O’Cathain a.ocathain{at}sheffield.ac.uk
• Accepted 8 June 2010

Techniques designed to combine the results of qualitative and quantitative studies can provide researchers with more knowledge than separate analysis

Health researchers are increasingly using designs that combine qualitative and quantitative methods, and this is often called mixed methods research. 1 Integration—the interaction or conversation between the qualitative and quantitative components of a study—is an important aspect of mixed methods research, and, indeed, is essential to some definitions. 2 Recent empirical studies of mixed methods research in health show, however, a lack of integration between components, 3 4 which limits the amount of knowledge that these types of studies generate. Without integration, the knowledge yield is equivalent to that from a qualitative study and a quantitative study undertaken independently, rather than achieving a “whole greater than the sum of the parts.” 5

Barriers to integration have been identified in both health and social research. 6 7 One barrier is the absence of formal education in mixed methods research. Fortunately, literature is rapidly expanding to fill this educational gap, including descriptions of how to integrate data and findings from qualitative and quantitative methods. 8 9 In this article we outline three techniques that may help health researchers to integrate data or findings in their mixed methods studies and show how these might enhance knowledge generated from this approach.

Triangulation protocol

Researchers will often use qualitative and quantitative methods to examine different aspects of an overall research question. For example, they might use a randomised controlled trial to assess the effectiveness of a healthcare intervention and semistructured interviews with patients and health professionals to consider the way in which the intervention was used in the real world. Alternatively, they might use a survey of service users to measure satisfaction with a service and focus groups to explore views of care in more depth. Data are collected and analysed separately for each component to produce two sets of findings. Researchers will then attempt to combine these findings, sometimes calling this process triangulation. The term triangulation can be confusing because it has two meanings. 10 It can be used to describe corroboration between two sets of findings or to describe a process of studying a problem using different methods to gain a more complete picture. The latter meaning is commonly used in mixed methods research and is the meaning used here.

The process of triangulating findings from different methods takes place at the interpretation stage of a study when both data sets have been analysed separately (figure ⇓ ). Several techniques have been described for triangulating findings. They require researchers to list the findings from each component of a study on the same page and consider where findings from each method agree (convergence), offer complementary information on the same issue (complementarity), or appear to contradict each other (discrepancy or dissonance). 11 12 13 Explicitly looking for disagreements between findings from different methods is an important part of this process. Disagreement is not a sign that something is wrong with a study. Exploration of any apparent “inter-method discrepancy” may lead to a better understanding of the research question, 14 and a range of approaches have been used within health services research to explore inter-method discrepancy. 15

Point of application for three techniques for integrating data in mixed methods research

• Download figure
• Open in new tab
• Download powerpoint

The most detailed description of how to carry out triangulation is the triangulation protocol, 11 which although developed for multiple qualitative methods, is relevant to mixed methods studies. This technique involves producing a “convergence coding matrix” to display findings emerging from each component of a study on the same page. This is followed by consideration of where there is agreement, partial agreement, silence, or dissonance between findings from different components. This technique for triangulation is the only one to include silence—where a theme or finding arises from one data set and not another. Silence might be expected because of the strengths of different methods to examine different aspects of a phenomenon, but surprise silences might also arise that help to increase understanding or lead to further investigations.

The triangulation protocol moves researchers from thinking about the findings related to each method, to what Farmer and colleagues call meta-themes that cut across the findings from different methods. 11 They show a worked example of triangulation protocol, but we could find no other published example. However, similar principles were used in an iterative mixed methods study to understand patient and carer satisfaction with a new primary angioplasty service. 16 Researchers conducted semistructured interviews with 16 users and carers to explore their experiences and views of the new service. These were used to develop a questionnaire for a survey of 595 patients (and 418 of their carers) receiving either the new service or usual care. Finally, 17 of the patients who expressed dissatisfaction with aftercare and rehabilitation were followed up to explore this further in semistructured interviews. A shift of thinking to meta-themes led the researchers away from reporting the findings from the interviews, survey, and follow-up interviews sequentially to consider the meta-themes of speed and efficiency, convenience of care, and discharge and after care. The survey identified that a higher percentage of carers of patients using the new service rated the convenience of visiting the hospital as poor than those using usual care. The interviews supported this concern about the new service, but also identified that the weight carers gave to this concern was low in the context of their family member’s life being saved.

Morgan describes this move as the “third effort” because it occurs after analysis of the qualitative and the quantitative components. 17 It requires time and energy that must be planned into the study timetable. It is also useful to consider who will carry out the integration process. Farmer and colleagues require two researchers to work together during triangulation, which can be particularly important in mixed methods studies if different researchers take responsibility for the qualitative and quantitative components. 11

Following a thread

Moran-Ellis and colleagues describe a different technique for integrating the findings from the qualitative and quantitative components of a study, called following a thread. 18 They state that this takes place at the analysis stage of the research process (figure ⇑ ). It begins with an initial analysis of each component to identify key themes and questions requiring further exploration. Then the researchers select a question or theme from one component and follow it across the other components—they call this the thread. The authors do not specify steps in this technique but offer a visual model for working between datasets. An approach similar to this has been undertaken in health services research, although the researchers did not label it as such, probably because the technique has not been used frequently in the literature (box)

An example of following a thread 19

Adamson and colleagues explored the effect of patient views on the appropriate use of services and help seeking using a survey of people registered at a general practice and semistructured interviews. The qualitative (22 interviews) and quantitative components (survey with 911 respondents) took place concurrently.

The researchers describe what they call an iterative or cyclical approach to analysis. Firstly, the preliminary findings from the interviews generated a hypothesis for testing in the survey data. A key theme from the interviews concerned the self rationing of services as a responsible way of using scarce health care. This theme was then explored in the survey data by testing the hypothesis that people’s views of the appropriate use of services would explain their help seeking behaviour. However, there was no support for this hypothesis in the quantitative analysis because the half of survey respondents who felt that health services were used inappropriately were as likely to report help seeking for a series of symptoms presented in standardised vignettes as were respondents who thought that services were not used inappropriately. The researchers then followed the thread back to the interview data to help interpret this finding.

After further analysis of the interview data the researchers understood that people considered the help seeking of other people to be inappropriate, rather than their own. They also noted that feeling anxious about symptoms was considered to be a good justification for seeking care. The researchers followed this thread back into the survey data and tested whether anxiety levels about the symptoms in the standardised vignettes predicted help seeking behaviour. This second hypothesis was supported by the survey data. Following a thread led the researchers to conclude that patients who seek health care for seemingly minor problems have exceeded their thresholds for the trade-off between not using services inappropriately and any anxiety caused by their symptoms.

Mixed methods matrix

A unique aspect of some mixed methods studies is the availability of both qualitative and quantitative data on the same cases. Data from the qualitative and quantitative components can be integrated at the analysis stage of a mixed methods study (figure ⇑ ). For example, in-depth interviews might be carried out with a sample of survey respondents, creating a subset of cases for which there is both a completed questionnaire and a transcript. Cases may be individuals, groups, organisations, or geographical areas. 9 All the data collected on a single case can be studied together, focusing attention on cases, rather than variables or themes, within a study. The data can be examined in detail for each case—for example, comparing people’s responses to a questionnaire with their interview transcript. Alternatively, data on each case can be summarised and displayed in a matrix 8 9 20 along the lines of Miles and Huberman’s meta-matrix. 21 Within a mixed methods matrix, the rows represent the cases for which there is both qualitative and quantitative data, and the columns display different data collected on each case. This allows researchers to pay attention to surprises and paradoxes between types of data on a single case and then look for patterns across all cases 20 in a qualitative cross case analysis. 21

We used a mixed methods matrix to study the relation between types of team working and the extent of integration in mixed methods studies in health services research (table ⇓ ). 22 Quantitative data were extracted from the proposals, reports, and peer reviewed publications of 75 mixed methods studies, and these were analysed to describe the proportion of studies with integrated outputs such as mixed methods journal articles. Two key variables in the quantitative component were whether the study was assessed as attempting to integrate qualitative or quantitative data or findings and the type of publications produced. We conducted qualitative interviews with 20 researchers who had worked on some of these studies to explore how mixed methods research was practised, including how the team worked together.

Example of a mixed methods matrix for a study exploring the relationship between types of teams and integration between qualitative and quantitative components of studies* 22

• View inline

The shared cases between the qualitative and quantitative components were 21 mixed methods studies (because one interviewee had worked on two studies in the quantitative component). A matrix was formed with each of the 21 studies as a row. The first column of the matrix contained the study identification, the second column indicated whether integration had occurred in that project, and the third column the score for integration of publications emerging from the study. The rows were then ordered to show the most integrated cases first. This ordering of rows helped us to see patterns across rows.

The next columns were themes from the qualitative interview with a researcher from that project. For example, the first theme was about the expertise in qualitative research within the team and whether the interviewee reported this as adequate for the study. The matrix was then used in the context of the qualitative analysis to explore the issues that affected integration. In particular, it helped to identify negative cases (when someone in the analysis doesn’t fit with the conclusions the analysis is coming to) within the qualitative analysis to facilitate understanding. Interviewees reported the need for experienced qualitative researchers on mixed methods studies to ensure that the qualitative component was published, yet two cases showed that this was neither necessary nor sufficient. This pushed us to explore other factors in a research team that helped generate outputs, and integrated outputs, from a mixed methods study.

Themes from a qualitative study can be summarised to the point where they are coded into quantitative data. In the matrix (table ⇑ ), the interviewee’s perception of the adequacy of qualitative expertise on the team could have been coded as adequate=1 or not=2. This is called “quantitising” of qualitative data 23 ; coded data can then be analysed with data from the quantitative component. This technique has been used to great effect in healthcare research to identify the discrepancy between health improvement assessed using quantitative measures and with in-depth interviews in a randomised controlled trial. 24

We have presented three techniques for integration in mixed methods research in the hope that they will inspire researchers to explore what can be learnt from bringing together data from the qualitative and quantitative components of their studies. Using these techniques may give the process of integration credibility rather than leaving researchers feeling that they have “made things up.” It may also encourage researchers to describe their approaches to integration, allowing them to be transparent and helping them to develop, critique, and improve on these techniques. Most importantly, we believe it may help researchers to generate further understanding from their research.

We have presented integration as unproblematic, but it is not. It may be easier for single researchers to use these techniques than a large research team. Large teams will need to pay attention to team dynamics, considering who will take responsibility for integration and who will be taking part in the process. In addition, we have taken a technical stance here rather than paying attention to different philosophical beliefs that may shape approaches to integration. We consider that these techniques would work in the context of a pragmatic or subtle realist stance adopted by some mixed methods researchers. 25 Finally, it is important to remember that these techniques are aids to integration and are helpful only when applied with expertise.

Summary points

Health researchers are increasingly using designs which combine qualitative and quantitative methods

However, there is often lack of integration between methods

Three techniques are described that can help researchers to integrate data from different components of a study: triangulation protocol, following a thread, and the mixed methods matrix

Use of these methods will allow researchers to learn more from the information they have collected

Cite this as: BMJ 2010;341:c4587

Funding: Medical Research Council grant reference G106/1116

Competing interests: All authors have completed the unified competing interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare financial support for the submitted work from the Medical Research Council; no financial relationships with commercial entities that might have an interest in the submitted work; no spouses, partners, or children with relationships with commercial entities that might have an interest in the submitted work; and no non-financial interests that may be relevant to the submitted work.

Contributors: AOC wrote the paper. JN and EM contributed to drafts and all authors agreed the final version. AOC is guarantor.

Provenance and peer review: Not commissioned; externally peer reviewed.

• ↵ Lingard L, Albert M, Levinson W. Grounded theory, mixed methods and action research. BMJ 2008 ; 337 : a567 . OpenUrl FREE Full Text
• ↵ Creswell JW, Fetters MD, Ivankova NV. Designing a mixed methods study in primary care. Ann Fam Med 2004 ; 2 : 7 -12. OpenUrl Abstract / FREE Full Text
• ↵ Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ 2009 ; 339 : b3496 . OpenUrl Abstract / FREE Full Text
• ↵ O’Cathain A, Murphy E, Nicholl J. Integration and publications as indicators of ‘yield’ from mixed methods studies. J Mix Methods Res 2007 ; 1 : 147 -63. OpenUrl CrossRef Web of Science
• ↵ Barbour RS. The case for combining qualitative and quantitative approaches in health services research. J Health Serv Res Policy 1999 ; 4 : 39 -43. OpenUrl PubMed
• ↵ O’Cathain A, Nicholl J, Murphy E. Structural issues affecting mixed methods studies in health research: a qualitative study. BMC Med Res Methodol 2009 ; 9 : 82 . OpenUrl CrossRef PubMed
• ↵ Bryman A. Barriers to integrating quantitative and qualitative research. J Mix Methods Res 2007 ; 1 : 8 -22. OpenUrl CrossRef
• ↵ Creswell JW, Plano-Clark V. Designing and conducting mixed methods research . Sage, 2007 .
• ↵ Bazeley P. Analysing mixed methods data. In: Andrew S, Halcomb EJ, eds. Mixed methods research for nursing and the health sciences . Wiley-Blackwell, 2009 :84-118.
• ↵ Sandelowski M. Triangles and crystals: on the geometry of qualitative research. Res Nurs Health 1995 ; 18 : 569 -74. OpenUrl CrossRef PubMed Web of Science
• ↵ Farmer T, Robinson K, Elliott SJ, Eyles J. Developing and implementing a triangulation protocol for qualitative health research. Qual Health Res 2006 ; 16 : 377 -94. OpenUrl Abstract / FREE Full Text
• ↵ Foster RL. Addressing the epistemologic and practical issues in multimethod research: a procedure for conceptual triangulation. Adv Nurs Sci 1997 ; 20 : 1 -12. OpenUrl PubMed
• ↵ Erzerberger C, Prein G. Triangulation: validity and empirically based hypothesis construction. Qual Quant 1997 ; 31 : 141 -54. OpenUrl CrossRef Web of Science
• ↵ Fielding NG, Fielding JL. Linking data . Sage, 1986 .
• ↵ Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and qualitative data in health services research—what happens when mixed method findings conflict? BMC Health Serv Res 2006 ; 6 : 28 . OpenUrl CrossRef PubMed
• ↵ Sampson FC, O’Cathain A, Goodacre S. Is primary angioplasty an acceptable alternative to thrombolysis? Quantitative and qualitative study of patient and carer satisfaction. Health Expectations (forthcoming).
• ↵ Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res 1998 ; 8 : 362 -76. OpenUrl Abstract / FREE Full Text
• ↵ Moran-Ellis J, Alexander VD, Cronin A, Dickinson M, Fielding J, Sleney J, et al. Triangulation and integration: processes, claims and implications. Qualitative Research 2006 ; 6 : 45 -59. OpenUrl Abstract / FREE Full Text
• ↵ Adamson J, Ben-Shlomo Y, Chaturvedi N, Donovan J. Exploring the impact of patient views on ‘appropriate’ use of services and help seeking: a mixed method study. Br J Gen Pract 2009 ; 59 : 496 -502. OpenUrl Web of Science
• ↵ Wendler MC. Triangulation using a meta-matrix. J Adv Nurs 2001 ; 35 : 521 -5. OpenUrl CrossRef PubMed Web of Science
• ↵ Miles M, Huberman A. Qualitative data analysis: an expanded sourcebook . Sage, 1994 .
• ↵ O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary or dysfunctional? Team working in mixed methods research. Qual Health Res 2008 ; 18 : 1574 -85. OpenUrl Abstract / FREE Full Text
• ↵ Sandelowski M. Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res Nurs Health 2000 ; 23 : 246 -55. OpenUrl CrossRef PubMed Web of Science
• ↵ Campbell R, Quilty B, Dieppe P. Discrepancies between patients’ assessments of outcome: qualitative study nested within a randomised controlled trial. BMJ 2003 ; 326 : 252 -3. OpenUrl FREE Full Text
• ↵ Mays N, Pope C. Assessing quality in qualitative research. BMJ 2000 ; 320 : 50 -2. OpenUrl FREE Full Text

The Use of Mixed Methods in Research

• Reference work entry
• First Online: 13 January 2019
• Cite this reference work entry

• Kate A. McBride 2 ,
• Freya MacMillan 3 ,
• Emma S. George 4 &
• Genevieve Z. Steiner 5  

2511 Accesses

9 Citations

Mixed methods research is becoming increasingly popular and is widely acknowledged as a means of achieving a more complex understanding of research problems. Combining both the in-depth, contextual views of qualitative research with the broader generalizations of larger population quantitative approaches, mixed methods research can be used to produce a rigorous and credible source of data. Using this methodology, the same core issue is investigated through the collection, analysis, and interpretation of both types of data within one study or a series of studies. Multiple designs are possible and can be guided by philosophical assumptions. Both qualitative and quantitative data can be collected simultaneously or sequentially (in any order) through a multiphase project. Integration of the two data sources then occurs with consideration is given to the weighting of both sources; these can either be equal or one can be prioritized over the other. Designed as a guide for novice mixed methods researchers, this chapter gives an overview of the historical and philosophical roots of mixed methods research. We also provide a practical overview of its application in health research as well as pragmatic considerations for those wishing to undertake mixed methods research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Appleton JV, King L. Journeying from the philosophical contemplation of constructivism to the methodological pragmatics of health services research. J Adv Nurs. 2002;40(6):641–8.

Article   Google Scholar  

Baba CT, Oliveira IM, Silva AEF, Vieira LM, Cerri NC, Florindo AA, de Oliveira Gomes GA. Evaluating the impact of a walking program in a disadvantaged area: using the RE-AIM framework by mixed methods. BMC Public Health. 2017;17(1):709. https://doi.org/10.1186/s12889-017-4698-5 .

Bryman A. The research question in social research: what is its role? Int J Soc Res Methodol. 2007;10(1):5–20. https://doi.org/10.1080/13645570600655282 .

Bryman A. The end of the paradigm wars. In: Alasuutari P, Bickman L, Brannen J, editors. The Sage handbook of social research methods. Thousand Oaks: Sage; 2008. p. 13–25.

Google Scholar  

Caracelli VJ, Greene JC. Data-analysis strategies for mixed-method evaluation designs. Educ Eval Policy Anal. 1993;15(2):195–207. https://doi.org/10.3102/01623737015002195 .

Castro FG, Kellison JG, Boyd SJ, Kopak A. A methodology for conducting integrative mixed methods research and data analyses. J Mixed Methods Res. 2010;4(4):342–60. https://doi.org/10.1177/1558689810382916 .

Center for Innovation in Teaching in Research. Choosing a mixed methods design. 2017. Retrieved 15 Nov 2017 from https://cirt.gcu.edu/research/developmentresources/research_ready/mixed_methods/choosing_design .

Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage; 2018.

Curry LA, Krumholz HM, O’Cathain A, Clark VLP, Cherlin E, Bradley EH. Mixed methods in biomedical and health services research. Circ-Cardiovasc Qual Outcomes. 2013;6(1):119–23. https://doi.org/10.1161/circoutcomes.112.967885 .

Denscombe M. Communities of practice: a research paradigm for the mixed methods approach. J Mixed Methods Res. 2008;2(3):270–83.

Doyle L, Brady A-M, Byrne G. An overview of mixed methods research. J Res Nurs. 2009;14(2):175–85.

Foster NE, Bishop A, Bartlam B, Ogollah R, Barlas P, Holden M, … Young J. Evaluating acupuncture and standard carE for pregnant women with back pain (EASE back): a feasibility study and pilot randomised trial. Health Technol Assess. 2016;20(33):1–236. https://doi.org/10.3310/hta20330 .

Haider AH, Schneider EB, Kodadek LM, Adler RR, Ranjit A, Torain M, … Lau BD. Emergency department query for patient-centered approaches to sexual orientation and gender identity the EQUALITY study. JAMA Intern Med. 2017;177(6):819–828. https://doi.org/10.1001/jamainternmed.2017.0906 .

Harding KE, Taylor NF, Bowers B, Stafford M, Leggat SG. Clinician and patient perspectives of a new model of triage in a community rehabilitation program that reduced waiting time: a qualitative analysis. Aust Health Rev. 2013;37(3):324–30. https://doi.org/10.1071/ah13033 .

Hoddinott P, Britten J, Prescott GJ, Tappin D, Ludbrook A, Godden DJ. Effectiveness of policy to provide breastfeeding groups ( BIG) for pregnant and breastfeeding mothers in primary care: cluster randomised controlled trial. Br Med J. 2009;338:a3026, 10. https://doi.org/10.1136/bmj.a3026 .

Ivankova NV, Creswell JW, Stick SL. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20. https://doi.org/10.1177/1525822x05282260 .

Johnson B, Gray R. A history of philosophical and theoretical issues for mixed methods research. In: Tashakkori A, Teddlie C, editors. Sage handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 69–94.

Chapter   Google Scholar  

Johnson BR, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7):14–26. https://doi.org/10.3102/0013189x033007014 .

Keeney S, McKenna H, Fleming P, McIlfatrick S. Attitudes to cancer and cancer prevention: what do people aged 35–54 years think? Eur J Cancer Care. 2010;19(6):769–77. https://doi.org/10.1111/j.1365-2354.2009.01137.x .

McBride KA, Ballinger ML, Schlub TE, Young MA, Tattersall MHN, Kirk J, et al. Psychosocial morbidity in TP53 mutation carriers: is whole-body cancer screening beneficial? Familial Cancer. 2017;16(3):423–32. https://doi.org/10.1007/s10689-016-9964-7 .

Mertens DM. Transformative mixed methods: addressing inequities. Am Behav Sci. 2012;56(6):802–13. https://doi.org/10.1177/0002764211433797 .

Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and qualitative data in health services research – what happens when mixed method findings conflict? BMC Health Serv Res. 2006;6:28. https://doi.org/10.1186/1472-6963-6-28 .

Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res. 1991a;40(2):120–3.

Morse JM. Principles of mixed methods and multimethod research design. In: Tashakkori A, Teddilie C, editors. SAGE handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage; 1991b.

Mutrie N, Doolin O, Fitzsimons CF, Grant PM, Granat M, Grealy M, et al. Increasing older adults’ walking through primary care: results of a pilot randomized controlled trial. Fam Pract. 2012;29(6):633–42. https://doi.org/10.1093/fampra/cms038 .

Newman I, Ridenour C, Newman C, De Marco G. A typology of research purposes and its relationship to mixed methods in social and behavioral research. Thousand Oaks: Sage; 2003. p. 167–88.

O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8. https://doi.org/10.1258/jhsrp.2007.007074 .

Onwuegbuzie AJ, Collins KMT. A typology of mixed methods sampling designs in social science research. Qual Rep. 2007;12(2):281–316.

Plano Clark VL, Badiee M. Research questions in mixed methods research. In: Tashakkori A, Teddlie C, editors. SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 275–304.

Prades, J., Algara, M., Espinas, J. A., Farrus, B., Arenas, M., Reyes, V., . . . Borras, J. M. (2017). Understanding variations in the use of hypofractionated radiotherapy and its specific indications for breast cancer: a mixed-methods study. Radiother Oncol, 123(1), 22–28. doi: https://doi.org/10.1016/j.radonc.2017.01.014 .

Tariq S, Woodman J. Using mixed methods in health research. JRSM Short Rep. 2013; 4 (6):1–8. https://doi.org/10.1177/2042533313479197 .

Tashakkori A, Creswell JW. Editorial: the new era of mixed methods. J Mixed Methods Res. 2007;1(1):3–7. https://doi.org/10.1177/2345678906293042 .

Tashakkori A, Teddlie C. The past and future of mixed methods research: from data triangulation to mixed model designs. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social & behavioral research. Thousand Oaks: Sage; 2003. p. 671–701.

Teddlie C, Tashakkori A. Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks: Sage; 2009.

Teddlie C, Yu F. Mixed methods sampling a typology with examples. J Mixed Methods Res. 2007;1(1):77–100. https://doi.org/10.1177/2345678906292430 .

Wellard SJ, Rasmussen B, Savage S, Dunning T. Exploring staff diabetes medication knowledge and practices in regional residential care: triangulation study. J Clin Nurs. 2013;22(13–14):1933–40. https://doi.org/10.1111/jocn.12043 .

Zhang W. Mixed methods application in health intervention research: a multiple case study. Int J Mult Res Approaches. 2014;8(1):24–35. https://doi.org/10.5172/mra.2014.8.1.24 .

Download references

Author information

Authors and affiliations.

School of Medicine and Translational Health Research Institute, Western Sydney University, Sydney, NSW, Australia

Kate A. McBride

School of Science and Health and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Freya MacMillan

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Emma S. George

NICM and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Genevieve Z. Steiner

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kate A. McBride .

Editor information

Editors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

McBride, K.A., MacMillan, F., George, E.S., Steiner, G.Z. (2019). The Use of Mixed Methods in Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_97

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_97

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 3: Developing a Research Question

3.5 Quantitative, Qualitative, & Mixed Methods Research Approaches

Generally speaking, qualitative and quantitative approaches are the most common methods utilized by researchers. While these two approaches are often presented as a dichotomy, in reality it is much more complicated. Certainly, there are researchers who fall on the more extreme ends of these two approaches, however most recognize the advantages and usefulness of combining both methods (mixed methods). In the following sections we look at quantitative, qualitative, and mixed methodological approaches to undertaking research. Table 2.3 synthesizes the differences between quantitative and qualitative research approaches.

Quantitative Research Approaches

A quantitative approach to research is probably the most familiar approach for the typical research student studying at the introductory level. Arising from the natural sciences, e.g., chemistry and biology), the quantitative approach is framed by the belief that there is one reality or truth that simply requires discovering, known as realism. Therefore, asking the “right” questions is key. Further, this perspective favours observable causes and effects and is therefore outcome-oriented. Typically, aggregate data is used to see patterns and “truth” about the phenomenon under study. True understanding is determined by the ability to predict the phenomenon.

Qualitative Research Approaches

On the other side of research approaches is the qualitative approach. This is generally considered to be the opposite of the quantitative approach. Qualitative researchers are considered phenomenologists, or human-centred researchers. Any research must account for the humanness, i.e., that they have thoughts, feelings, and experiences that they interpret of the participants. Instead of a realist perspective suggesting one reality or truth, qualitative researchers tend to favour the constructionist perspective: knowledge is created, not discovered, and there are multiple realities based on someone’s perspective. Specifically, a researcher needs to understand why, how and to whom a phenomenon applies. These aspects are usually unobservable since they are the thoughts, feelings and experiences of the person. Most importantly, they are a function of their perception of those things rather than what the outside researcher interprets them to be. As a result, there is no such thing as a neutral or objective outsider, as in the quantitative approach. Rather, the approach is generally process-oriented. True understanding, rather than information based on prediction, is based on understanding action and on the interpretive meaning of that action.

Table 3.3 Differences between quantitative and qualitative approaches (from Adjei, n.d).

Note: Researchers in emergency and safety professions are increasingly turning toward qualitative methods. Here is an interesting peer paper related to qualitative research in emergency care.

Qualitative Research in Emergency Care Part I: Research Principles and Common Applications by Choo, Garro, Ranney, Meisel, and Guthrie (2015)

Interview-based Qualitative Research in Emergency Care Part II: Data Collection, Analysis and Results Reporting.

Research Methods for the Social Sciences: An Introduction Copyright © 2020 by Valerie Sheppard is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Community Engagement Program

Supporting bi-directional community engagement to improve the relevance, quality, and impact of research.

• Getting Started
• Resources for Equity in Research
• Community-Engaged Student Practice Placement
• Maternal Health Equity
• Youth Mental Health
• Leadership and Membership
• Past Members
• Study Review Rubric
• Community Ambassador Initiative
• Implementation Science Working Group
• Past Webinars & Podcasts
• Policy Atlas
• Community Advisory Board

For more information:

Mixed methods research.

According to the National Institutes of Health , mixed methods strategically integrates or combines rigorous quantitative and qualitative research methods to draw on the strengths of each. Mixed method approaches allow researchers to use a diversity of methods, combining inductive and deductive thinking, and offsetting limitations of exclusively quantitative and qualitative research through a complementary approach that maximizes strengths of each data type and facilitates a more comprehensive understanding of health issues and potential resolutions.¹ Mixed methods may be employed to produce a robust description and interpretation of the data, make quantitative results more understandable, or understand broader applicability of small-sample qualitative findings.

Integration

This refers to the ways in which qualitative and quantitative research activities are brought together to achieve greater insight. Mixed methods is not simply having quantitative and qualitative data available or analyzing and presenting data findings separately. The integration process can occur during data collection, analysis, or in the presentation of results.

¹ NIH Office of Behavioral and Social Sciences Research: Best Practices for Mixed Methods Research in the Health Sciences

Basic Mixed Methods Research Designs 

View image description .

Five Key Questions for Getting Started

• What do you want to know?
• What will be the detailed quantitative, qualitative, and mixed methods research questions that you hope to address?
• What quantitative and qualitative data will you collect and analyze?
• Which rigorous methods will you use to collect data and/or engage stakeholders?
• How will you integrate the data in a way that allows you to address the first question?

Rationale for Using Mixed Methods

• Obtain different, multiple perspectives: validation
• Build comprehensive understanding
• Explain statistical results in more depth
• Have better contextualized measures
• Track the process of program or intervention
• Study patient-centered outcomes and stakeholder engagement

Sample Mixed Methods Research Study

The EQUALITY study used an exploratory sequential design to identify the optimal patient-centered approach to collect sexual orientation data in the emergency department.

Qualitative Data Collection and Analysis : Semi-structured interviews with patients of different sexual orientation, age, race/ethnicity, as well as healthcare professionals of different roles, age, and race/ethnicity.

Builds Into : Themes identified in the interviews were used to develop questions for the national survey.

Quantitative Data Collection and Analysis : Representative national survey of patients and healthcare professionals on the topic of reporting gender identity and sexual orientation in healthcare.

Other Resources:

  Introduction to Mixed Methods Research : Harvard Catalyst’s eight-week online course offers an opportunity for investigators who want to understand and apply a mixed methods approach to their research.

Best Practices for Mixed Methods Research in the Health Sciences [PDF] : This guide provides a detailed overview of mixed methods designs, best practices, and application to various types of grants and projects.

Mixed Methods Research Training Program for the Health Sciences (MMRTP ): Selected scholars for this summer training program, hosted by Johns Hopkins’ Bloomberg School of Public Health, have access to webinars, resources, a retreat to discuss their research project with expert faculty, and are matched with mixed methods consultants for ongoing support.

Michigan Mixed Methods : University of Michigan Mixed Methods program offers a variety of resources, including short web videos and recommended reading.

To use a mixed methods approach, you may want to first brush up on your qualitative skills. Below are a few helpful resources specific to qualitative research:

• Qualitative Research Guidelines Project : A comprehensive guide for designing, writing, reviewing and reporting qualitative research.
• Fundamentals of Qualitative Research Methods – What is Qualitative Research : A six-module web video series covering essential topics in qualitative research, including what is qualitative research and how to use the most common methods, in-depth interviews, and focus groups.

View PDF of the above information.

• Privacy Policy

Home » Mixed Methods Research – Types & Analysis

Mixed Methods Research – Types & Analysis

Table of Contents

Mixed Methods Research

Mixed methods research is an approach to research that combines both quantitative and qualitative research methods in a single study or research project. It is a methodological approach that involves collecting and analyzing both numerical (quantitative) and narrative (qualitative) data to gain a more comprehensive understanding of a research problem.

Types of Mixed Research

There are different types of mixed methods research designs that researchers can use, depending on the research question, the available data, and the resources available. Here are some common types:

Convergent Parallel Design

This design involves collecting both qualitative and quantitative data simultaneously, analyzing them separately, and then merging the findings to draw conclusions. The qualitative and quantitative data are given equal weight, and the findings are integrated during the interpretation phase.

Sequential Explanatory Design

In this design, the researcher collects and analyzes quantitative data first, and then uses qualitative data to explain or elaborate on the quantitative findings. The researcher may use the qualitative data to clarify unexpected or contradictory results from the quantitative analysis.

Sequential Exploratory Design

This design involves collecting qualitative data first, analyzing it, and then collecting and analyzing quantitative data to confirm or refute the qualitative findings. Qualitative data are used to generate hypotheses that are tested using quantitative data.

Concurrent Triangulation Design

This design involves collecting both qualitative and quantitative data concurrently and then comparing the results to find areas of agreement and disagreement. The findings are integrated during the interpretation phase to provide a more comprehensive understanding of the research question.

Concurrent Nested Design

This design involves collecting one type of data as the primary method and then using the other type of data to elaborate or clarify the primary data. For example, a researcher may use quantitative data as the primary method and qualitative data as a secondary method to provide more context and detail.

Transformative Design

This design involves using mixed methods research to not only understand the research question but also to bring about social change or transformation. The research is conducted in collaboration with stakeholders and aims to generate knowledge that can be used to improve policies, programs, and practices.

Concurrent Embedded Design

Concurrent embedded design is a type of mixed methods research design in which one type of data is embedded within another type of data. This design involves collecting both quantitative and qualitative data simultaneously, with one type of data being the primary method and the other type of data being the secondary method. The secondary method is embedded within the primary method, meaning that it is used to provide additional information or to clarify the primary data.

Data Collection Methods

Here are some common data collection methods used in mixed methods research:

Surveys are a common quantitative data collection method used in mixed methods research. Surveys involve collecting standardized responses to a set of questions from a sample of participants. Surveys can be conducted online, in person, or over the phone.

Interviews are a qualitative data collection method that involves asking open-ended questions to gather in-depth information about a participant’s experiences, perspectives, and opinions. Interviews can be conducted in person, over the phone, or online.

Focus groups

Focus groups are a qualitative data collection method that involves bringing together a small group of participants to discuss a topic or research question. The group is facilitated by a researcher, and the discussion is recorded and analyzed for themes and patterns.

Observations

Observations are a qualitative data collection method that involves systematically watching and recording behavior in a natural setting. Observations can be structured or unstructured and can be used to gather information about behavior, interactions, and context.

Document Analysis

Document analysis is a qualitative data collection method that involves analyzing existing documents, such as reports, policy documents, or media articles. Document analysis can be used to gather information about trends, policy changes, or public attitudes.

Experimentation

Experimentation is a quantitative data collection method that involves manipulating one or more variables and measuring their effects on an outcome. Experiments can be conducted in a laboratory or in a natural setting.

Data Analysis Methods

Mixed methods research involves using both quantitative and qualitative data analysis methods to analyze data collected through different methods. Here are some common data analysis methods used in mixed methods research:

Quantitative Data Analysis

Quantitative data collected through surveys or experiments can be analyzed using statistical methods. Statistical analysis can be used to identify relationships between variables, test hypotheses, and make predictions. Common statistical methods used in quantitative data analysis include regression analysis, t-tests, ANOVA, and correlation analysis.

Qualitative Data Analysis

Qualitative data collected through interviews, focus groups, or observations can be analyzed using a variety of qualitative data analysis methods. These methods include content analysis, thematic analysis, narrative analysis, and grounded theory. Qualitative data analysis involves identifying themes and patterns in the data, interpreting the meaning of the data, and drawing conclusions based on the findings.

Integration of Data

The integration of quantitative and qualitative data involves combining the results from both types of data analysis to gain a more comprehensive understanding of the research question. Integration can involve either a concurrent or sequential approach. Concurrent integration involves analyzing quantitative and qualitative data at the same time, while sequential integration involves analyzing one type of data first and then using the results to inform the analysis of the other type of data.

Triangulation

Triangulation involves using multiple sources or types of data to validate or corroborate findings. This can involve using both quantitative and qualitative data or multiple qualitative methods. Triangulation can enhance the credibility and validity of the research findings.

Mixed Methods Meta-analysis

Mixed methods meta-analysis involves the systematic review and synthesis of findings from multiple studies that use mixed methods designs. This involves combining quantitative and qualitative data from multiple studies to gain a broader understanding of a research question.

How to conduct Mixed Methods Research

Here are some general steps for conducting mixed methods research:

• Identify the research problem: The first step is to clearly define the research problem and determine if mixed methods research is appropriate for addressing it.
• Design the study: The research design should include both qualitative and quantitative data collection and analysis methods. The specific design will depend on the research question and the purpose of the study.
• Collect data : Data collection involves collecting both qualitative and quantitative data through various methods such as surveys, interviews, observations, and document analysis.
• Analyze data: Both qualitative and quantitative data need to be analyzed separately and then integrated. Analysis methods may include coding, statistical analysis, and thematic analysis.
• Interpret results: The results of the analysis should be interpreted, taking into account both the quantitative and qualitative findings. This involves integrating the results and identifying any patterns, themes, or discrepancies.
• Draw conclusions : Based on the interpretation of the results, conclusions should be drawn that address the research question and objectives.
• Report findings: Finally, the findings should be reported in a clear and concise manner, using both quantitative and qualitative data to support the conclusions.

Applications of Mixed Methods Research

Mixed methods research can be applied to a wide range of research fields and topics, including:

• Education : Mixed methods research can be used to evaluate educational programs, assess the effectiveness of teaching methods, and investigate student learning experiences.
• Health and social sciences: Mixed methods research can be used to study health interventions, understand the experiences of patients and their families, and assess the effectiveness of social programs.
• Business and management: Mixed methods research can be used to investigate customer satisfaction, assess the impact of marketing campaigns, and analyze the effectiveness of management strategies.
• Psychology : Mixed methods research can be used to explore the experiences and perspectives of individuals with mental health issues, investigate the impact of psychological interventions, and assess the effectiveness of therapy.
• Sociology : Mixed methods research can be used to study social phenomena, investigate the experiences and perspectives of marginalized groups, and assess the impact of social policies.
• Environmental studies: Mixed methods research can be used to assess the impact of environmental policies, investigate public perceptions of environmental issues, and analyze the effectiveness of conservation strategies.

Examples of Mixed Methods Research

Here are some examples of Mixed-Methods research:

• Evaluating a school-based mental health program: A researcher might use a concurrent embedded design to evaluate a school-based mental health program. The researcher might collect quantitative data through surveys and qualitative data through interviews with students and teachers. The quantitative data might be analyzed using statistical methods, while the qualitative data might be analyzed using thematic analysis. The results of the two types of data analysis could be integrated to provide a comprehensive evaluation of the program’s effectiveness.
• Understanding patient experiences of chronic illness: A researcher might use a sequential explanatory design to investigate patient experiences of chronic illness. The researcher might collect quantitative data through surveys and then use the results of the survey to inform the selection of participants for qualitative interviews. The qualitative data might be analyzed using content analysis to identify common themes in the patients’ experiences.
• Assessing the impact of a new public transportation system : A researcher might use a concurrent triangulation design to assess the impact of a new public transportation system. The researcher might collect quantitative data through surveys and qualitative data through focus groups with community members. The results of the two types of data analysis could be triangulated to provide a more comprehensive understanding of the impact of the new transportation system on the community.
• Exploring teacher perceptions of technology integration in the classroom: A researcher might use a sequential exploratory design to investigate teacher perceptions of technology integration in the classroom. The researcher might collect qualitative data through in-depth interviews with teachers and then use the results of the interviews to develop a survey. The quantitative data might be analyzed using descriptive statistics to identify trends in teacher perceptions.

When to use Mixed Methods Research

Mixed methods research is typically used when a research question cannot be fully answered by using only quantitative or qualitative methods. Here are some common situations where mixed methods research is appropriate:

• When the research question requires a more comprehensive understanding than can be achieved by using only quantitative or qualitative methods.
• When the research question requires both an exploration of individuals’ experiences, perspectives, and attitudes, as well as the measurement of objective outcomes and variables.
• When the research question requires the examination of a phenomenon in its natural setting and context, which can be achieved by collecting rich qualitative data, as well as the generalization of findings to a larger population, which can be achieved through the use of quantitative methods.
• When the research question requires the integration of different types of data or perspectives, such as combining data collected from participants with data collected from stakeholders or experts.
• When the research question requires the validation of findings obtained through one method by using another method.
• When the research question involves studying a complex phenomenon that cannot be understood by using only one method, such as studying the impact of a policy on a community’s well-being.
• When the research question involves studying a topic that has not been well-researched, and using mixed methods can help provide a more comprehensive understanding of the topic.

Purpose of Mixed Methods Research

The purpose of mixed methods research is to provide a more comprehensive understanding of a research problem than can be obtained through either quantitative or qualitative methods alone.

Mixed methods research is particularly useful when the research problem is complex and requires a deep understanding of the context and subjective experiences of participants, as well as the ability to generalize findings to a larger population. By combining both qualitative and quantitative methods, researchers can obtain a more complete picture of the research problem and its underlying mechanisms, as well as test hypotheses and identify patterns that may not be apparent with only one method.

Overall, mixed methods research aims to provide a more holistic and nuanced understanding of the research problem, allowing researchers to draw more valid and reliable conclusions, make more informed decisions, and develop more effective interventions and policies.

Advantages of Mixed Methods Research

Mixed methods research offers several advantages over using only qualitative or quantitative research methods. Here are some of the main advantages of mixed methods research:

• Comprehensive understanding: Mixed methods research provides a more comprehensive understanding of the research problem by combining both qualitative and quantitative data, which allows for a more nuanced interpretation of the data.
• Triangulation : Mixed methods research allows for triangulation, which is the use of multiple sources of data to verify findings. This improves the validity and reliability of the research.
• Addressing limitations: Mixed methods research can address the limitations of qualitative or quantitative research by compensating for the weaknesses of each method.
• Flexibility : Mixed methods research is flexible, allowing researchers to adapt the research design and methods as needed to best address the research question.
• Validity : Mixed methods research can increase the validity of the research by using multiple methods to measure the same concept.
• Generalizability : Mixed methods research can improve the generalizability of the findings by using quantitative data to test the applicability of qualitative findings to a larger population.
• Practical applications: Mixed methods research is useful for developing practical applications, such as interventions or policies, as it provides a more comprehensive understanding of the research problem.

Limitations of Mixed Methods Research

Here are some of the main limitations of mixed methods research:

• Time-consuming: Mixed methods research can be time-consuming and may require more resources than using only one research method.
• Complex data analysis: Integrating qualitative and quantitative data can be challenging and requires specialized skills for data analysis.
• Sampling bias: Mixed methods research can be subject to sampling bias, particularly if the sampling strategies for the qualitative and quantitative components are not aligned.
• Validity and reliability: Mixed methods research requires careful attention to the validity and reliability of both the qualitative and quantitative data, as well as the integration of the two data types.
• Difficulty in balancing the two methods: Mixed methods research can be difficult to balance the qualitative and quantitative methods effectively, particularly if one method dominates the other.
• Theoretical and philosophical issues: Mixed methods research raises theoretical and philosophical questions about the compatibility of qualitative and quantitative research methods and the underlying assumptions about the nature of reality and knowledge.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Questionnaire – Definition, Types, and Examples

Case Study – Methods, Examples and Guide

Observational Research – Methods and Guide

Quantitative Research – Methods, Types and...

Qualitative Research Methods

Explanatory Research – Types, Methods, Guide

• Open access
• Published: 15 December 2015

Qualitative and mixed methods in systematic reviews

• David Gough 1  

Systematic Reviews volume  4 , Article number:  181 ( 2015 ) Cite this article

22k Accesses

46 Citations

23 Altmetric

Metrics details

Expanding the range of methods of systematic review

The logic of systematic reviews is very simple. We use transparent rigorous approaches to undertake primary research, and so we should do the same in bringing together studies to describe what has been studied (a research map) or to integrate the findings of the different studies to answer a research question (a research synthesis). We should not really need to use the term ‘systematic’ as it should be assumed that researchers are using and reporting systematic methods in all of their research, whether primary or secondary. Despite the universality of this logic, systematic reviews (maps and syntheses) are much better known in health research and for answering questions of the effectiveness of interventions (what works). Systematic reviews addressing other sorts of questions have been around for many years, as in, for example, meta ethnography [ 1 ] and other forms of conceptual synthesis [ 2 ], but only recently has there been a major increase in the use of systematic review approaches to answer other sorts of research questions.

There are probably several reasons for this broadening of approach. One may be that the increased awareness of systematic reviews has made people consider the possibilities for all areas of research. A second related factor may be that more training and funding resources have become available and increased the capacity to undertake such varied review work.

A third reason could be that some of the initial anxieties about systematic reviews have subsided. Initially, there were concerns that their use was being promoted by a new managerialism where reviews, particularly effectiveness reviews, were being used to promote particular ideological and theoretical assumptions and to indirectly control research agendas. However, others like me believe that explicit methods should be used to enable transparency of perspectives driving research and to open up access to and participation in research agendas and priority setting [ 3 ] as illustrated, for example, by the James Lind Alliance (see http://www.jla.nihr.ac.uk/ ).

A fourth possible reason for the development of new approaches is that effectiveness reviews have themselves broadened. Some ‘what works’ reviews can be open to criticism for only testing a ‘black box’ hypothesis of what works with little theorizing or any logic model about why any such hypothesis should be true and the mechanisms involved in such processes. There is now more concern to develop theory and to test how variables combine and interact. In primary research, qualitative strategies are advised prior to undertaking experimental trials [ 4 , 5 ] and similar approaches are being advocated to address complexity in reviews [ 6 ], in order to ask questions and use methods that address theories and processes that enable an understanding of both impact and context.

This Special Issue of Systematic Reviews Journal is providing a focus for these new methods of review whether these use qualitative review methods on their own or mixed together with more quantitative approaches. We are linking together with the sister journal Trials for this Special Issue as there is a similar interest in what qualitative approaches can and should contribute to primary research using experimentally controlled trials (see Trials Special Issue editorial by Claire Snowdon).

Dimensions of difference in reviews

Developing the range of methods to address different questions for review creates a challenge in describing and understanding such methods. There are many names and brands for the new methods which may or may not withstand the changes of historical time, but another way to comprehend the changes and new developments is to consider the dimensions on which the approaches to review differ [ 7 , 8 ].

One important distinction is the research question being asked and the associated paradigm underlying the method used to address this question. Research assumes a particular theoretical position and then gathers data within this conceptual lens. In some cases, this is a very specific hypothesis that is then tested empirically, and sometimes, the research is more exploratory and iterative with concepts being emergent and constructed during the research process. This distinction is often labelled as quantitative or positivist versus qualitative or constructionist. However, this can be confusing as much research taking a ‘quantitative’ perspective does not have the necessary numeric data to analyse. Even if it does have such data, this might be explored for emergent properties. Similarly, research taking a ‘qualitative’ perspective may include implicit quantitative themes in terms of the extent of different qualitative findings reported by a study.

Sandelowski and colleagues’ solution is to consider the analytic activity and whether this aggregates (adds up) or configures (arranges) the data [ 9 ]. In a randomized controlled trial and an effectiveness review of such studies, the main analysis is the aggregation of data using a priori non-emergent strategies with little iteration. However, there may also be post hoc analysis that is more exploratory in arranging (configuring) data to identify patterns as in, for example, meta regression or qualitative comparative analysis aiming to identify the active ingredients of effective interventions [ 10 ]. Similarly, qualitative primary research or reviews of such research are predominantly exploring emergent patterns and developing concepts iteratively, yet there may be some aggregation of data to make statements of generalizations of extent.

Even where the analysis is predominantly configuration, there can be a wide variation in the dimensions of difference of iteration of theories and concepts. In thematic synthesis [ 11 ], there may be few presumptions about the concepts that will be configured. In meta ethnography which can be richer in theory, there may be theoretical assumptions underlying the review question framing the analysis. In framework synthesis, there is an explicit conceptual framework that is iteratively developed and changed through the review process [ 12 , 13 ].

In addition to the variation in question, degree of configuration, complexity of theory, and iteration are many other dimensions of difference between reviews. Some of these differences follow on from the research questions being asked and the research paradigm being used such as in the approach to searching (exhaustive or based on exploration or saturation) and the appraisal of the quality and relevance of included studies (based more on risk of bias or more on meaning). Others include the extent that reviews have a broad question, depth of analysis, and the extent of resultant ‘work done’ in terms of progressing a field of inquiry [ 7 , 8 ].

Mixed methods reviews

As one reason for the growth in qualitative synthesis is what they can add to quantitative reviews, it is not surprising that there is also growing interest in mixed methods reviews. This reflects similar developments in primary research in mixing methods to examine the relationship between theory and empirical data which is of course the cornerstone of much research. But, both primary and secondary mixed methods research also face similar challenges in examining complex questions at different levels of analysis and of combining research findings investigated in different ways and may be based on very different epistemological assumptions [ 14 , 15 ].

Some mixed methods approaches are convergent in that they integrate different data and methods of analysis together at the same time [ 16 , 17 ]. Convergent systematic reviews could be described as having broad inclusion criteria (or two or more different sets of criteria) for methods of primary studies and have special methods for the synthesis of the resultant variation in data. Other reviews (and also primary mixed methods studies) are sequences of sub-reviews in that one sub-study using one research paradigm is followed by another sub-study with a different research paradigm. In other words, a qualitative synthesis might be used to explore the findings of a prior quantitative synthesis or vice versa [ 16 , 17 ].

An example of a predominantly aggregative sub-review followed by a configuring sub-review is the EPPI-Centre’s mixed methods review of barriers to healthy eating [ 18 ]. A sub-review on the effectiveness of public health interventions showed a modest effect size. A configuring review of studies of children and young people’s understanding and views about eating provided evidence that the public health interventions did not take good account of such user views research, and that the interventions most closely aligned to the user views were the most effective. The already mentioned qualitative comparative analysis to identify the active ingredients within interventions leading to impact could also be considered a qualitative configuring investigation of an existing quantitative aggregative review [ 10 ].

An example of a predominantly configurative review followed by an aggregative review is realist synthesis. Realist reviews examine the evidence in support of mid-range theories [ 19 ] with a first stage of a configuring review of what is proposed by the theory or proposal (what would need to be in place and what casual pathways would have to be effective for the outcomes proposed by the theory to be supported?) and a second stage searching for empirical evidence to test for those necessary conditions and effectiveness of the pathways. The empirical testing does not however use a standard ‘what works’ a priori methods approach but rather a more iterative seeking out of evidence that confirms or undermines the theory being evaluated [ 20 ].

Although sequential mixed methods approaches are considered to be sub-parts of one larger study, they could be separate studies as part of a long-term strategic approach to studying an issue. We tend to see both primary studies and reviews as one-off events, yet reviews are a way of examining what we know and what more we want to know as a strategic approach to studying an issue over time. If we are in favour of mixing paradigms of research to enable multiple levels and perspectives and mixing of theory development and empirical evaluation, then we are really seeking mixed methods research strategies rather than simply mixed methods studies and reviews.

Noblit G. Hare RD: meta-ethnography: synthesizing qualitative studies. Newbury Park NY: Sage Publications; 1988.

Google Scholar  

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Gough D, Elbourne D. Systematic research synthesis to inform policy, practice and democratic debate. Soc Pol Soc. 2002;2002:1.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance 2015. BMJ. 2015;350:h1258

Candy B, Jone L, King M, Oliver S. Using qualitative evidence to help understand complex palliative care interventions: a novel evidence synthesis approach. BMJ Support Palliat Care. 2014;4:Supp A41–A42.

Article   Google Scholar  

Noyes J, Gough D, Lewin S, Mayhew A, Michie S, Pantoja T, et al. A research and development agenda for systematic reviews that ask complex questions about complex interventions. J Clin Epidemiol. 2013;66:11.

Gough D, Oliver S, Thomas J. Introduction to systematic reviews. London: Sage; 2012.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:4.

Thomas J, O’Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3:67.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Exp. 2008;11:72–84.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Qual Saf. 2015. 2014-003642.

Brannen J. Mixed methods research: a discussion paper. NCRM Methods Review Papers, 2006. NCRM/005.

Creswell J. Mapping the developing landscape of mixed methods research. In: Teddlie C, Tashakkori A, editors. SAGE handbook of mixed methods in social & behavioral research. New York: Sage; 2011.

Morse JM. Principles of mixed method and multi-method research design. In: Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioural research. London: Sage; 2003.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.

Harden A, Thomas J. Mixed methods and systematic reviews: examples and emerging issues. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in the social and behavioral sciences. 2nd ed. London: Sage; 2010. p. 749–74.

Chapter   Google Scholar  

Pawson R. Evidenced-based policy: a realist perspective. London: Sage; 2006.

Book   Google Scholar  

Gough D. Meta-narrative and realist reviews: guidance, rules, publication standards and quality appraisal. BMC Med. 2013;11:22.

Download references

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, University College London, London, WC1H 0NR, UK

David Gough

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The author is a writer and researcher in this area. The author declares that he has no other competing interests.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Gough, D. Qualitative and mixed methods in systematic reviews. Syst Rev 4 , 181 (2015). https://doi.org/10.1186/s13643-015-0151-y

Download citation

Received : 13 October 2015

Accepted : 29 October 2015

Published : 15 December 2015

DOI : https://doi.org/10.1186/s13643-015-0151-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

Quantitative, Qualitative, and Mixed-Methods Research: Home

Quantitative, qualitative, and mixed-methods research.

Depending on the philosophy of the researcher, the nature of the data, and how it is collected, behavioral science can be classified into qualitative, quantitative, or mixed methods research. Below are descriptions of each method. 

Quantitative Research

Collects numerical data, such as frequencies or scores to focus on cause-and-effect relationships among variables

Variables and research methodologies are defined in advance by theories and hypotheses derived from other theories. These remain unchanged throughout the research process. 

The researcher tries to achieve objectivity by distancing himself or herself from the research, not allowing himself or herself to be emotionally involved.

The researcher mostly studies research in artificial or less than its natural setting, and manipulates behavior as opposed to studying the behavior in its natural context.

The researcher tries to maintain internal validity and focuses on average behavior or thoughts of people in a population

Qualitative Research

Where researchers collect non-numerical information, such as descriptions of behavioral phenomena, how people experience or interpret events, and/or answers to participants' open-ended responses.

The researcher's variables andmethods used come from the researcher's experiences and can be modified as the research progresses.

The researcher is involved and his or her experiences are valuable as well as the participants' experiences. 

The researcher studies behavior as it naturally happens in the natural context.

The researcher tries to maximize ecological validity.

The researcher focuses on similarities and differences in experiences and how people interpret them. 

Mixed-Methods Research

Involves both quantitative and qualitative components. 

The researcher specifies in advance the types of information necessary to accomplish the study's goals.

The researcher needs to carefully consider the order in which the data types will be collected and the selection criteria for participants in the various parts of the study (e.g., which people will participate in the qualitative assessment if a sub-selection of participants will be involved). 

Involves development (where the researcher uses one method to inform data collection or analysis with another method) initiation (where unexpected results change protocol in the other method), corroboration (where consistency is evaluated and compared between methods), and elaboration (where one method is used to expand on the results of the other method).

Whitley, B. E. & Kite, M. E. (2013).  Principles of research in behavioral science  (3rd ed.). Routledge. 

• Last Updated: Sep 2, 2020 12:29 PM
• URL: https://library.divinemercy.edu/research-types

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Mixed Methods Research | Definition, Guide, & Examples

Mixed Methods Research | Definition, Guide, & Examples

Published on 4 April 2022 by Tegan George . Revised on 25 October 2022.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, benefits of mixed methods research, disadvantages of mixed methods research, frequently asked questions about mixed methods research.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalisability : Qualitative research usually has a smaller sample size , and thus is not generalisable . In mixed methods research, this comparative weakness is mitigated by the comparative strength of ‘large N’, externally valid quantitative research.
• Contextualisation: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help ‘put meat on the bones’ of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions. Mixed methods can be very challenging to put into practice, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism, run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyse them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyse cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyse accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyse both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualise your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses. Then you can use the quantitative data to test or confirm your qualitative findings.

‘Best of both worlds’ analysis

Combining the two types of data means you benefit from both the detailed, contextualised insights of qualitative data and the generalisable, externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalisable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labour-intensive. Collecting, analysing, and synthesising two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organisations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analysed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analysed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualise your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analysed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

George, T. (2022, October 25). Mixed Methods Research | Definition, Guide, & Examples. Scribbr. Retrieved 22 April 2024, from https://www.scribbr.co.uk/research-methods/mixed-methods/

Is this article helpful?

Tegan George

• Open access
• Published: 26 April 2024

Overcoming barriers to equality, diversity, inclusivity, and sense of belonging in healthcare education: the Underrepresented Groups’ Experiences in Osteopathic Training (UrGEnT) mixed methods study

• Jerry Draper-Rodi   ORCID: orcid.org/0000-0002-1900-6141 1 , 2 ,
• Hilary Abbey 1 ,
• John Hammond 3 ,
• Oliver T. Thomson 1 ,
• Kevin Brownhill 1 ,
• Andrew MacMillan 1 , 4 ,
• Yinka Fabusuyi 1 &
• Steven Vogel 1  

BMC Medical Education volume  24 , Article number:  468 ( 2024 ) Cite this article

Metrics details

Individuals from minority groups have historically faced social injustices. Those from underrepresented groups have been less likely to access both healthcare services and higher education. Little is known about the experiences of underrepresented students during their undergraduate studies in osteopathy in the UK. The aim of this project was to explore awareness of cultural diversity and beliefs about patients from underrepresented groups in current osteopathic educational environments and evaluate students’ preparedness to manage patients from diverse groups. The project also aimed to investigate the educational experiences of students from underrepresented backgrounds during their training and their opinions on changes that could support better levels of recruitment and achievement. The findings were discussed with stakeholders in interactive workshops with the aim to develop recommendations for action and change.

A transformative action research paradigm informed this mixed methods project. It included: 1/ a survey of students from all seven osteopathic educational providers in the UK using the Multidimensional Cultural Humility Scale (MCHS); 2/ a series of focus groups with students from underrepresented groups (women, students with disabilities, students from minority ethnic backgrounds, and students identifying as LGBTQIA+); and 3/ a workshop forum to discuss findings.

A total of 202 participants completed the MCHS and demographic questionnaire and seven focus groups were conducted. A model was developed to describe participants’ training experiences comprising two main themes: institutional contextual obstacles (with four sub-themes) and underrepresented students’ conceptual understanding of Equity, Diversity and Inclusion (EDI). Recommendations for change identified in the workshops were based on three topics: institutions, staff, and students.

Our findings confirm conclusions from other institutions that staff education is urgently needed to create and maintain equitable, inclusive environments in osteopathic educational institutions in the UK to support all students, particularly those from underrepresented groups. Institutional EDI processes and policies also need to be clarified or modified to ensure their usefulness, accessibility, and implementation.

Peer Review reports

Social injustices affecting people from minority groups have been highlighted in recent worldwide initiatives such as the ‘Black Lives Matter’ [ 1 ] and ‘Me Too’ [ 2 ] movements and investigations have identified institutional racism, sexism and homophobia in the police, other public services, and business organisations [ 3 , 4 , 5 , 6 ]. Limited demographic diversity and evidence of discrimination against minority groups have been reported in higher education in the United Kingdom (UK) [ 7 ] and in healthcare services including medicine, psychiatry, and physiotherapy [ 8 , 9 , 10 ]. Data from higher education institutions suggest there is an urgent need to improve recruitment, educational experiences, and attainment for students from minority groups [ 11 , 12 , 13 ].

The terms ‘minority’ or ‘under-represented’ are often used interchangeably to describe groups of people identified by specific demographic or cultural characteristics. In this paper, the term ‘under-represented’ is used to emphasise that experiences of inequity are typically created and maintained by social constructs such as ‘othering’: the process of identifying people as different from oneself or the mainstream culture, often associated with negative beliefs and expectations [ 14 ]. Social constructs can provide both unearned advantage (‘privilege’) and disadvantage (‘oppression’) [ 15 ]. Characteristics used to identify others can include skin colour, ethnicity, religion, gender identity, sexual identity, ability, size, socioeconomic status, history of trauma, addiction, and family environment [ 15 ].

People from under-represented groups (UrGs) have historically been less likely to access higher education [ 16 ], although the number of BAME, LGBTQIA + and disabled students is gradually increasing in England [ 17 , 18 ]. Enrolled students from these groups are reported to experience more negative experiences during training and more limited later career opportunities afterwards [ 19 , 20 ]. The General Medical Council (GMC) recently set new targets to improve access and outcomes for students from UrGs [ 21 ] as lack of diversity and limited cultural awareness among practitioners from different healthcare professions also impacts the quality and outcomes of healthcare for patients from UrGs [ 12 , 22 , 23 ]. The Council of Deans recently published a report on how to build an inclusive environment which highlights issues that affect students from minority ethnic groups in Allied Health Professions [ 24 ].

Patients from UrG experience substantial health disparities in the UK and across the globe due to structural and interpersonal discrimination [ 25 , 26 ]. Developing cultural humility in clinicians is seen as key to bridging the gap of interpersonal discrimination. Cultural competence was once considered as an adequate way to provide an inclusive environment. It is defined as “a set of congruent behaviours, attitudes and policies that come together in a system, agency or among professionals that enable that system, agency or professions to work effectively in cross-cultural situations” ([ 27 ] p. iv). The concept shifted to cultural humility, defined as “the ability to maintain an interpersonal stance that is other-oriented (or open to the other) in relation to aspects of cultural identity that are most important to the client” ([ 28 ] p. 354).

Osteopathy is a form of manual therapy which is now recognised as one of 14 Allied Healthcare Professions in England [ 29 ]. In the UK, there are currently seven osteopathic education providers (OEPs) and approximately 5,300 qualified osteopaths. Training is typically over four or five years in the form of Bachelor’s or Integrated Masters awards and practitioners then register with the statutory regulator, the General Osteopathic Council (GOsC), and are required to comply with professional standards of practice [ 30 ].

There is little known about discrimination, bullying and harassment in osteopathy education as highlighted in a recent systematic review [ 31 ]. Therefore, the current research project aimed to assess osteopathic students’ awareness of cultural diversity and beliefs about patients from UrGs and their preparedness in managing them; to explore the educational experiences of students with UrG backgrounds during training and their opinions on changes to support better levels of recruitment and achievement. Finally, the research was disseminated to stakeholders in workshops with the overall aim of developing recommendations for action and creating change.

To meet the multiple aims, a mixed methods approach was implemented and included the following stages; a survey of students attending all seven OEPs in the UK; focus groups with UrG students; and a workshop forum to explore the findings with diverse stakeholders. This design was based on a transformative action research paradigm with students participating as collaborators (Mertens 2007; 2010), informed by previous research into EDI, cultural competence and cultural humility in healthcare education, outlined below. The research complies with the Good Reporting of A Mixed Methods Study (GRAMMS) guidance [ 32 ] (see supplementary material 1 – GRAMMS reporting).

Figure  1 below details the mixed method stages with the quantitative data collection (top half of figure), qualitative data collection (bottom half), and mixed methods stages (middle). The stages are represented chronologically, starting on the left.

Study design

Methodology

This research project sits within a transformative paradigm that places central importance on studying the lives and experiences of marginalised groups and is appropriate for addressing inequality and injustice in society [ 33 ]. An explanatory sequential mixed methods design (survey followed by focus groups) was implemented to gain insight [ 34 ] and community members were involved in initial discussions about operationalising the research focus. Transformative research has power issues and inequalities at its core and a political agenda that aims to change the experiences of the participants and institutions involved [ 35 ]. The study was approved by the University College of Osteopathy Research Ethics Committee.

Community engagement

Two community engagement meetings with students from underrepresented groups were established prior to the project to ensure it was designed ‘with’ students rather than ‘to’, ‘about’ or ‘for’ them. Based on principles by [ 36 ], these community engagement meetings co-created the study design, modified the research questionnaire and recruitment approaches.

Quantitative stage

A survey of all students currently enrolled on an osteopathic course in the UK was chosen to explore the research objectives. All students enrolled at the seven OEPs in the UK (excluding postgraduate and CPD courses) were eligible to take part in the anonymous online survey on Qualtrics©. Invitations, study information and accessible links were disseminated via OEP contacts who sent it to their student body between 7th and 31st March 2022. Two reminders were sent.

Survey instrument

The Multidimensional Cultural Humility Scale (MCHS) was selected for this project as there is good evidence of convergent and discriminant validity and internal reliability [ 37 ]. The MCHS has five dimensions, contains 15 items with a 6-point Likert scale from ‘strongly disagree’ to ‘strongly agree’ where higher scores represent greater levels of cultural humility. The MCHS was used to understand to understand awareness of cultural sensitivity in the environment in which UrG students were learning. This project was not about clinical services. Modifications to the MCHS were necessary to contextualise it for osteopathy students, so a factor analysis was conducted to assess the validity of the adapted version. Following the community engagement meetings, a 7th category was added: ‘This has never crossed my mind’ to assess whether students were comfortable, confident or aware of particular issues (see supplementary material 2 for the adapted versions used in this study).

Questions related to demographics and personal characteristics (clinical or pre-clinical student, age, birth sex, gender, ethnicity, health and disability status, sexual orientation, and religion), and to their experience of education were included at the end of the MCHS and were analysed separately to the MCHS questionnaire.

Qualitative stage

Focus groups were selected for this phase and represented four UrG: ethnic minority, disability, LGBTQIA + or women. Whilst women are not numerically under-represented in UK osteopathic undergraduate training, socially they are more oppressed than men, including in manual therapy training [ 31 ]. The choice of these four groups was discussed and agreed as important priorities in the community engagement meetings. For sensitive topics, homogeneous groups foster a sense of belonging and facilitate disclosure [ 38 ]. Focus groups usually comprise 6 to 8 people who meet once for approximately 90–120 min, and the usual number of groups is around 4 but depends on the complexity of the topic and heterogeneity of the samples [ 39 ].

Students from any UK OEP who identified as belonging to at least one UrG (ethnic minority, disability, LGBTQIA + or women) were eligible to participate with students from the same and/or other OEPs. Each OEP was responsible for forwarding invitations to participate to their students. For convenience, focus groups were conducted online as students from different OEPs were geographically dispersed [ 40 ]. The research team members acting as focus group facilitators identified with one or more minority groups, representing diversity and were therefore part of the data, as is good practice in transformative paradigmatic research [ 41 ]. All facilitators had previously used focus group methods, participated in training, or were used to managing student group discussions. Teams© created automatic initial draft transcriptions to aid later transcription if participants talked simultaneously [ 39 ]. Final transcripts only included pseudonyms, as is common in qualitative research [ 42 ]. Focus groups sessions ran for approximately 90 min. Students who had participated in one of the four initial groups were invited to join one final mixed group to discuss the previous findings, and students who participated in at least one group were invited to take part in the workshop forum.

Dissemination forum and discussion workshops

An interactive face-to-face workshop-based forum was held on 06/04/2023 to disseminate the survey and focus group results, discuss their implications, and develop recommendations for action. Key stakeholders invited to attend free of charge included UK OEPs, the General Osteopathic Council, the Institute of Osteopathy, the Osteopathic Foundation, and other healthcare profession organisations, NHS representatives, and Health Education England. Approximately 70 people attended the event. Three interactive workshops focusing on specific aspects of EDI (students, staff and institutional governance), with different methods to promote open discussion, explored responses about ways to develop a more supportive educational environment and inclusive curriculum.

Mixed methods analysis

To assess whether the 5-factor model of the MCHS remained valid following changes made to the scale, a confirmatory factor analysis was carried out using using R (version 4.3.2) [ 43 ]and the R lavaan package (version 0.6–16) [ 44 ]. Missing MCHS data was imputed using multivariate imputation by chained equations [ 45 ]. MCHS data was checked for normality using QQ plots and the Henze-Zirkler test.

A sum of all MCHS items (reverse coded as appropriate) was calculated as an overall measure of cultural humility. Linear regression was carried out to determine which demographic factors influenced this total score. Additionally, a Welch Two Sample t-test [ 46 ] was carried out to determine if MCHS total score differed between clinical and pre-clinical students. Chi-squared tests, with p-values estimated by Monte-Carlo simulation, were used to test for associations between students’ report of having been treated differently one the one hand, and demographic factors on the other. Descriptive statistics were used to report survey results.

Focus group data analysis was conducted within a reflexive thematic analysis framework [ 47 ], which aligns with a transformative paradigm (Creswell 2014). Data was co-created by participants and facilitators, and themes were co-created with analysts through their thoughtful engagement with the data [ 47 ]. After conducting one focus group with each UrG ( n  = 4), early analysis was conducted. Another 4 focus groups with different students were conducted to analyse how these participants’ experiences resonated with the initial findings. The last focus groups ran with students from mixed UrGs to discuss the findings, conduct a meta-synthesis, and prioritise what actions students thought OEPs should prioritise.

Three interactive workshops were run to explore the resonance and implications of the quantitative and qualitative findings to date. Each workshop focused on either student, staff or institutional EDI issues, although there was inevitably some overlap, and each workshop ran three times to enable participants to contribute fully. Small groups of mixed stakeholders worked took part in varied activities to discuss the study’s findings and their ideas were recorded on post-it notes, flipcharts or noted by facilitators during plenary discussions. After the workshop, written comments were collated by the facilitators (YF, HA, SV) and categorised into themes by members of the research team (JDR, HA), using frequency analysis (where data was available) to identify strong and recurring recommendations for change.

The data from the quantitative and qualitative phases were analysed separately, but then were considered together both in the forum workshops and within the research team. When considering the quantitative and qualitative datasets together, the research team operated within the methodological spirit of pragmatism, whereby both data sets were integrated in such a way that a useful insight to the research provided useful insights to participants’ experiences and generate knowledge with social utility [ 48 ]. In practice, this meant that survey results were presented to focus group participants to stimulate reflection and discussion and explore how the results compared with their personal experiences. Finally, the workshops provided an additional method to explore, situate and integrate the synthesised qualitative and quantitative data sets to support development of the final thematic model.

Quantitative results

Two hundred and two participants filled in the survey, of which 117 (58%) were complete. The response rate was 20% (Table  1 ). Responses per OEP ranged between 6 and 68 (Table  2 – OEP Responses).

Seventy percent of the respondents provided demographic information ( n  = 142). Participants were mostly white ( n  = 95), female ( n  = 74), without a disability ( n  = 106), heterosexual ( n  = 89), and identifying with no religion ( n  = 69) (see Table  3 – respondent demographics).

Most participants identified to some extent with an UrG ( n  = 62, 53%). Of all the students who responded (53% self-identifying as UrG to some extent, and 47% who did not identify as UrG), 67.8% ( n  = 80) reported that they had not been treated differently because of their cultural background or identity. Those who had been treated differently ( n  = 19; 16%) stated that it happened at least a few times per year ( n  = 15, 79%) (supplementary material 3 , table a– underrepresented groups treatment). Of the 28 who reported having been treated differently, 18 reported whether they had complained: 15 had not complained (6 open-ended responses: not significant enough ( n  = 2), unlikely to lead to change ( n  = 2), fear of being identified ( n  = 1), happened once and felt that mistakes happen ( n  = 1)). Six of the 15 who did not complain did not know how or to whom to complain.

Associations between demographic characteristics and UrG self-identification found that ethnicity (merging all categories excluding White), Disability and Sexual Orientation (merging all categories excluding heterosexual) were significantly associated with identifying as belonging to an UrG group (Supplementary material 3 , Table b - UrG identification vs. demographic group).

No significant associations were found between demographic characteristics and reports of being treated differently (Supplementary material 3 , table c - treated differently vs. demographic group).

Of the 19 participants who reported having been treated differently because of their culture or identity, 79% ( n  = 15) did not report it to their OEP, 15.8% ( n  = 3) did, and 5.2% ( n  = 1) did not answer.

It was not possible to confirm or deny the adequacy of the 5-factor model proposed by Gonzales et al. [ 37 ] (Supplementary material 4 ), so our analysis was based on their 5-factor model (see Table  4 – MCHS results). Regarding the MCHS total score, no differences were found between clinical and preclinical students (Welch’s t = -0.194, df = 79.3, p  = 0.847). A weak correlation between MCHS total score and importance to individual was found (Spearman’s rho(114) = 0.27, p  = 0.003), and a weak relationship between self-rating of skills and MCHS total score (rho(114) = 0.26, p  = 0.005). There was no apparent relationship between MCHS total score and participants’ perception of support in the clinical environment for exploring patients’ backgrounds and experiences (rho(106) = 0.097, p  = 0.3). No scores on these three questions differed significantly between clinical and preclinical students.

Qualitative results

Seven groups were conducted, each were facilitated by two members of the research team (from AMM, HA, JDR, SV, YF). Data from the first six focus groups were organised into two themes which provide descriptive insights of participants’ reflections on the quantitative findings and how these results related to and resonated with their own experiences. The two primary themes were named institutional contextual obstacles (with 4 sub-themes) and UrG students’ conceptual understanding of EDI (with 3 sub-themes). The themes and sub-themes were modelled and presented to the final focus group to facilitate reflective discussions, see Fig.  2 .

Model based on focus groups’ themes and sub-themes

Theme 1: Institutional contextual obstacles

The first sub-theme, Faculty’s lack of awareness & knowledge , was a commonly reported barrier.

I think there’s a lot of talk of self-reflection, at least at the OEP, and it doesn’t to me feel like all of our teachers practise that. I’ve had more problems with staff understanding than student understanding”. (Talking about their disability) There was no awareness, you know, of that within the class or from the tutors, in those circumstances (managing an LGBTQ + patient), what do we do, what language do we use, (…) when it was raised the tutor was sort of like, actually, I don’t have an answer, I’m not sure.

Racist, sexist and ableist comments made by staff negatively affected the way students interacted with patients in the OEPs clinics, and with other students, particularly in practical classes.

I was doing a neck and then teacher wants me to talk when I’m doing it and I say, because / when I’m doing it, I can’t talk and he made a comment, as a woman, you should talk and you should do it, you should multi-task and at that time I couldn’t say anything because I [was] already panicking and I’m doing this thing. I couldn’t say anything. [A male tutor] put [a female tutor acting as a model] side-lying and [he] was going to crack her back but then when he pulled her shirt up her scrubs pants were like mid-way / quite / kind of showing her underwear (…) When we told him that he should pull her scrubs up, he made the thing super uncomfortable. When they make an attack, as a joke, and people laugh, that’s positive behaviour, they’re going to make the joke again because it’s funny, so I don’t know if they can understand that it’s actually a knife that you’re throwing at someone and not just a joke.

The second sub-theme related to a lack of support from institutions for students from UrG, and a lack of clarity of processes available to them to complain about discriminatory behaviours against them.

When I was sort of going through the process of applying for the disabled students’ allowance, which I didn’t even know that I was / its existence to be honest, (…) I had to get the OEP to fill out a form and rubber stamp it and it seemed to get lost in this abyss of I don’t know where it went. (…) but there was a lot of chasing up to do [laughs] and even getting the form signed again, because I have to reapply every year, was a bit of a faff.

Participants who reported discrimination, were lacking certainty that reported these instances would lead to change.

Particularly when it’s a comment like that that’s made and it almost leaves you like gobsmacked and you’re like well what do I say to that, how do I go about telling someone about that?

The third sub-theme was Student attitudes e.g., peers making sexist comments and using negative language about UrGs.

People have said things, especially kind of bisexual tropes and things like that about you know being greedy and I know it’s / (…) people think oh that’s funny (…) it just makes you feel like you are going inward kind of thing. I was practicing thoracic HVT with (…) some first years [students] and I started doing thoracic HVT and one of the first years asked me to do it on him, so I was like, okay, umm, I explained to him you know everything, asked for his consent and stuff, but because he was like a funny guy, he was talking all the time, I was like, okay, can you just sit down for me to do the technique and I told him my nationality before that and then he goes like oh that’s how I know you are Brazilian, your attitude, you probably go on top. I’m just like what? You know / yes, I didn’t even know what to say at this time, because I was just / I just told him, look, I’m not doing the technique, I thought, goodbye.

Participants reported instances where students from privileged backgrounds remained silent when facing discriminatory comments from educational faculty; a factor that perpetuated a non-inclusive culture, as people who used discriminatory or ‘othering’ language were not challenged to reflect on their attitudes and behaviour. In contrast, participants from UrGs felt a sense of duty to raise concerns:

I don’t create problems and stuff, but if there is something if I see it not going right, I like to raise my voice as much as I can and I try to make changes.

The fourth sub-theme was Lack of representation in the student body, patient population and the curriculum.

Everything that we get taught is 99% on like a male sex anatomy. Like I remember when I was learning how to do all the like umm cardiac testing and respiratory we were taught by a male teacher on a male body and then when it came to like a female and like you have boobs and they’re like, oh, you can’t do this bit at the front, or you have to be more careful, but then there was no example of how. I think I felt surprised when coming into the / into osteopathy how less diverse (in student demographics? ) it is than my previous position. I feel quite diverse but people that we see in clinic are mainly Caucasian, so I also think there’s something about the outreach of osteopathy into different cultural communities, for example, most of my family, though we’ve all been brought up here, nobody would use an osteopath (…). When we learn about physiology and pathologies, I feel like there’s now a real effort to talk about say like black people, which is fantastic, but then you know what about Asian (…).

Theme 2: Underrepresented students’ understanding of EDI

The first sub-theme related to the definition of discrimination and echoed findings from community engagement discussions. Students distinguished between ‘othering’ and ‘intent’. Participants perceived discrimination only when actions had an intent to discriminate against individuals or communities, rather than actions that led to people or groups being treated differently regardless of intent. During the focus groups, participants reported equal treatment, but data analysis suggests instances of discrimination.

No, only in so much as, you know, the reasonable adjustments aspect, but then I’ll ask for that, but besides that, I haven’t / I haven’t had any different treatment. I’ve definitely been treated differently as a woman and / but I’ve witnessed the / in my class Asian women being treated differently, but the Asian men not so much so.

The second sub-theme related to the advocacy of UrG students as role models for their peers. Students used their own experience of belonging to an UrG as personal knowledge to help inform their peers about what it is like to be a person from wider UrG communities. This helped to fill gaps in the EDI training or make up for a lack of training received by educators. UrG students acted as advocates to prevent wrong messages, jokes being shared, e.g.,

I think it’s / not just from my disability, but yes, from / for all other students I think when they / things come up, sometimes quite surprising things actually, it’s usually / yes, pretty interesting and helpful for all of us. We use it [disability] sometimes in class as part of like chronic pain, as part of that kind of presentation and things like that because I have an understanding of it, whereas instead of just pulling stories out of thin air.

The third related sub-theme was that students from UrGs appeared to have a better understanding of EDI than their peers and faculty members. Students’ advocacy role included training and supporting their peers in how they should manage situations when facing patients with specific conditions, e.g. type 1 diabetes, and offered a useful insight which would be valued by patient.

I mean do they have to? Should they? I think you know, like I’ve said, the only reason I do [disclose] it is because you know I wouldn’t want to put anybody else in a tricky position if I was to, you know, have like a hypo in class or anything like that, which you know, I may do one day.

This created an environment where students from UrGs not only had to teach other students and faculty, but also had to learn on their own, as they were not able to gain knowledge from staff on topics related to UrG, and then had to teach what they learned to their peers and faculty members.

But we don’t get taught about how to deal with somebody that’s transgender or anything like that. It’s like well you’ll have to you know just find out about that yourself. I don’t have that much of an understanding of the difference that ethnicity has on sort of different diseases and different morphologies and things like that, so it’s something (…) I’d love to learn more about.

The final mixed focus group was used to explore whether the above findings represented the experiences of these participants, and to generate suggestions for OEP action to become more inclusive. Goals thought to be quickly achievable and likely to lead to sustained change was providing urgent training for staff, and then students, to improve awareness and knowledge, and to break the issue of the cycle of unaware students becoming unaware teachers.

Lack of diversity ‘breeds’ a lack of diversity. A lot of the main institutional barriers is the university’s lack of knowledge and the best way to deal with that is directly linked to how the under-represented students can like just you know break this barrier by teaching others and also by getting contact with the university.

Active bystander training was recommended to promote collective responsibility in challenging bias and negative views. Other suggestions included providing support for students from UrGs, countering negative views amongst peers and faculty, employing active strategies to promote patient diversity, being more equitable in services offered, and ensuring training was implemented. The final recommendation was to increase representativeness in the curriculum, as a way of training staff and students through regular exposure to up-to-date information regarding UrGs.

if the institutions were to be more aware [of EDI] and have [EDI training]…., I don’t know what training’s mandatory training’s given, but it would seem like potentially a lot of it [othering] could potentially be stopped. It just seems because you’ve got the lack of representation to faculty, race in faculty, they all sort of interlink with the other parts.

Participants felt that more and better training was needed for staff on EDI issues; a potential barrier to implementation was time, but short courses were expected to be effective.

every job I’ve ever done, either private sector, public sector, there is mandatory training and EDI’s, (…) human trafficking, (…) blackmail. (…) But I think we’re only talking like a half an hour.

Workshop results

Comments from nine workshop sessions (three each on student, staff and institutional EDI issues) were combined using frequency analysis to identify key themes and recommendations for change (Table  5 ). The strongest theme addressed stakeholders’ opinions about staff issues (96 comments in total), with recommendations about the need to improve staff attitudes [ 36 ], increase their awareness of students’ needs [ 15 ], and enhance communication skills [ 26 ]. The second main theme was student support [ 49 ], including the need to explore barriers to change [ 26 ] and improve access to support services [ 14 ]. Two other themes focused on the need to clarify and improve institutional EDI policies and processes [ 26 ] and ways to improve representation and diversity among student osteopaths, OEP staff and patients seeking osteopathic treatment [ 25 ] (also see Supplementary Material 5 ).

Overlapping themes were organised in Fig.  3 in relation to the groups involved in the recommended actions.

Workshop themes

The aims of this innovative mixed methods study were to survey student osteopaths’ levels of cultural humility to assess levels of awareness in the current educational environment and as a proxy for preparedness to work with patients from diverse backgrounds. It also explored the educational experiences of UrG students with the aim of improving equity, diversity and inclusivity (EDI) and sense of belonging in Osteopathic Educational Providers (OEPs). The survey response rate was 20%, but data was collected from 202 students from seven OEIs. 62 students identified with at least one UrG and 19 reported that they had been treated differently but 15 had not reported it.

Qualitative data from focus groups with students from the four selected UrGs suggested the main challenges faced were staff attitudes and lack of awareness; limited student support; and lack of representation in the curriculum and in institutional processes. These themes were explored and refined in interactive workshops, which generated recommendations to improve staff education, support students, and develop effective institutional policies. The implications of these findings are discussed below.

Educating staff

Cultural humility is a lifelong commitment to developing awareness to disparities experienced by people from diverse cultural groups, reflecting and being open to learning [ 49 , 50 , 51 , 52 ]. This model encourages practitioners to collaborate with patients, and educators to collaborate with students, to find solutions to discrimination and inequality based on their lived experiences and priorities [ 53 ]. Qualitative findings from the focus groups and workshops in this study indicated that experiences of ‘othering’ and discrimination were often associated with lack of cultural humility, self-awareness, ignorance, or overtly negative attitudes, mainly among staff. (Focus group theme 1: “ I don’t know if they can understand that it’s actually a knife that you’re throwing at someone and not just a joke ”).

There is limited evidence exploring the impact of cultural humility training with healthcare professional educators. Bakaa et al. [ 54 ] surveyed cultural competence in a sample of 3,000 chiropractors and reported similar findings which suggested that gaps between knowledge and self-reported behaviour required further research to clarify barriers and guide future training. Flateland et al. [ 55 ] concluded that inclusivity could be increased through mandatory diversity training which emphasised individual learning needs for students from all backgrounds and was supported by mentoring from personal academic tutors and a buddy system for UrG students.

A focus group study by Shapiro et al. [ 56 ] suggests that training increased awareness among third year medical students (first year of clinical training) but was less helpful in developing specific management skills. In contrast, another study found that, medical students tended to minimise the importance of self-awareness or the need to reflect on, and confront, personal biases [ 50 ]. Despite uncertainty about the impact of training, there is consensus that lack of training is also problematic. Whether based on concepts of cultural awareness, competence and humility [ 51 ] it is important that the sceptical perception that training is trying to be ‘politically correct’ is transformed into a way of rehumanising healthcare education [ 56 ]. Education in EDI and inclusive communication skills was strongly recommended by the participants in this study, but the challenges cited above suggest that ongoing monitoring would be needed to explore its’ impact on staff and students (Focus group theme 2: “ I don’t know what mandatory [EDI] training’s given, but it would seem like potentially a lot of it [othering] could potentially be stopped ”).

Supporting students

Inequalities in healthcare education are well documented [ 11 , 12 , 16 , 18 ]. Physiotherapy students from black, Asian and minority ethnic (BAME) backgrounds received lower marks in observed assessments compared to white students, with gaps in attainment also recorded for people with disabilities and students with non-traditional entry routes [ 10 ]. Overseas students, especially those who do not speak English as a first language, report isolation, loneliness, and lack of support, which is increased by intersectionality including race and gender [ 9 , 57 ]). In the survey, 15 students who felt they had been treated differently because of UrG characteristics did not report their difficulties, sometimes because they were unclear about whether an incident would count as discrimination or whether reporting a problem would have negative consequences (Workshop theme 3: “ Need to clarify what language/behaviour (e.g., ‘banter’) is acceptable ”).

Barriers to reporting misconduct include fear of not being believed, fear of repercussions and lack of confidence that complaints will be taken seriously [ 58 ]. Focus group and workshop comments suggested that students felt concerns were ignored, whether reported by individuals or year group representatives. The institution was rarely seen to take action to address the problems identified and there were concerns about consequences for people who spoke out. In contrast, some participants felt that whistleblowers should be valued and that incidents of discrimination could be reduced by encouraging more people to speak up (Workshop theme 4: “ Value all experiences and validate the ‘disruptor’ voice ”).

Research suggests that some of the factors that hinder the delivery of effective student support include limited disclosure of individual difficulties, especially for ‘invisible’ disabilities [ 18 ], the complex challenges faced by students with intersectional backgrounds [ 59 , 60 ], and lack of staff awareness, as discussed above [ 20 , 61 ]. Inconsistent institutional support practices also reinforce students’ disabled status and limit participation, rather than optimising their abilities and resilience [ 61 ], so there is a need to develop clear, robust systems to support students from UrGs, such as Active Bystander training (Workshop theme 2).

Improving institutional policies and processes

The practical processes used to support students and manage staff are grounded in an institution’s values and policies. Training inequalities are known to be a concern in medical and allied health professions and all HEIs in the UK have a responsibility to overcome the challenges of inaction in the face of discrimination. The General Medical Council has recently set new targets to eradicate disadvantage and discrimination in medical education and training [ 21 ]. Equality, diversity and inclusion pose challenges for small specialist universities, as noted in the ‘Changing the Culture’ (2016) framework, developed by Universities UK and GuildHE [ 62 ]. OEPs are expected to cultivate and maintain a culture of inclusion between staff, students and patients, train staff in EDI and ensure that staff are involved in the development of EDI policies [ 63 ]. This is reflected in the Quality Assurance Agency for Higher Education Subject Benchmark Statement for Osteopathy [ 64 ]: expectations and guidance on how OEPs can promote an EDI culture are provided. Participants in this study reported concerns about institutional knowledge (Focus group theme 2) and lack of clarity about how to access and use existing EDI policies (Focus group theme 1: “ How do I go about telling someone about that? ”).

In recent decades, access and participation from minority groups to higher education in the UK has been a core focus and entry rates for non-white students have increased: in 2019 they were higher for all ethnic groups compared with rates in 2006 and the entry rates increased in 2019 compared with 2018 [ 65 ]. There is limited information about experiences of inequalities reported by UrG students in osteopathic education or discrepancies in levels of attainment. A systematic review by MacMillan et al. [ 31 ] analysed discrimination, bullying and harassment in manual therapy education. They reported that there was evidence of widespread discrimination, harassment and bullying within manual therapy education; and there was a clear need for further research to focus upon the intersection of the characteristics identified as being linked to these experiences. Unfortunately, no osteopathic studies were found, although findings from physiotherapy and chiropractic education are likely to be transferable. Practising osteopaths from UrGs are also reported to be dissatisfied with lack of diversity within the profession and concerns have been raised about a lack of cultural competence training in OEPs [ 66 ].

Norris et al. [ 61 ] recommended that healthcare education institutions need to provide consistent and accessible information to help students find appropriate support and education to increase staff awareness about how individual experiences of disability affect learning. Complex EDI issues require university-wide approaches and AdvanceHE’s UK Equality Charter team proposes an ‘holistic approach’ [ 62 ]. Further research is needed to identify actions which would enhance educational experiences and outcomes for student osteopaths from UrGs. New data would also provide insights into the extent that osteopathic education prepares students to work with patients from UrGs and support long-term plans to enhance access and quality of patient care and attract more students from these UrGs to enhance the profession and represent more inclusively the communities they serve [ 31 , 64 ].

Limitations of the study

It is difficult to collect data from people who feel marginalised or vulnerable to discrimination, as demonstrated by low survey response rates with participants who typically have strong positive or negative views but few from the ‘silent majority’ (Shapiro et al. 2016). The MCHS is a new instrument which was adapted to osteopathy students, and due to the small sample size, it was not possible to get useful results with the confirmatory factor analysis. More research is also needed with this instrument to establish meaningful scores for dimensions of questionnaire. The response rate to this survey was low at 20% and there were fewer than 8 participants in all the focus groups. However, mixed designs enable compensation for some limitations of individual methods and data was collected from all seven UK OEPs. Two stages of qualitative analysis (focus groups and workshops) also enabled triangulation of the findings. The impact of facilitators as ‘insiders’ on data collection was not assessed and it was challenging to synthesise and weight results from the three stages.

Conclusions

The aims of this mixed methods study were to assess awareness of cultural humility among student osteopaths in the UK and to explore educational experiences of discrimination and ‘othering’ among students from underrepresented groups. Our findings are consistent with conclusions from other studies and the suggestions for action generated in workshops with diverse stakeholders are aligned with current EDI guidelines. Our three main recommendations are that OEIs prioritise actions to clarify institutional policies and processes to ensure they are accessible and effective in maintaining an inclusive educational environment; to review the adequacy of current student support services, particularly for underrepresented groups; and to provide EDI and communications skills training for staff to increase awareness about students’ learning needs and explore attitudinal barriers to change.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to the sensitivity of data collected and risk of identification of participants but are available from the corresponding author on reasonable request.

Szetela A. Black lives matter at five: limits and possibilities. Ethn Racial Stud. 2020;43(8):1358–83.

Article   Google Scholar  

Regulska J. The #MeToo movement as a global learning moment. Int High Educ. 2018;94:5–6.

Dodd V. Met police found to be institutionally racist, misogynistic and homophobic. The guardian [Internet]. 2023 Mar 21 [cited 2024 Jan 10]; https://www.theguardian.com/uk-news/2023/mar/21/metropolitan-police-institutionally-racist-misogynistic-homophobic-louise-casey-report .

Quinn B. People like you still uttered: BAME armed forces personnel on racism in services. The guardian [Internet]. 2020 Jan 7 [cited 2024 Jan 10]; https://www.theguardian.com/uk-news/2020/jan/07/people-like-you-still-uttered-bame-armed-forces-personnel-on-racism-in-services .

The report of the Commission on Race. and Ethnic Disparities [Internet]. Gov.uk. 2021 [cited 2024 Jan 10]. https://www.gov.uk/government/publications/the-report-of-the-commission-on-race-and-ethnic-disparities .

Adams R. Minority ethnic Britons’ educational success not reflected in pay. 2022.

Higher Education Statistics Agency. Higher Education Graduate Outcomes Statistics: UK, 2017/18 - Graduate activities and characteristics [Internet]. 2020 [cited 2024 Jan 10]. https://www.hesa.ac.uk/news/18-06-2020/sb257-higher-education-graduate-outcomes-statistics/activities .

Rimmer A. One in four trainee doctors report discriminatory behaviours from colleagues, GMC finds. BMJ. 2023;1601.

British Medical Association. Factors preventing progress of ethnic minority doctors into senior positions [Internet]. 2022 [cited 2024 Jan 10]. https://www.bma.org.uk/media/5748/bma-why-are-we-still-here-commissioned-report-june-15-2022.pdf .

Norris M, Hammond JA, Williams A, Grant R, Naylor S, Rozario C. Individual student characteristics and attainment in pre registration physiotherapy: a retrospective multi site cohort study. Physiotherapy. 2018;104(4):446–52.

Neves J, Hillman N. Student academic experience survey. H. E. P. I. a. H. E. Academy, editor.; 2020.

Bachmann CL, Gooch B. LGBT in Britain - universities Report. London: Stonewall; 2018.

Google Scholar  

Universities, and Colleges Admissions Service. Entry rates into higher education. Ethnicity, factsfigures [Internet]. 2023 [cited 2024 Jan 10]. https://www.ethnicity-facts-figures.service.gov.uk/education-skills-and-training/higher-education/entry-rates-into-higher-education/latest#by-ethnicity .

Johnson JL, Bottorff JL, Browne AJ, Grewal S, Hilton BA, Clarke H. Othering and being othered in the context of health care services. Health Commun. 2004;16(2):255–71.

Nixon SA. The coin model of privilege and critical allyship: implications for health. BMC Public Health. 2019;19(1):1637.

ESFA. Education and Skills agreements 2020 to 2021 [Internet]. Gov.uk. 2020 [cited 2024 Jan 10]. https://www.gov.uk/guidance/esfa-education-and-skills-agreements-2020-to-2021 .

Office for Students. Equality, diversity and student characteristics data - Office for Students. 2022 Jun 7 [cited 2024 Jan 10]; https://www.officeforstudents.org.uk/publications/equality-diversity-and-student-characteristics-data/?_cldee=Hgjd4C0mN-Ei1iAE9EzevenIhwSZ7Dxxo3wjIT8PIAo1SUA3S-H3H-VFCowOM6WI&recipientid=lead-46be407d3275ec1189430022481aa1a1-74564b23bb16438c9a14de1abd81377a&esid=cd365273-6de5-ec11-bb3c-002248004e75

UCAS. What is the experience of disabled students in education? [Internet]. 2022 [cited 2024 Jan 10]. https://www.ucas.com/file/610106/download?token=1kwt_gKE .

Hammond J, Marshall-Lucette S, Davies N, Ross F, Harris R. Spotlight on equality of employment opportunities: a qualitative study of job seeking experiences of graduating nurses and physiotherapists from black and minority ethnic backgrounds. Int J Nurs Stud. 2017;74:172–80.

Hammond JA, Williams A, Walker S, Norris M. Working hard to belong: a qualitative study exploring students from black, Asian and minority ethnic backgrounds experiences of pre-registration physiotherapy education. BMC Med Educ [Internet]. 2019;19(1). https://doi.org/10.1186/s12909-019-1821-6 .

General Medical Council. GMC targets elimination of disproportionate complaints and training inequalities [Internet]. 2021 [cited 2021 Autumn 5]. https://www.gmc-uk.org/news/news-archive/gmc-targets-elimination-of-disproportionate-complaints-and-training-inequalities .

Ross MH, Setchell J. People who identify as LGBTIQ + can experience assumptions, discomfort, some discrimination, and a lack of knowledge while attending physiotherapy: a survey. J Physiother. 2019;65(2):99–105.

Ross MH, Setchell J. Physiotherapists vary in their knowledge of and approach to working with patients who are LGBTQIA+: a qualitative study. J Physiother. 2023;69(2):114–22.

Council of Deans. Anti-racism in AHP Education: Building an Inclusive Environment [Internet]. 2023 [cited 2024 Jan 10]. https://www.councilofdeans.org.uk/wp-content/uploads/2023/03/Anti.racism.in_.ahp_.education.report.pdf .

House of Commons. House of Commons Women and Equalities Committee Black maternal health Third Report of. Session 2022–23 Report, together with formal minutes relating to the report Ordered by the House of Commons [Internet]. 2023 [cited 2024 Jan 10]. https://committees.parliament.uk/publications/38989/documents/191706/default/ .

Gonzales G, Przedworski J. Comparison of health and health risk factors between lesbian, gay, and bisexual adults and heterosexual adults in the United States: Results from the National Health…. JAMA Intern Med [Internet]. 2016; https://jamanetwork.com/journals/jamainternalmedicine/article-abstract/2530417?casa_token=CRF1uNk3F_8AAAAA:pJQ_7UypXiwhMMO0KeqLEGf5Jb6fZC_EBg0QrmXRF507vY5Pl7srcTta6fWsDqHmuOQCZYIGoQ&casa_token=OWWuxn8UEDwAAAAA:4YRNKNxWmG-6LZfnl_MEATduHjHiawDrLprat8Y59Nopll1K-Yw1HV-zL5K15z5lZBn1X9FPUg

Cross TL. Towards a culturally competent system of care: a monograph on effective services for minority. Washington, DC: Georgetown University; 1989.

Hook JN, Davis DE, Owen J, Worthington EL, Utsey SO. Cultural humility: measuring openness to culturally diverse clients. J Couns Psychol. 2013;60(3):353–66.

NHS England. Chief Allied Health Professions Officer extends her remit to two additional professions [Internet]. 2017 [cited 2024 Jan 10]. https://www.england.nhs.uk/2017/04/chief-allied-health-professions-officer-extends-her-remit-to-two-additional-professions/ .

General Osteopathic Council. Standards of Practice [Internet]. 2023 [cited 2024 Jan 10]. https://www.osteopathy.org.uk/standards/ .

MacMillan A, Hohenschurz-Schmidt D, Migliarini DV, Draper-Rodi DJ. Discrimination, bullying or harassment in undergraduate education in the osteopathic, chiropractic and physiotherapy professions: a systematic review with critical interpretive synthesis. Int J Educ Res Open. 2022;3(100105):100105.

O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8.

Creswell JW, Clark VP. Designing and conducting mixed methods research. SAGE; 2018.

Mertens DM. Transformative paradigm. J Mix Methods Res. 2007;1(3):212–25.

Creswell JW. Research Design: qualitative, quantitative, and mixed methods approaches. SAGE; 2014.

Ball S, Harshfield S, Carpenter A, Bertscher A, Marjanovic S. Patient and public involvement and engagement in research. Santa Monica: RAND Corporation; 2019.

Gonzalez E, Sperandio KR, Mullen PR, Tuazon VE. Development and initial testing of the multidimensional cultural humility scale. Meas Eval Couns Dev. 2021;54(1):56–70.

Finch H. Focus groups. Qualitative research practice: A guide for social science students and researchers. Ritchie J, London LJ, editors. J Ritchie and L J. 2003;170–98.

Holloway I, Wheeler S. Qualitative research in nursing and healthcare. Wiley; 2010.

Kite J, Phongsavan P. Insights for conducting real-time focus groups online using a web conferencing service. F1000Res. 2017;6:122.

Mertens DM. Transformative mixed methods research. Qual Inq. 2010;16(6):469–74.

Kaiser K. Protecting respondent confidentiality in qualitative research. Qual Health Res. 2009;19(11):1632–41.

R Core Team. A Language and Environment for Statistical Computing [Internet]. R Foundation for Statistical Computing, Vienna, Austri; 2023 [cited 2024 Oct 1]. https://www.R-project.org/ .

Rosseel Y. lavaan: AnRPackage for Structural Equation Modeling. J Stat Softw [Internet]. 2012;48(2). https://doi.org/10.18637/jss.v048.i02 .

Van Buuren S. Groothuis-Oudshoorn, mice: Multivariate imputation by chained equations in r. J Stat Softw. 2011;45:1–67.

Welch BL. The generalization of ‘student’s’ problem when several different population varlances are involved. Biometrika. 1947;34(1–2):28–35.

Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc Health. 2019;11(4):589–97.

Yvonne Feilzer M. Doing mixed methods research pragmatically: implications for the rediscovery of pragmatism as a research paradigm. J Mix Methods Res. 2010;4(1):6–16.

Tollemache N, Shrewsbury D, Llewellyn C. Que(e) rying undergraduate medical curricula: a cross-sectional online survey of lesbian, gay, bisexual, transgender, and queer content inclusion in UK undergraduate medical education. BMC Med Educ [Internet]. 2021;21(1). https://doi.org/10.1186/s12909-021-02532-y .

Shepherd SM. Cultural awareness workshops: limitations and practical consequences. BMC Med Educ. 2019;19(1):14.

Tervalon M, Murray-García J. Cultural humility versus cultural competence: a critical distinction in defining physician training outcomes in multicultural education. J Health Care Poor Underserved. 1998;9(2):117–25.

Foronda C, Baptiste D-L, Reinholdt MM, Ousman K. Cultural humility. J Transcult Nurs. 2016;27(3):210–7.

Solchanyk D, Ekeh O, Saffran L, Burnett-Zeigler IE, Doobay-Persaud A. Integrating cultural humility into the medical education curriculum: strategies for educators. Teach Learn Med. 2021;33(5):554–60.

Bakaa N, Southerst D, Côté P, Macedo L, Carlesso LC, MacDermid J et al. Assessing cultural competency among Canadian chiropractors: a cross-sectional survey of Canadian Chiropractic Association members. Chiropr Man Therap [Internet]. 2023;31(1). https://doi.org/10.1186/s12998-023-00474-4 .

Flateland SM, Pryce-Miller M, Skisland AV-S, Tønsberg AF, Söderhamn U. Exploring the experiences of being an ethnic minority student within undergraduate nurse education: a qualitative study. BMC Nurs [Internet]. 2019;18(1). https://doi.org/10.1186/s12912-019-0389-0 .

Jowsey T. Three zones of cultural competency: surface competency, bias twilight, and the confronting midnight zone. BMC Med Educ [Internet]. 2019;19(1). https://doi.org/10.1186/s12909-019-1746-0 .

Shapiro J, Lie D, Gutierrez D, Zhuang G. That never would have occurred to me: a qualitative study of medical students’ views of a cultural competence curriculum. BMC Med Educ [Internet]. 2006;6(1). https://doi.org/10.1186/1472-6920-6-31 .

Equality. and Human Rights Commission. Tackling racial harassment: Universities challenged. 2019.

Gottlieb M. The case for a cultural humility framework in social work practice. J Ethn Cult Divers Soc Work. 2021;30(6):463–81.

LeBlanc C, Sonnenberg LK, King S, Busari J. Medical education leadership: from diversity to inclusivity. GMS J Med Educ. 2020;37(2):Doc18.

Norris M, Hammond J, Williams A, Walker S. Students with specific learning disabilities experiences of pre-registration physiotherapy education: a qualitative study. BMC Med Educ [Internet]. 2020;20(1). https://doi.org/10.1186/s12909-019-1913-3 .

GuildHE. EDI & ANTI-RACISM BRIEFING #15. Tackling Harassment and the importance of EDI. 2021.

General Osteopathic Council. Graduate Outcomes and Standards for Education and Training [Internet]. London. 2022. https://www.osteopathy.org.uk/news-and-resources/document-library/publications/graduate-outcomes-and-standards-for-education-and-training/ .

Quality Assurance Agency for Higher Education. Subject Benchmark Statement Osteopathy. Gloucester: QAA; 2024.

Universities, and Colleges Admissions Service. Entry rates into higher education [Internet]. Entry rates into higher education. Ethnicity, factsfigures. 2020 [cited 2021 Aug 2]. https://www.ethnicity-facts-figures.service.gov.uk/education-skills-and-training/higher-education/entry-rates-into-higher-education/latest#by-ethnicity .

MacMillan A, Corser A, Clark Z. Inclusivity and accessibility in undergraduate osteopathic education for students with disability: a scoping review. Int J Osteopath Med. 2021;40:38–45.

Download references

Acknowledgements

Our thanks to Dr Phil Bright and Mr Dévan Rajendran for their support with the project, including the community engagement meetings.

This project received funding from four organisations: The Osteopathic Foundation provided £20,000, the General Osteopathic Council provided £7,500, the University College of Osteopathy provided £7,500, and the Institute of Osteopathy provided £3,000. The authors, including the Principal Investigator, are employed by the University College of Osteopathy. However, the University College of Osteopathy and other funders did not have any specific role in the conceptualization, design, data collection, analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

University College of Osteopathy, 275 Borough High Street, SE1 1JE, London, UK

Jerry Draper-Rodi, Hilary Abbey, Oliver T. Thomson, Kevin Brownhill, Andrew MacMillan, Yinka Fabusuyi & Steven Vogel

National Council for Osteopathic Research, 275 Borough High Street, SE1 1JE, London, UK

Jerry Draper-Rodi

School of Allied and Public Health Professions, North Holmes Road, CT1 1QU, Canterbury, Kent, UK

John Hammond

University of Portsmouth, University House, Winston Churchill Ave, PO1 2UP, Hampshire, Portsmouth, England

Andrew MacMillan

You can also search for this author in PubMed   Google Scholar

Contributions

JDR, HA and SV designed the study and applied for ethical approval. AMM, HA, JDR, SV, YF facilitated the focus groups. KB collected and analysed the quantitative data; OT analysed and interpreted the qualitative data. HA, JDR, SV, YF facilitated the workshops. HA and JDR analysed the workshop data. All authors contributed, read and approved the final manuscript.

Corresponding author

Correspondence to Jerry Draper-Rodi .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the University College of Osteopathy Research Ethics Committee. For the survey, consent was assumed by return of the completed questionnaire (this was explained in the participant information sheet), and for the focus groups, informed consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Draper-Rodi, J., Abbey, H., Hammond, J. et al. Overcoming barriers to equality, diversity, inclusivity, and sense of belonging in healthcare education: the Underrepresented Groups’ Experiences in Osteopathic Training (UrGEnT) mixed methods study. BMC Med Educ 24 , 468 (2024). https://doi.org/10.1186/s12909-024-05404-3

Download citation

Received : 02 February 2024

Accepted : 09 April 2024

Published : 26 April 2024

DOI : https://doi.org/10.1186/s12909-024-05404-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Underrepresented
• Social Justice
• Equity diversity and inclusion
• Transformative action research
• Osteopathic undergraduate education
• Ethnic minorities

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Nepal J Epidemiol
• v.12(1); 2022 Mar

The Growing Importance of Mixed-Methods Research in Health

Sharada prasad wasti.

1,2 School of Human and Health Sciences, University of Huddersfield, United Kingdom

Padam Simkhada

3 Centre for Midwifery, Maternal and Perinatal Health, Bournemouth University, Bournemouth, United Kingdom

Edwin R. van Teijlingen

Brijesh sathian.

4 Geriatrics and long term care Department, Rumailah Hospital, Hamad Medical Corporation, Doha, Qatar

Indrajit Banerjee

5 Sir Seewoosagur Ramgoolam Medical College, Belle Rive, Mauritius

All authors have made substantial contributions to all of the following: (1) the conception and design of the study (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted

There is no conflict of interest for any author of this manuscript.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sector.

This paper illustrates the growing importance of mixed-methods research to many health disciplines ranging from nursing to epidemiology. Mixed-methods approaches requires not only the skills of the individual quantitative and qualitative methods but also a skill set to bring two methods/datasets/findings together in the most appropriate way. Health researchers need to pay careful attention to the ‘best’ approach to designing, implementing, analysing, integrating both quantitative (number) and qualitative (word) information and writing this up in a way offers greater insights and enhances its applicability. This paper highlights the strengths and weaknesses of mixed-methods approaches as well as some of the common mistakes made by researchers applying mixed-methods for the first time.

Quantitative and qualitative research methods each address different types of questions, collect different kinds of data and deliver different kinds of answers. Each set of methods has its own inherent strengths and weaknesses, and each offers a particular approach to address specific types of research questions (and agendas). Health disciplines such as dentistry, nursing, speech and language therapy, and physiotherapy often use either quantitative or qualitative research methods on their own. However, there is a steadily growing literature showing the advantages of mixed-methods research is used in the health care and health service field [ 1-2 ]. Although we have advocated the use of mixed-methods in this journal eight years ago [ 3 ], there is still not enough mixed-methods research training in the health research field, particularly for health care practitioners, such as nurses, physiotherapists, midwives, and doctors, wanting to do research. Mixed-methods research has been popular in the social sciences since the twentieth century [ 4 ], and it has been growing in popularity among healthcare professionals [ 5 ], although it is still underdeveloped in disciplines such nursing and midwifery [ 6 , 7 ].

Underpinning philosophies

To help understand that mixed-methods research is not simply employing two different methods in the same study, one needs to consider their underpinning research philosophies (also called paradigms). First, quantitative research is usually underpinned by positivism. This includes most epidemiological studies; such research is typically based on the assumption that there is one single real world out there that can be measured. For example, quantitative research would address the question “What proportion of the population of India drinks coffee?” Secondly, qualitative research is more likely to be based on interpretivism. This includes research based on interviews and focus groups, research which us is typically based on the assumption that we all experience the world differently. Since we all live in a slightly different world in our heads the task of qualitative research is to analyse the interpretations of the people in the sample. For example, qualitative research would address the question “How do people experience drinking coffee in India?”, and “What does drinking coffee mean to them?”

Mixed-methods research brings together questions from two different philosophies in what is being referred to as the third path [ 8 ], third research paradigm [ 9 , 10 ], the third methodology movement [ 11 , 12 ] and pragmatism [ 5 ]. The two paradigms differ in key underlying assumptions that ultimately lead to choices in research methodology and methods and often give a breadth by answering more complicated research questions [ 4 ]. The roles of mixed-methods are clear in an understanding of the situation (the what), meaning, norms, values (the why or how) within a single research question which combine the strength of two different method and offer multiple ways of looking at the research question [ 13 ]. Epidemiology sits strongly in the quantitative research corner, with a strong emphasis on large data sets and sophisticated statistical analysis. Although the use of mixed methods in health research has been discussed widely researchers raised concerns about the explanation of why and how mixed methods are used in a single research question [ 5 ].

The relevance of mixed-methods in health research

The overall goal of the mixed-methods research design is to provide a better and deeper understanding, by providing a fuller picture that can enhance description and understanding of the phenomena [ 4 ]. Mixed-methods research has become popular because it uses quantitative and qualitative data in one single study which provides stronger inference than using either approach on its own [ 4 ]. In other words, a mixed-methods paper helps to understand the holistic picture from meanings obtained from interviews or observation to the prevalence of traits in a population obtained from surveys, which add depth and breadth to the study. For example, a survey questionnaire will include a limited number of structured questions, adding qualitative methods can capture other unanticipated facets of the topic that may be relevant to the research problem and help in the interpretation of the quantitative data. A good example of a mixed-methods study, it one conducted in Australia to understand the nursing care in public hospitals and also explore what factors influence adherence to nursing care [ 14 ]. Another example is a mixed-methods study that explores the relationship between nursing care practices and patient satisfaction. This study started with a quantitative survey to understand the general nursing services followed by qualitative interviews. A logistic regression analysis was performed to quantify the associations between general nursing practice variables supplemented with a thematic analysis of the interviews [ 15 ]. These research questions could not be answered if the researchers had used either qualitative or quantitative alone. Overall, this fits well with the development of evidence-based practice.

Despite the strengths of mixed-methods research but there is not much of it in nursing and other fields [ 7 ]. A recent review paper shows that the prevalence of mixed-methods studies in nursing was only 1.9% [ 7 ]. Similarly, a systematic review synthesised a total of 20 papers [ 16 ], and 16 papers [ 17 ] on nursing-related research paper among these only one mixed-methods paper was identified. Worse, a further two mixed-methods review recently revealed that out of 48 [ 18 , 19 ] synthesised nursing research papers, not one single mixed-methods paper was identified. This clearly depicts that mixed-methods research is still in its infancy stage in nursing but we can say there is huge scope to implement it to understand research questions on both sides of coin [ 4 ]. Therefore, there is a great need for mixed-methods training to enhance the evidence-based decision making in health and nursing practices.

Strengths and weaknesses of mixed-methods

There are several challenges in identifying expertise of both methods and in working with a multidisciplinary, interdisciplinary, or transdisciplinary team [ 20 ]. It increases costs and resources, takes longer to complete as mixed-methods design often involves multiple stages of data collection and separate data analysis [ 4 , 5 ]. Moreover, conducting mixed-methods research does not necessarily guarantee an improvement in the quality of health research. Therefore, mixed-methods research is only appropriate when there are appropriate research questions [ 4 , 6 ].

Identifying an appropriate mixed-methods journal can also be challenging when writing mixed-methods papers [ 21 ]. Mixed-methods papers need considerably more words than single-methods papers as well as sympathetic editors who understand the underlying philosophy of a mixed-methods approach. Such papers, simply require more words. The mixed-methods researcher must be reporting two separate methods with their own characteristics, different samples, and ways of analysing, therefore needs more words to describe both methods as well as both sets of findings. Researcher needs to find a journal that accepts longer articles to help broaden existing evidence-based practice and promote its applicability in the nursing field [ 22 ].

Common mistakes in applying mixed-methods

Not all applied researchers have insight into the underlying philosophy and/or the skills to apply each set of methods appropriately. Younas and colleagues’ review identified that around one-third (29%) of mixed-methods studies did not provide an explicit label of the study design and 95% of studies did not identify the research paradigm [ 7 ]. Whilst several mixed-methods publications did not provide clear research questions covering both quantitative and qualitative approaches. Another common issue is how to collect data either concurrent or sequential and the priority is given to each approach within the study where equal or dominant which are not clearly stated in writing which is important to mention while writing in the methods section. Similarly, a commonly overlooked aspect is how to integrate both findings in a paper. The responsibility lies with the researcher to ensure that findings are sufficiently plausible and credible [ 4 ]. Therefore, intensive mixed-methods research training is required for nursing and other health practitioners to ensure its appropriate.

The way forward

Despite the recognised strengths and benefits of doing mixed-methods research, there is still only a limited number of nursing and related-health research publications using such this approach. Researchers need training in how to design, conduct, analyse, synthesise and disseminate mixed-methods research. Most importantly, they need to consider appropriate research questions that can be addressed using a mixed methods approach to add to our knowledge in evidence-based practice. In short, we need more training on mixed-methods research for a range of health researchers and health professionals.

Acknowledgement

• Open access
• Published: 23 April 2024

A mixed methods evaluation of the impact of ECHO ® telementoring model for capacity building of community health workers in India

• Rajmohan Panda 1 ,
• Supriya Lahoti   ORCID: orcid.org/0000-0001-6826-5273 2 ,
• Nivedita Mishra 2 ,
• Rajath R. Prabhu 3 ,
• Kalpana Singh 4 ,
• Apoorva Karan Rai 2 &
• Kumud Rai 2  

Human Resources for Health volume  22 , Article number:  26 ( 2024 ) Cite this article

61 Accesses

1 Altmetric

Metrics details

Introduction

India has the largest cohort of community health workers with one million Accredited Social Health Activists (ASHAs). ASHAs play vital role in providing health education and promoting accessible health care services in the community. Despite their potential to improve the health status of people, they remain largely underutilized because of their limited knowledge and skills. Considering this gap, Extension for Community Healthcare Outcomes (ECHO) ® India, in collaboration with the National Health System Resource Centre (NHSRC), implemented a 15-h (over 6 months) refresher training for ASHAs using a telementoring interface. The present study intends to assess the impact of the training program for improving the knowledge and skills of ASHA workers.

We conducted a pre–post quasi-experimental study using a convergent parallel mixed-method approach. The quantitative survey ( n  = 490) assessed learning competence, performance, and satisfaction of the ASHAs. In addition to the above, in-depth interviews with ASHAs ( n  = 12) and key informant interviews with other stakeholders ( n  = 9) examined the experience and practical applications of the training. Inferences from the quantitative and qualitative approaches were integrated during the reporting stage and presented using an adapted Moore’s Expanded Outcomes Framework.

There was a statistically significant improvement in learning ( p =  0.038) and competence ( p =  0.01) after attending the training. Participants were satisfied with the opportunity provided by the teleECHO™ sessions to upgrade their knowledge. However, internet connectivity, duration and number of participants in the sessions were identified as areas that needed improvement for future training programs. An improvement in confidence to communicate more effectively with the community was reported. Positive changes in the attitudes of ASHAs towards patient and community members were also reported after attending the training. The peer-to-peer learning through case-based discussion approach helped ensure that the training was relevant to the needs and work of the ASHAs.

Conclusions

The ECHO Model ™ was found effective in improving and updating the knowledge and skills of ASHAs across different geographies in India. Efforts directed towards knowledge upgradation of ASHAs are crucial for strengthening the health system at the community level. The findings of this study can be used to guide future training programs.

Trial registration The study has been registered at the Clinical Trials Registry, India (CTRI/2021/10/037189) dated 08/10/2021.

Peer Review reports

The Alma Ata Declaration of 1978 has recognized primary health care as an essential element for improving community health. Community health workers (CHWs) have the potential to complement an overstrained health workforce and enhance primary healthcare access and quality [ 1 ]. Low- and middle-income countries (LMICs) face a triple burden of low density of doctors and nurse-midwives, low government expenditure on health, and disproportionately larger poor health outcomes [ 2 ]. The roles and responsibilities of CHWs vary across LMICs [ 3 ]. A systematic review has documented that the socio-cultural, economic, health system, and political context in which CHW interventions operate in LMICs influence the implementation and success of interventions [ 4 ].

The National Rural Health Mission (NRHM), India introduced Accredited Social Health Activists (ASHAs) as female CHWs in 2005. The ASHAs are women volunteers selected from the local village and were initially conceptualized with a vision to improve maternal and child health in the country; however, over time, they are now involved in different national health programmes [ 5 , 6 ]. Despite their potential to contribute to preventive and promotive healthcare, they remain largely underutilized because of their limited knowledge and skills [ 1 ]. The World Health Organisation (WHO) has suggested ‘regular training and supervision’ for CHWs to fulfil their role successfully [ 7 ]. In India, the health system lacks methods for continuous education and routine upgradation of the ASHA’s skills [ 8 , 9 , 10 ].

In LMICs, digital training programs can help expand the reach of training to large numbers of healthcare workers at a low cost without interfering with the delivery of routine healthcare services [ 11 , 12 ]. An evidence-mapping study of 88 studies that used technology for training CHWs in LMICs found that the focus of trainings was maternal and child health and other high-burden diseases were neglected [ 13 ]. In India, studies evaluating digital trainings for CHWs have focussed on specific diseases or have been limited to specific states in the last decade [ 10 , 14 ]. This study was conducted across multiple states. More such studies with larger sample size are needed on the evaluation of such training initiatives in India [ 13 , 15 , 16 ].

Project Extension for Community Healthcare Outcomes (ECHO) presents an educational opportunity for capacity-building through a telementoring platform that uses video conferencing to create a continuous loop of learning and peer support. The sessions are facilitated by didactic presentation and case-based learning that allows problem-solving through shared best practices [ 17 ]. ECHO India, in collaboration with National Health System Resource Centre (NHSRC), provided refresher training for ASHAs [ 18 ]. There is increasing evidence of the positive effect of ECHO training on medical provider’s learning and self-efficacy. However, its value as a training platform to CHWs in LMICs is limited. Previous studies that evaluated the use of the ECHO Model ™ for CHWs focussed on specific diseases and were conducted in high-income countries (HICs) [ 19 , 20 , 21 ]. For the adoption of digital technology, CHWs in LMICs encounter challenges such as poor proficiency levels in accessing and using digital platforms, limited access to troubleshooting, poor internet connectivity, and in-house support for resolving issues [ 22 ]. The present study was designed to assess the impact of the ECHO telementoring model for improving the knowledge and skills of ASHA workers in delivering comprehensive health services. This will provide new insights for measuring outcomes of digital training programs for CHWs (ASHA workers).

Study design

A pre–post quasi-experimental design using a convergent parallel mixed-method approach [ 23 ] was employed. The quantitative and qualitative data were collected concurrently. Inferences from both approaches were integrated during the reporting stage. This allowed for a comprehensive understanding of the effect of training on the knowledge and skills of ASHAs.

The ECHO training intervention and curriculum

Project ECHO ® designed a 15-h (over 6 months from October 2021 to March 2022), virtual, refresher training program to enhance the capacity of ASHAs to deliver counselling services for comprehensive healthcare in four states ( n  = 2293). Each session lasted for 90 min. The ECHO NHSRC training used a “hub and spoke” structure in which a multidisciplinary team of experts (trainers) based at a regional academic medical centre (the “hub”) engaged with the ASHAs (the “spokes”) [ 24 ] who attended the sessions from dedicated learning sites (PHCs). Each site also had a coordinator who would help facilitate the discussions and questions. The training curriculum was developed based on the NHSRC ‘ASHA training modules’ [ 18 ] in the regional languages in consultation with partners (hub-leaders and trainers). It comprised 10 sessions covering a range of topics, such as maternal health, new-born care, child health, nutrition, reproductive health, violence against women, tuberculosis, vector-borne diseases, non-communicable diseases, COVID-19, palliative care, and mental health. The training presentations included text with visual learning methods, such as images, videos, and links to training resources.

Study settings

The evaluation study was conducted in four states of India, where training sessions were held. These states represented the four geographical regions—northern (Himachal Pradesh) ( n =  499), southern (Tamil Nadu) ( n =  500), eastern (West Bengal) ( n =  618), and north-eastern (Sikkim) ( n =  676). The intervention (training sessions) was completed in March 2022. The end-point data were collected from March 2022 to May 2022.

Study participants and recruitment

Simple random sampling was used to select the ASHAs from each state for the quantitative survey. The participants were recruited from a list of ASHAs who would be receiving the ECHO NHSRC training. To be included, ASHAs had to be enrolled in the refresher training, planning to continue working for the next 10 months, with available contact details and consenting voluntarily. The ASHAs were contacted through mobile phones in each state. Key informant interviews (KIIs) were conducted with hub leaders who were involved in implementing the training, trainers (faculty) who delivered the lectures, and in-depth interviews (IDIs) with ASHAs.

Sample size

The sample size for the quantitative study was estimated by assuming a 25% improvement in knowledge and skills, 80% power, and a design effect factor of 1.7%. An adjustment of 30% loss to follow up and 20% non-response (from previous experience) led to a sample of 591 participants across four states, i.e., 148 participants from each state. For the qualitative study, purposive sampling with maximum variation across age, education, practice sites, and years of work experience was used for the selection of the participants. A total of 12 IDIs were conducted with ASHAs and nine KIIs with stakeholders (Additional file 2 : Appendix S2).

Study tools and data collection

For quantitative data collection, a structured questionnaire was designed through a collaborative approach with the research and program implementation team. The knowledge of ASHAs was assessed by a combination of 18 technical questions and case vignettes. Learning and competence, performance, and satisfaction were assessed with a 5-point Likert scale, using 1 = Strongly Disagree; 2 = Disagree; 3 = Neither Agree nor Disagree; 4 = Agree; and 5 = Strongly Agree. The face validity of the questionnaire was tested with ten ASHAs, separate from those recruited in the study and five primary care experts. The changes related to language, clarity, and relevance were made in the questionnaire based on the feedback from experts and participants. Separate discussion guides were developed for KIIs with trainers (Additional file 3 : Appendix S3) and hub-leaders (Additional file 4 : Appendix S4) and IDIs with ASHAs (Additional file 5 : Appendix S5). The guide focussed on examining the experience and practical applications of the training and was field tested before being administered in the main study. All study tools were translated into the local languages of the states and back-translated to check discrepancies.

The data were collected on the cell phone by experienced and trained researchers from social sciences backgrounds. Due to telephonic data collection, we were unable to capture non-verbal interview data such as emotions or gestures, particularly important in qualitative data. This may affect the richness of data and interpretation of responses. The quantitative tool was designed in the CS Pro software (version 7.5) and data were collected using its smartphone application. The qualitative interviews lasted around 40–50 min and were audio recorded. All interviews were translated and transcribed verbatim.

Data analysis

We summarized the quantitative data using descriptive statistics. Continuous variables were summarized using mean ± SD, and categorical variables were summarized using percentages and frequencies. The responses recorded using the 5-point Likert scale were recategorized during the analysis into three categories, i.e., ‘agree’ (combining strongly agree and agree), ‘disagree’ (combining strongly disagree and disagree), and ‘neutral [ 25 ]. Paired t test was used to find the difference between the pre- and post-scores of learning and competence and the attitude of participants toward ECHO training. McNemar’s test was used to assess changes in pre- and post-test scores for the technical domain. A p value of less than 0.05 was considered significant. STATA 16.0 statistical software was used for the analysis.

Qualitative data were analyzed according to the principles of the Framework approach [ 26 ], which combines inductive and deductive approaches. As a first step, two authors (SL and NM) familiarized themselves with four randomly selected transcripts and independently coded them using initial codes that were developed based on Moore’s framework levels of participation, satisfaction, learning, competence, and performance [ 27 ]. New codes that emerged while undertaking the analysis were included. The discussion and comparison of the double-coded transcripts enabled the development of an agreed set of codes. Any disagreements were discussed and resolved with the help of the third author (RP) to achieve inter-coder agreement. A final codebook was developed and applied to all the transcripts. The codes were combined and categorized into key emerging themes., The themes, including quotes (respondents’ exact words), were included to represent the main findings. Atlas.ti (version 8) software was used for data analysis.

Moore’s level 1—participation

Table 1 represents the baseline demographics of the recruited participants. From 610 participants who completed the pre-training survey, 490 participants completed the post-training survey, resulting in a follow-up rate of 80% (95% CI 76.6, 83.1). A total of 120 (20%, 95% CI 16.8, 23.3) participants were lost to follow up. This was due to a) contact numbers not being operational ( n =  96) and b) refusal due to time considerations ( n =  24). The field investigators attempted three additional phone calls, coordinated with hubs for participants’ alternate contact information, and offered flexible phone appointments to ensure maximum participation in the post-training survey. The majority (68%) of ASHAs were posted at sub-centres. A sub-centre is the most peripheral unit of contact of the health system with the community [ 28 ]. The majority of the participants (75%) had completed their high school (10th) education.

A hub leader described the efforts made by the ECHO to facilitate the participation of the ASHAs in the training.

“ECHO provided a facility where everyone can gather at the nearest block for the training. Physical and online modes [are] both available” (Hub-leader, Himachal Pradesh).

Moore’s level 2—satisfaction

The end-point survey assessed participants’ satisfaction with the ECHO training. The survey included eight items that measured overall training satisfaction and five items that measured satisfaction with factors specific to the telementoring model using close-ended questions. Satisfaction with the training content and environment was measured with four items. Except for one topic area (sharing of additional resources and training material), over 90% of participants were satisfied with almost all of the different components of the ECHO telementoring intervention (Additional file 1 : Appendix S1, Tables S1.1, S1.2, S1.3). While participants found the overall intervention favourable, 54.5% of all participants were dissatisfied with internet connectivity in the training sessions. Around one fourth of the participants faced challenges with the duration (31.2%), frequency (31.2%), and number of participants (28.4%) in the sessions (Additional file 1 : Appendix S1: Table S1.3).

The qualitative findings also show that most of the trainees were satisfied with the learning opportunity provided by the ECHO training.

“After attending these ECHO sessions, I felt we are constantly learning new techniques and it’s a deep sense of satisfaction” (ASHA, Tamil Nadu).

The ASHAs also shared areas or features of the ECHO model that did not meet their requirements and need improvement. They felt that the duration allotted for a session was not sufficient and some topics were covered very fast.

“They rush a lot while teaching over phone. It will be more helpful if they take more time and explain the things in a more detailed manner” (ASHA, WB)

Another ASHA suggested increasing the duration of training to improve their understanding of some topics.

"Increase the time of the training. Topics can be made deeper, and richer for better explanations" (ASHA, Tamil Nadu)

ASHAs described challenges related to connectivity while attending the training.

“The network connection was a problem and video used to lag” (ASHA, Sikkim)

Trainers shared their opinion about aspects of online trainings that did not meet their expectations.

“The problem is that they only join the meeting [online training] and do their own work, they actually do not listen properly.” (Trainer, WB)

A trainer mentioned that the large number of participants in some sessions affected the interaction among participant ASHAs.

“Sometimes a session has too many participants causing coordination efforts to be a challenge in these sessions” (Trainer, TN)

Difficulties in reaching the PHCs were recorded from the state of Sikkim. The geographical location and lack of transport facilities were mentioned by a trainer.

“We have transportation problem, our ASHA comes from rural area and it’s difficult to get taxi, which makes [it] harder to attend classes” (Trainer, Sikkim)

Many participants regarded organizational support as a facilitator for attending the training program. An ASHA from Tamil Nadu described how the issue of distance was resolved through management interventions from the organization.

“Our Block is 30 km away. There is another Block nearby that is 1 km only from here, they sent us there… so there was no problem” (ASHA, TN)

Moore’s level 3—learning

McNemar’s Chi-square statistic showed a significant difference between pre-ECHO and post-ECHO proportions in various aspects of health-related technical knowledge. Before the training, 1% of participants were aware of the correct schedule to be followed in the first week after the delivery of a child, which increased to 40% of participants post-training (p < 0.001). Overall, a statistically significant increase of 6% (95% CI 0.0003, 0.12; p =  0.038) in participants’ technical knowledge after ECHO training was found. After the training, a 7% increase in knowledge of malaria ( p =  0.002) and its symptoms and a 9% increase in knowledge of the right action to be undertaken (p < 0.001) was reported. Knowledge related to some areas such as recommended duration of physical activity or exercise (p < 0.001), immunisation after child birth ( p =  0.001), family planning in women after child birth ( p =  0.002) showed a decrease after attending the training (Additional file 1 : Appendix S1, Table S2). Post ECHO training, ASHAs reported an improvement in their knowledge of using a smartphone (switch on and off, and navigate) ( p =  0.0005) and navigating a mobile application ( p =  0.59). The ASHAs reported a 2% decrease in their knowledge of downloading content in the mobile ( p =  0.07) (Fig.  1 ).

Self-rated ICT knowledge of ASHAs

The qualitative data show that ASHAs who did not have a smartphone found it difficult to download and save content. One of the participants reported receiving additional training content in the form of a pdf file. She also mentioned that those who do not use a smartphone find it challenging to access this additional resource.

“We get the study material in a pdf so that simplifies our work further. But those who do not have a smartphone, find it difficult to get this opportunity” (ASHA, WB)

3A—Declarative learning

Declarative learning assesses how participants articulate the knowledge that the educational activity intended them to know (knowing what). The qualitative findings show that the training had increased the ASHA’s knowledge in specific domains such as breastfeeding during COVID-19.

“The doubt was whether a mother can breastfeed the baby when suffering from COVID-19. I got clarity about that… many such topics were cleared” (ASHA, Himachal Pradesh)

3B—Procedural learning

Procedural learning assesses the participants' articulation of how to do what the educational activity intended them to know (knowing how).

Participants reported that they had gained new skills related to the approach and identification of healthcare issues after attending the ECHO training.

“Earlier we wouldn’t know if ear related issues had a resolution – But following the ear related training we are aware that such issues can be cured or have treatments” (ASHA, Tamil Nadu).

The qualitative interviews revealed additional themes that described the value of the ECHO training program in improving the learning experience of ASHAs.

ASHA workers felt that the case presentations from their peers enhanced their learning experience.

“One ASHA shared a case of an anaemic mother. Based on this case we learned that this could have been prevented if iron tablets are provided from the adolescent stage” (ASHA, Tamil Nadu).

The interactive nature of the sessions and the discussions benefitted the learning experience of the ASHAs.

“Open discussion helped us so much. We can discuss any topics if we haven’t understood and sir used to explain again” (ASHA, Sikkim)

Moore’s level 4—competence

The participants reported significant improvement in their confidence to identify and manage several health conditions like birth asphyxia (for home deliveries) and management with a mucus extractor ( p =  0.01), screen and refer pregnant women ( p =  0.01), disseminate information on domestic violence and sexual harassment ( p =  0.001). Overall, a statistically significant increase of 6% (95% CI 0.01, 0.10; p =  0.01) in participants’ competence after attending the ECHO training was found. Participants reported a decrease in their confidence to track child immunisation ( p  < 0.001), monitor symptoms of COVID (p < 0.001), and clarify concerns of the community ( p  < 0.001) after attending the training (Additional file 1 : Appendix S1, Table S3).

Participants mentioned an improvement in their confidence while communicating with patients and their families.

“Initially we could not talk to people so comfortably, we hesitated at times but after being trained we can talk to people and their families properly and easily now” (ASHA, West Bengal)

An ASHA described a gap in their ability to talk to mothers in the field and suggested including more training content on efficient communication skills.

“We go on field and talk to mothers. There was no training for these, but I feel it will be good if we can have training on how to talk to mothers comfortably” (ASHA, WB)

Moore’s level 5—performance

The study identified a significant improvement in ASHAs’ positive attitude toward maternal and child health issues. Overall, a 5% improvement (95% CI − 0.009, 0.10; p value = 0.09) in participants’ attitudes post-ECHO training was found. Almost all the participants (99%) reported applying the skills learnt during the training at their workplaces. More than 90% of the participants felt that the ECHO training expanded access to healthcare in their community (Fig.  2 ). The ASHAs reported an improvement in their attitude towards inclusion of HIV patients in the community ( p =  0.01) and home visits for new born babies (p < 0.001) (Additional file 1 : Appendix S1, Table S4).

Self-reported performance of ASHAs

The ASHAs shared specific examples where they made changes in their practice or treatment strategies after attending the training.

“[Earlier] the implementation was not proper [correct]. As an example, if a child’s life has to be saved on the spot, we would take the medicines and syringes separately. Now we take the necessary items section wise including the AFI kit. So that’s the change” (ASHA, Tamil Nadu).

The results of this evaluation suggest that Project ECHO provides a suitable and efficacious platform for training for ASHAs. The participants reported an improvement in their knowledge, skills, and practices. They also described improved confidence to communicate more effectively. Some areas in which the ASHAs reported a lack of knowledge and confidence include newborn immunisation and family planning after pregnancy.

The NRHM guidelines for the recruitment of ASHAs require candidates to have at least eight or 10 completed years of formal education. Low literacy and inadequate training of ASHAs have been observed in different states in India [ 30 , 31 ]. However, with the proper training and support, ASHAs can provide comprehensive preventive and promotive healthcare services [ 29 ]. In this study, the majority (75%) of ASHAs across all states had ten or more years of schooling. The ECHO training will bolster their knowledge, skills, and confidence in providing effective services.

The ASHAs receive 23 days of training in the first year, followed by 12 days of training in every subsequent year to keep them updated with the knowledge and skills needed to effectively perform their roles and responsibilities. Previous studies have identified many challenges in the training of ASHAs, such as lack of regular refresher training [ 32 ], shortage of competent trainers, insufficient funds, and use of obsolete health information [ 33 ]. The training programs have mostly been didactic-based and had limitations in the engagement of participants [ 34 ]. The ECHO NHSRC refresher training addresses these limitations by promoting peer-to-peer learning and through a case-based discussion approach [ 35 ].

Our findings report a significant increase in the knowledge of ASHA workers with respect to specific domains like maternal and child health. A randomized controlled trial in Karnataka, India, found a significant improvement in mental health knowledge, attitude, and practice (KAP) scores amongst ASHAs trained by a hybrid training (traditional 1-day in-person classroom training and seven online sessions using the ECHO Model) against conventional classroom training [ 14 ]. This study findings highlight the improvement in knowledge of ASHAs related to oral health and palliative care post-ECHO training. An improvement in knowledge has also been observed in other studies that have evaluated ECHO telementoring interventions in cancer screening [ 36 ], palliative care [ 37 , 38 ], HIV [ 39 ], and chronic pain [ 40 ] In this study, ASHAs reported poor knowledge of the immunisation schedule for a newborn as well as the confidence to record and track immunisation in the community even after the ECHO training. A critical function of ASHAs is to assist ANMs or nurses with all immunisation activities [ 41 ]. A previous study in Karnataka in 2020 found inadequate knowledge among ASHAs about child immunisation. The above study also documented that by increasing the number of days and focusing on child care the ASHAs had a better understanding of interventions related to child healthcare [ 42 ]. As a part of the course structure, ECHO provides one session on new born and post-partum care. An assessment of the number of sessions needed to cover the topics was beyond the scope of our research but would be beneficial.

Previous studies have identified several shortcomings in ASHAs' communication and counselling abilities [ 43 , 44 , 45 ]. The findings of this study revealed that the ASHAs faced communication issues while discussing health matters related to family planning and COVID-19 with the community. Previous research has found that interpersonal communication of ASHAs are influenced by factors such as health system support and community context [ 46 ]. A study exploring the perspectives of ASHAs on a mobile training course in India also found that they encountered barriers in their interactions with beneficiaries such as resistance from family members, fear of poor quality of care, and financial costs of care [ 44 ]. Training programs must therefore, also incorporate how ASHAs can navigate social behaviours and norms to improve the impact of counselling [ 47 , 48 ]. The extent to which the ECHO training can identify and incorporate community hierarchies to improve communication of the ASHAs needs further exploration. In this study, large batch size ( n =  40) and limited use of video by participants during the training hampered the engagement between ASHAs as well as with the trainers. A previous study in the USA suggested that limiting batch size and ensuring face-to-face interactions on the virtual platform ensured a higher level of accountability and made it easier to engage with others in the ECHO training sessions [ 49 ].

CHWs face significant barriers when using digital technology in LMICs, making it challenging for them to access training on digital platforms [ 50 ]. The ASHAs in this study reported an improvement in their ability to use smartphones and navigate mobile applications. Our findings also suggest that ASHAs should be better oriented for accessing content on hand held devices.

The mentorship by trainers added value to participants’ knowledge and helped improve their skills. In this study, participants’ attitudes towards their work changed after attending the ECHO training suggesting that the learning and confidence developed during the training would be transferable to their work in healthcare settings and communities. The ECHO participants of previous studies have also demonstrated similar changes in their practices [ 35 , 40 ]. Our study findings indicate that the ECHO Model is an effective platform that can help foster a virtual community of practice through case-based learning, shared best practices, and online mentorship by experts.

Future directions

There should be more sessions on topics related to post-natal and newborn care as the ASHAs showed poor knowledge and competence in these areas.

There should be more training on counselling and development of communication skills for ASHAs, specially for maternal and child health and COVID-19.

An orientation for ASHAs should be conducted to facilitate the use of technology and the platform for learning. This may also help overcome some of the challenges described by the ASHAs in this study.

Strengths and limitations

The study used a rigorous quasi-experimental design across four different states of India. Our follow-up rate in the study was 80%, indicating a high response from participants completing the pre–post assessment. The presented study has certain limitations. It was not possible to use randomisation and a pure experimental design in this study, and this affects the internal validity of the study. The inclusion of a control group would have strengthened study validity. The self-reported outcomes can be subject to social desirability bias. We did not document the information on attendance and drop outs from the training program. The qualitative results have to be carefully interpreted because of the small sample size of the qualitative study relative to the study sample.

There is increasing recognition of the importance of CHWs globally for promoting a continuum of care and expanding access to health services. ASHA workers constitute critical human resources in the Indian health system and efforts to empower them are crucial for strengthening the health system at the community level. The encouraging results of this study indicate the effectiveness of Project ECHO in building the capacity of ASHA workers across different geographies in the country.

Availability of data and materials

All data generated or analyzed during this study are included in this published article (as Additional files).

Abbreviations

Community health workers

Sustainable development goals

National Rural Health Mission

Accredited social health activists

Digital infrastructure knowledge sharing

Ministry of Human Resource Development

Coronavirus Disease 2019

National Health System Resource Centre

World Health Organization

High-Income Countries

LMICs: Low- and Middle-Income Countries

Extension for Community Healthcare Outcomes

In-depth interviews

Key informant interviews

Continuing medical education

Institutional Ethics Committee

Participant Information Sheet

Jodhpur School of Public Health

Public Health Foundation of India

Hartzler AL, Tuzzio L, Hsu C, Wagner EH. Roles and functions of community health workers in primary care. Ann Fam Med. 2018;16(3):240–5.

Article   PubMed   PubMed Central   Google Scholar  

Olaniran A, Banke-Thomas A, Bar-Zeev S, Madaj B. Not knowing enough, not having enough, not feeling wanted: challenges of community health workers providing maternal and newborn services in Africa and Asia. PLoS ONE. 2022;17(9): e0274110.

Article   CAS   PubMed   PubMed Central   Google Scholar  

O’Donovan J, O’Donovan C, Kuhn I, Sachs SE, Winters N. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open. 2018;8(4): e021467.

Kok MC, Kane SS, Tulloch O, Ormel H, Theobald S, Dieleman M, et al. How does context influence performance of community health workers in low- and middle-income countries? Evidence from the literature. Health Res Policy Syst. 2015;13(1):13.

Saprii L, Richards E, Kokho P, Theobald S. Community health workers in rural India: analysing the opportunities and challenges accredited social health activists (ASHAs) face in realising their multiple roles. Hum Resour Health. 2015;13(1):95.

Ministry of Health and Family Welfare, Government of India. Non Communicable Disease Control Programmes: National Health Mission. 2023. https://nhm.gov.in/index1.php?lang=1&level=1&sublinkid=1041&lid=614 . Accessed 13 Feb 2023.

World Health Organization. What do we know about community health workers? A systematic review of existing reviews. 2020. https://www.who.int/publications-detail-redirect/what-do-we-know-about-community-health-workers-a-systematic-review-of-existing-reviews . Accessed 13 Feb 2023.

Bajpai N, Dholakia RH. Improving the performance of accredited social health activists in India. Mumbai: Columbia Global Centres South Asia; 2011.

Google Scholar  

Panwar DS, Naidu V, Das E, Verma S, Khan AA. Strengthening support mechanisms for accredited social health activists in order to improve home-based newborn care in Uttar Pradesh, India. BMC Proc. 2012;6(5):O33.

Article   PubMed Central   Google Scholar  

Yadav D, Singh P, Montague K, Kumar V, Sood D, Balaam M, Sharma D, Duggal M, Bartindale T, Varghese D, Olivier P. Sangoshthi: Empowering community health workers through peer learning in rural india. In: Proceedings of the 26th International Conference on World Wide Web 2017 Apr 3 (pp. 499–508).

Labrique AB, Wadhwani C, Williams KA, Lamptey P, Hesp C, Luk R, et al. Best practices in scaling digital health in low and middle income countries. Glob Health. 2018;14(1):103.

Article   Google Scholar  

Bashingwa JJH, Shah N, Mohan D, Scott K, Chamberlain S, Mulder N, et al. Examining the reach and exposure of a mobile phone-based training programme for frontline health workers (ASHAs) in 13 states across India. BMJ Glob Health. 2021;6(Suppl 5): e005299.

Winters N, Langer L, Nduku P, Robson J, O’Donovan J, Maulik P, et al. Using mobile technologies to support the training of community health workers in low-income and middle-income countries: mapping the evidence. BMJ Glob Health. 2019;4(4): e001421.

Nirisha PL, Malathesh BC, Kulal N, Harshithaa NR, Ibrahim FA, Suhas S, et al. Impact of technology driven mental health task-shifting for accredited social health activists (ASHAs): results from a randomised controlled trial of two methods of training. Commun Ment Health J. 2023;59(1):175–84.

Long LA, Pariyo G, Kallander K. Digital technologies for health workforce development in low- and middle-income countries: a scoping review. Glob Health Sci Pract. 2018;6(Supplement 1):S41–8.

Tyagi V, Khan A, Siddiqui S, KakraAbhilashi M, Dhurve P, Tugnawat D, et al. Development of a digital program for training community health workers in the detection and referral of schizophrenia in rural India. Psychiatr Q. 2023;94(2):141–63.

Article   PubMed   Google Scholar  

Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, et al. Outcomes of treatment for Hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199–207.

Article   CAS   PubMed   Google Scholar  

Ministry of Health and Family Welfare, Government of India. ASHA Training Modules: National Health Mission. 2022. http://nhm.gov.in/index1.php?lang=1&level=3&sublinkid=184&lid=257 . Accessed 11 Aug 2022.

Zurawski A, Komaromy M, Ceballos V, McAuley C, Arora S. Project ECHO brings innovation to community health worker training and support. J Health Care Poor Underserved. 2016;27(4):53–61.

Komaromy M, Ceballos V, Zurawski A, Bodenheimer T, Thom DH, Arora S. Extension for community healthcare outcomes (ECHO): a new model for community health worker training and support. J Public Health Policy. 2018;39(2):203–16.

Damian AJ, Robinson S, Manzoor F, Lamb M, Rojas A, Porto A, et al. A mixed methods evaluation of the feasibility, acceptability, and impact of a pilot project ECHO for community health workers (CHWs). Pilot Feasibility Stud. 2020;6(1):132.

Feroz AS, Khoja A, Saleem S. Equipping community health workers with digital tools for pandemic response in LMICs. Arch Public Health. 2021;79(1):1.

Creswell JW, Clark VLP. Designing and conducting mixed methods research. Thousand Oaks: SAGE Publications; 2017. p. 521.

Tran L, Feldman R, Riley T III, Jung J. Association of the extension for community healthcare outcomes project with use of direct-acting antiviral treatment among US adults with hepatitis C. JAMA Netw Open. 2021;4(7): e2115523.

Harpe SE. How to analyze Likert and other rating scale data. Curr Pharm Teach Learn. 2015;7(6):836–50.

Hackett A, Strickland K. Using the framework approach to analyse qualitative data: a worked example. Nurse Res. 2018;26(2):8.

Moore DEJ, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1.

Chokshi M, Patil B, Khanna R, Neogi SB, Sharma J, Paul VK, et al. Health systems in India. J Perinatol. 2016;36(Suppl 3):S9-12.

National health systems resource centre. ASHA: which way forward. Evaluation of the ASHA Programme. 2010. https://nhm.gov.in/images/pdf/communitisation/asha/Studies/Evaluation_of_ASHA_Program_2010-11_Report.pdf . Accessed 17 Dec 2022.

National health systems resource centre. Tenth common review mission: National health mission. https://nhm.gov.in/images/pdf/monitoring/crm/10th-crm/Report/10th_CRM_Main_Report.pdf . Accessed 29 Dec 2022.

DeRenzi B, Wacksman J, Dell N, Lee S, Lesh N, Borriello G, Ellner A. Closing the feedback loop: a 12-month evaluation of ASTA, a self-tracking application for ASHAs. In: DeRenzi B, editor. Proceedings of the Eighth international conference on information and communication technologies and development. New York: Association for Computing Machinery; 2016. p. 1–10. https://doi.org/10.1145/2909609.2909652 .

Chapter   Google Scholar  

Yadav D. Low-cost mobile learning solutions for community health workers. In: Yadav D, editor. Proceedings of the 26th international on world wide web companion. Geneva: International World Wide Web Conferences Steering Committee; 2017. p. 729–34. https://doi.org/10.1145/3041021.3053377 .

Molapo M, Marsden G. Health education in rural communities with locally produced and locally relevant multimedia content. In: Molapo M, editor. Proceedings of the 3rd ACM symposium on computing for development. New York: Association for Computing Machinery; 2013. p. 1–2. https://doi.org/10.1145/2442882.2442913 .

Bhowmick S, Sorathia K. Findings of the user study conducted to understand the training of rural ASHAs in India. In: Bhowmick S, editor. Proceedings of the tenth international conference on information and communication technologies and development. New York: Association for Computing Machinery; 2019. p. 1–5. https://doi.org/10.1145/3287098.3287150 .

Bikinesi L, O’Bryan G, Roscoe C, Mekonen T, Shoopala N, Mengistu AT, et al. Implementation and evaluation of a Project ECHO telementoring program for the Namibian HIV workforce. Hum Resour Health. 2020;18(1):61.

Adsul P, Nethan ST, deCortina SH, Dhanasekaran K, Hariprasad R. Implementing cancer screening programs by training primary care physicians in India—findings from the national institute of cancer prevention research project ECHO for cancer prevention. Glob Implement Res Appl. 2022;2(1):34–41.

White C, McIlfatrick S, Dunwoody L, Watson M. Supporting and improving community health services—a prospective evaluation of ECHO technology in community palliative care nursing teams. BMJ Support Palliat Care. 2019;9(2):202–8.

Usher R, Payne C, Real S, Carey L. Project ECHO: Enhancing palliative care for primary care occupational therapists and physiotherapists in Ireland. Health Soc Care Commun. 2022;30(3):1143–53.

Wood BR, Unruh KT, Martinez-Paz N, Annese M, Ramers CB, Harrington RD, et al. Impact of a telehealth program that delivers remote consultation and longitudinal mentorship to community HIV providers. Open Forum Infect Dis. 2016;3(3):123.

Katzman JG, Comerci GJ, Boyle JF, Duhigg D, Shelley B, Olivas C, et al. Innovative telementoring for pain management: project ECHO pain. J Contin Educ Health Prof. 2014;34(1):68–75.

Kalne PS, Kalne PS, Mehendale AM. Acknowledging the role of community health workers in providing essential healthcare services in rural india—a review. Cureus. 2022;14(9): e29372.

PubMed   PubMed Central   Google Scholar  

Rohith M, Angadi MM. Evaluation of knowledge and practice of ASHAs, regarding child health services in Vijyapaura district, Karnataka. J Fam Med Prim Care. 2020;9(7):3272–6.

Article   CAS   Google Scholar  

Shrivastava A, Srivastava A. Measuring communication competence and effectiveness of ASHAs (accredited social health activist) in their leadership role at rural settings of Uttar Pradesh (India). Leadersh Health Serv. 2016;29(1):69–81.

Scott K, Ummer O, Chamberlain S, Sharma M, Gharai D, Mishra B, et al. ’[We] learned how to speak with love’: a qualitative exploration of accredited social health activist (ASHA) community health worker experiences of the Mobile Academy refresher training in Rajasthan, India. BMJ Open. 2022;12(6): e050363.

Goel AD, Gosain M, Amarchand R, Sharma H, Rai S, Kapoor SK, et al. Effectiveness of a quality improvement program using difference-in-difference analysis for home based newborn care—results of a community intervention trial. Indian J Pediatr. 2019;86(11):1028–35.

Ved R, Scott K. Counseling is a relationship not just a skill: re-conceptualizing health behavior change communication by India’s accredited social health activists. Glob Health Sci Pract. 2020;8(3):332–4.

Abdel-All M, Putica B, Praveen D, Abimbola S, Joshi R. Effectiveness of community health worker training programmes for cardiovascular disease management in low-income and middle-income countries: a systematic review. BMJ Open. 2017;7(11): e015529.

Smittenaar P, Ramesh BM, Jain M, Blanchard J, Kemp H, Engl E, et al. Bringing greater precision to interactions between community health workers and households to improve maternal and newborn health outcomes in India. Glob Health Sci Pract. 2020;8(3):358–71.

Shimasaki S, Bishop E, Guthrie M, Thomas JF. Strengthening the health workforce through the ECHO stages of participation: participants’ perspectives on key facilitators and barriers. J Med Educ Curric Dev. 2019;6:2382120518820922.

Medhanyie AA, Moser A, Spigt M, Yebyo H, Little A, Dinant G, et al. Mobile health data collection at primary health care in Ethiopia: a feasible challenge. J Clin Epidemiol. 2015;68(1):80–6.

Download references

Acknowledgements

The authors wish to thank all the healthcare workers who kindly participated in this study giving their time, experience, and insights. We also thank Dr. Sourabh Chakraborty (Professor, JSPH), Mr. Swapnil Gupta, and the JSPH data collection team for their contribution to the collection of good quality data in a short time.

The study was funded by Extension for Community Healthcare Outcomes (ECHO) India.

Author information

Authors and affiliations.

Public Health Foundation of India (PHFI), Gurugram, Haryana, India

Rajmohan Panda

Extension for Community Healthcare Outcomes (ECHO) India, Okhla Phase III, New Delhi, India

Supriya Lahoti, Nivedita Mishra, Apoorva Karan Rai & Kumud Rai

HexaHealth, Gurugram, Haryana, India

Rajath R. Prabhu

Hamad Medical Corporation, Doha, Qatar

Kalpana Singh

You can also search for this author in PubMed   Google Scholar

Contributions

R.M. contributed to the conception and design of the study and significant inputs for data analysis and made a significant contribution to the drafting of the discussion and conclusion of the paper. S.L. wrote the first draft of the manuscript. N.M. and S.L. contributed to the implementation of the study and development of interview guides, analysis, and validation of qualitative data. R.P. and K.S. contributed to the analysis and validation of quantitative data. R.M., N.M., R.P., K.S, A.K.R. and K.R. reviewed the manuscript and gave significant inputs for improving the paper. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Supriya Lahoti .

Ethics declarations

Ethics approval and consent to participate.

Ethical clearance was received from the Institutional Ethical Committee (IEC) of the Public Health Foundation of India (PHFI) (ref: TRC-IEC 472/21, dated 26 August 2021). The study has also been registered at the Clinical Trials Registry, India (CTRI/2021/10/037189). All methods were performed in accordance with the relevant guidelines and regulations. A written Participant Information Sheet (PIS) and informed consent form was provided to the participants before conducting the interviews. Verbal informed consent was taken from all participants, and the process of verbal informed consent was approved by the ethics committee (Institutional Ethics Committee (IEC) of the PHFI). Data confidentiality was maintained by coding with the unique identification (ID) of all the participants. The interviews were audio-recorded, and audio files and transcripts were kept in a password-protected folder.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

: Appendix S1. Table S1.1. Satisfaction with different factors of the training. Table S1.2. Satisfaction with content and environment of the training. Table S1.3. Challenges faced with respect to ECHO tele-mentoring model. Table S2. Technical knowledge and skills. Table S3. Statements assessing competence. Table S4. Statements assessing attitude and performance.

Additional file 2

: Appendix S2. Participants in qualitative interviews.

Additional file 3

: Appendix S3. Key informant Interview Guide for Trainers End line Evaluation.

Additional file 4

: Appendix S4. Key informant interview guide for Hub leaders End line Evaluation.

Additional file 5

: Appendix S5. In-depth Interview Guide for ASHAs End line Evaluation.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Panda, R., Lahoti, S., Mishra, N. et al. A mixed methods evaluation of the impact of ECHO ® telementoring model for capacity building of community health workers in India. Hum Resour Health 22 , 26 (2024). https://doi.org/10.1186/s12960-024-00907-y

Download citation

Received : 14 March 2023

Accepted : 10 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12960-024-00907-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Community health workers (CHWs)
• Accredited social health activists (ASHAs)
• Maternal and child health
• Primary healthcare
• Health worker training
• ECHO telementoring
• Mixed-method study

Human Resources for Health

ISSN: 1478-4491

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Systematic Review
• Open access
• Published: 26 April 2024

Systematic review on the frequency and quality of reporting patient and public involvement in patient safety research

• Sahar Hammoud   ORCID: orcid.org/0000-0003-4682-9001 1 ,
• Laith Alsabek 1 , 2 ,
• Lisa Rogers 1 &
• Eilish McAuliffe 1  

BMC Health Services Research volume  24 , Article number:  532 ( 2024 ) Cite this article

Metrics details

In recent years, patient and public involvement (PPI) in research has significantly increased; however, the reporting of PPI remains poor. The Guidance for Reporting Involvement of Patients and the Public (GRIPP2) was developed to enhance the quality and consistency of PPI reporting. The objective of this systematic review is to identify the frequency and quality of PPI reporting in patient safety (PS) research using the GRIPP2 checklist.

Searches were performed in Ovid MEDLINE, EMBASE, PsycINFO, and CINAHL from 2018 to December, 2023. Studies on PPI in PS research were included. We included empirical qualitative, quantitative, mixed methods, and case studies. Only articles published in peer-reviewed journals in English were included. The quality of PPI reporting was assessed using the short form of the (GRIPP2-SF) checklist.

A total of 8561 studies were retrieved from database searches, updates, and reference checks, of which 82 met the eligibility criteria and were included in this review. Major PS topics were related to medication safety, general PS, and fall prevention. Patient representatives, advocates, patient advisory groups, patients, service users, and health consumers were the most involved. The main involvement across the studies was in commenting on or developing research materials. Only 6.1% ( n  = 5) of the studies reported PPI as per the GRIPP2 checklist. Regarding the quality of reporting following the GRIPP2-SF criteria, our findings show sub-optimal reporting mainly due to failures in: critically reflecting on PPI in the study; reporting the aim of PPI in the study; and reporting the extent to which PPI influenced the study overall.

Conclusions

Our review shows a low frequency of PPI reporting in PS research using the GRIPP2 checklist. Furthermore, it reveals a sub-optimal quality in PPI reporting following GRIPP2-SF items. Researchers, funders, publishers, and journals need to promote consistent and transparent PPI reporting following internationally developed reporting guidelines such as the GRIPP2. Evidence-based guidelines for reporting PPI should be encouraged and supported as it helps future researchers to plan and report PPI more effectively.

Trial registration

The review protocol is registered with PROSPERO (CRD42023450715).

Peer Review reports

Patient safety (PS) is defined as “the absence of preventable harm to a patient and reduction of risk of unnecessary harm associated with healthcare to an acceptable minimum” [ 1 ]. It is estimated that one in 10 patients are harmed in healthcare settings due to unsafe care, resulting in over three million deaths annually [ 2 ]. More than 50% of adverse events are preventable, and half of these events are related to medications [ 3 , 4 ]. There are various types of adverse events that patients can experience such as medication errors, patient falls, healthcare-associated infections, diagnostic errors, pressure ulcers, unsafe surgical procedures, patient misidentification, and others [ 1 ].

Over the last few decades, the approach of PS management has shifted toward actively involving patients and their families in managing PS. This innovative approach has surpassed the traditional model where healthcare providers were the sole managers of PS [ 5 ]. Recent research has shown that patients have a vital role in promoting their safety and decreasing the occurrence of adverse events [ 6 ]. Hence, there is a growing recognition of patient and family involvement as a promising method to enhance PS [ 7 ]. This approach includes involving patients in PS policy development, research, and shared decision making [ 1 ].

In the last decade, research involving patients and the public has significantly increased. In the United Kingdom (U.K), the National Institute for Health Research (NIHR) has played a critical role in providing strategic and infrastructure support to integrate Public and Patient Involvement (PPI) throughout publicly funded research [ 8 ]. This has established a context where PPI is recognised as an essential element in research [ 9 ]. In Ireland, the national government agency responsible for the management and delivery of all public health and social services; the National Health Service Executive (HSE) emphasise the importance of PPI in research and provide guidance for researchers on how to involve patients and public in all parts of the research cycle and knowledge translation process [ 10 ]. Similar initiatives are also developing among other European countries, North America, and Australia. However, despite this significant expansion of PPI research, the reporting of PPI in research articles continues to be sub-optimal, inconsistent, and lacks essential information on the context, process, and impact of PPI [ 9 ]. To address this problem, the Guidance for Reporting Involvement of Patients and the Public (GRIPP) was developed in 2011 following the EQUATOR methodology to enhance the quality, consistency, and transparency of PPI reporting. Additionally, to provide guidance for researchers, patients, and the public to advance the quality of the international PPI evidence-base [ 11 ]. The first GRIPP checklist was a significant start in producing higher-quality PPI reporting; however, it was developed following a systematic review, and did not include any input from the international PPI research community. Given the importance of reaching consensus in generating current reporting guidelines, a second version of the GRIPP checklist (GRIPP2) was developed to tackle this problem by involving the international PPI community in its development [ 9 ]. There are two versions of the GRIPP2 checklist, a long form (GRIPP2-LF) for studies with PPI as the primary focus, and a short form (GRIPP2-SF) for studies with PPI as secondary or tertiary focus.

Since the publication of the GRIPP2 checklist, several systematic reviews have been conducted to assess the quality of PPI reporting on various topics. For instance, Bergin et al. in their review to investigate the nature and impact of PPI in cancer research, reported a sub-optimal quality of PPI reporting using the GRIPP2-SF, mainly due to failure to address PPI challenges [ 12 ]. Similarly, Owyang et al. in their systematic review to assess the prevalence, extent, and quality of PPI in orthopaedic practice, described a poor PPI reporting following the GRIPP2-SF checklist criteria [ 13 ]. While a few systematic reviews have been conducted to assess theories, strategies, types of interventions, and barriers and enablers of PPI in PS [ 5 , 14 , 15 , 16 ], no previous review has assessed the quality of PPI reporting in PS research. Thus, our systematic review aims to address this knowledge gap. The objective of this review is to identify the frequency PPI reporting in PS research using the GRIPP2 checklist from 2018 (the year after GRIPP2 was published) and the quality of reporting following the GRIPP2-SF. The GRIPP2 checklist was chosen as the benchmark as it is the first international, evidence-based, community consensus informed guideline for the reporting of PPI in research and more specifically in health and social care research [ 9 ]. Additionally, it is the most recent report-focused framework and the most recommended by several leading journals [ 17 ].

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to plan and report this review [ 18 ]. The review protocol was published on PROSPERO the International Database of Prospectively Registered Systematic Reviews in August 2023 (CRD42023450715).

Search strategy

For this review, we used the PICo framework to define the key elements in our research. These included articles on patients and public (P-Population) involvement (I- phenomenon of Interest) in PS (C-context). Details are presented in Table  1 . Four databases were searched including Ovid MEDLINE, EMBASE, PsycINFO, and CINAHL to identify papers on PPI in PS research. A systematic search strategy was initially developed using MEDLINE. MeSH terms and keywords relevant to specific categories (e.g., patient safety) were combined using the “OR” Boolean term (i.e. patient safety OR adverse event OR medical error OR surgical error) and categories were then combined using the “AND” Boolean term. (i.e. “patient and public involvement” AND “patient safety”). The search strategy was adapted for the other three databases. Full search strategies are provided in Supplementary file 1 . The search was conducted on July 27th, 2023, and was limited to papers published from 2018. As the GRIPP2 tool was published in 2017, this limit ensured the retrieval of relevant studies. An alert system was set on the four databases to receive all new published studies until December 2023, prior to the final analysis. The search was conducted without restrictions on study type, research design, and language. To reduce selection bias, hand searching was carried out on the reference lists of all the eligible articles in the later stages of the review. This was done by the first author. The search strategy was developed by the first author and confirmed by the research team and a Librarian. The database search was conducted by the first author.

Inclusion and exclusion criteria

Studies on PPI in PS research with a focus on health/healthcare were included in this review. We defined PPI as active involvement which is in line with the NIHR INVOLVE definition as “research being carried out ‘with’ or ‘by’ members of the public rather than ‘to’, ‘about’ or ‘for’ them” [ 19 ]. This includes any PPI including, being a co-applicant on a research project or grant application, identifying research priorities, being a member of an advisory or steering group, participating in developing research materials or giving feedback on them, conducting interviews with study participants, participating in recruitment, data collection, data analysis, drafting manuscripts and/or dissemination of results. Accordingly, we excluded studies where patients or the public were only involved as research participants.

We defined patients and public to include patients, relatives, carers, caregivers and community, which is also in line with the NIHR PPI involvement in National Health Service [ 19 ].

Patient safety included topics on medication safety, adverse events, communication, safety culture, diagnostic errors, and others. A full list of the used terms for PPI and PS is provided in Supplementary file 1 . Regarding the research type and design, we included empirical qualitative, quantitative, mixed methods, and case studies. Only articles published in peer-reviewed journals and in English were included.

Any article that did not meet the inclusion criteria was excluded. Studies not reporting outcomes were excluded. Furthermore, review papers, conference abstracts, letters to editor, commentary, viewpoints, and short communications were excluded. Finally, papers published prior to 2018 were excluded.

Study selection

The selection of eligible studies was done by the first and the second authors independently, starting with title and abstracts screening to eliminate papers that failed to meet our inclusion criteria. Then, full text screening was conducted to decide on the final included papers in this review. Covidence, an online data management system supported the review process, ensuring reviewers were blinded to each other’s decisions. Disagreements between reviewers were discussed first, in cases where the disagreement was not resolved, the fourth author was consulted.

Data extraction and analysis

A data extraction sheet was developed using excel then piloted, discussed with the research team and modified as appropriate. The following data were extracted: citation and year of publication, objective of the study, country, PS topic, design, setting, PPI participants, PPI stages (identifying research priorities, being a member of an advisory or steering group, etc.…), frequency of PPI reporting as per the GRIPP2 checklist, and the availability of a plain language summary. Additionally, data against the five items of GRIPP2-SF (aim of PPI in the study, methods used for PPI, outcomes of PPI including the results and the extent to which PPI influenced the study overall, and reflections on PPI) were extracted. To avoid multiple publication bias and missing outcomes, data extraction was done by the first and the second authors independently and then compared. Disagreements between reviewers were first discussed, and then resolved by the third and fourth authors if needed.

Quality assessment

The quality of PPI reporting was assessed using GRIPP2-SF developed by Staniszewska et al. [ 9 ] as it is developed to improve the quality, consistency, and reporting of PPI in social and healthcare research. Additionally the GRIPP2-SF is suitable for all studies regardless of whether PPI is the primary, secondary, or tertiary focus, whereas the GRIPP2-LF is not suitable for studies where PPI serves as a secondary or tertiary focus. The checklist includes five items (mentioned above) that authors should include in their studies. It is important to mention that Staniszewska et al. noted that “while GRIPP2-SF aims to guide consistent reporting, it is not possible to be prescriptive about the exact content of each item, as the current evidence-base is not advanced enough to make this possible” ([ 9 ] p5). For that reason, we had to develop criteria for scoring the five reporting items. We used three scoring as Yes, No, and partial for each of the five items of the GRIPP2-SF. Yes, was given when authors presented PPI information on the item clearly in the paper. No, when no information was provided, and partial when the information partially met the item requirement. For example, as per GRIPP2-SF authors should provide a clear description of the methods used for PPI in the study. In the example given by Staniszewska et al., information on patient/public partners and how many of them were provided, as well as the stages of the study they were involved in (i.e. refining the focus of the research questions, developing the search strategy, interpreting results). Thus, in our evaluation of the included studies, we gave a yes if information on PPI participants (i.e. patient partners, community partners, or family members etc..) and how many of them were involved was provided, and information on the stages or actions of their involvement in the study was provided. However, we gave a “partial” if information was not fully provided (i.e. information on patient/public partners and how many were involved in the study without describing in what stages or actions they were involved, and vice versa), and a “No” if no information was presented at all.

The quality of PPI reporting was done by the first and the second authors independently and then compared. Disagreements between reviewers were first discussed, and then resolved by the third and fourth author when needed.

Assessing the quality or risk of bias of the included studies was omitted, as the focus in this review was on appraising the quality of PPI reporting rather than assessing the quality of each research article.

Data synthesis

After data extraction, a table summarising the included studies was developed. Studies were compared according to the main outcomes of the review; frequency of PPI reporting following the GRIPP2 checklist and the quality of reporting as per GRIPP2-SF five items, and the availability of a plain language summary.

Search results and study selection

The database searches yielded a total of 8491 studies. First, 2496 were removed as duplicates. Then, after title and abstract screening, 5785 articles were excluded leaving 210 articles eligible for the full text review. After a careful examination, 68 of these studies were included in this review. A further 38 studies were identified from the alert system that was set on the four databases and 32 studies from the reference check of the included studies. Of these 70 articles, 56 were further excluded and 14 were added to the previous 68 included studies. Thus, 82 studies met the inclusion criteria and were included in this review. A summary of the database search results and the study selection process are presented in Fig.  1 .

PRISMA flow diagram of the study selection process. The PRISMA flow diagram details the review search results and selection process

Overview of included studies

Details of the study characteristics including first author and year of publication, objective, country, study design, setting, PS topic, PPI participants and involvement stages are presented in Supplementary file 2 . The majority of the studies were conducted in the U.K ( n  = 24) and the United States of America ( n  = 18), with the remaining 39 conducted in other high income countries, the exception being one study in Haiti. A range of study designs were identified, the most common being qualitative ( n  = 31), mixed methods ( n  = 13), interventional ( n  = 5), and quality improvement projects ( n  = 4). Most PS topics concerned medication safety ( n  = 17), PS in general (e.g., developing a PS survey or PS management application) ( n  = 14), fall prevention ( n  = 13), communication ( n  = 11), and adverse events ( n  = 10), with the remaining PS topics listed in Supplementary file 2 .

Patient representatives, advocates, and patient advisory groups ( n  = 33) and patients, service users, and health consumers ( n  = 32) were the main groups involved. The remaining, included community members/ organisations. Concerning PPI stages, the main involvement across the studies was in commenting on or developing research materials ( n  = 74) including, patient leaflets, interventional tools, mobile applications, and survey instruments. Following this stage, involvement in data analysis, drafting manuscripts, and disseminating results ( n  = 30), and being a member of a project advisory or steering group ( n  = 18) were the most common PPI evident in included studies. Whereas the least involvement was in identifying research priorities ( n  = 5), and being a co-applicant on a research project or grant application ( n  = 6).

Regarding plain language summary, only one out of the 82 studies (1.22%) provided a plain language summary in their paper [ 20 ].

Frequency and quality of PPI reporting

The frequency of PPI reporting following the GRIPP2 checklist was 6.1%, where only five of the 82 included studies reported PPI in their papers following the GRIPP2 checklist. The quality of PPI reporting in those studies is presented in Table  2 . Of these five studies, one study (20%) did not report the aim of PPI in the study and one (20%) did not comment on the extent to which PPI influenced the study overall.

The quality of PPI reporting of the remaining 77 studies is presented in Table  3 . The aim of PPI in the study was reported in 62.3% of articles ( n  = 48), while 3.9% ( n  = 3) partially reported this. A clear description of the methods used for PPI in the study was reported in 79.2% of papers ( n  = 61) and partially in 20.8% ( n  = 16). Concerning the outcomes, 81.8% of papers ( n  = 63) reported the results of PPI in the study, while 10.4% ( n  = 8) partially did. Of the 77 studies, 68.8% ( n  = 53) reported the extent to which PPI influenced the study overall and 3.9% ( n  = 3) partially reported this. Finally, 57.1% ( n  = 44) of papers critically reflected on the things that went well and those that did not and 2.6% ( n  = 2) partially reflected on this.

Summary of main findings

This systematic review assessed the frequency of reporting PPI in PS research using the GRIPP2 checklist and quality of reporting using the GRIPP2-SF. In total, 82 studies were included in this review. Major PS topics were related to medication safety, general PS, and fall prevention. Patient representatives, advocates, patient advisory groups, patients, service users, and health consumers were the most involved. The main involvement across the studies was in commenting on or developing research materials such as educational and interventional tools, survey instruments, and applications while the least was in identifying research priorities and being a co-applicant on a research project or grant application. Thus, significant effort is still needed to involve patients and the public in the earlier stages of the research process given the fundamental impact of PS on their lives.

Overall completeness and applicability of evidence

A low frequency of reporting PPI in PS research following the GRIPP2 guidelines was revealed in this review, where only five of the 82 studies included mentioned that PPI was reported as per the GRIPP2 checklist. This is despite it being the most recent report-focused framework and the most recommended by several leading journals [ 17 ]. This was not surprising as similar results were reported in recent reviews in other healthcare topics. For instance, Musbahi et al. in their systematic review on PPI reporting in bariatric research reported that none of the 90 papers identified in their review mentioned or utilised the GRIPP2 checklist [ 102 ]. Similarly, a study on PPI in orthodontic research found that none of the 363 included articles reported PPI against the GRIPP2 checklist [ 103 ].

In relation to the quality of reporting following the GRIPP2-SF criteria, our findings show sub-optimal reporting within the 77 studies that did not use GRIPP2 as a guide/checklist to report their PPI. Similarly, Bergin et al. in their systematic review to investigate the nature and impact of PPI in cancer research concluded that substandard reporting was evident [ 12 ]. In our review, this was mainly due to failure to meet three criteria. First, the lowest percentage of reporting (57.1%, n  = 44) was related to critical reflection on PPI in the study (i.e., what went well and what did not). In total, 31 studies (42.9%) did not provide any information on this, and two studies were scored as partial. The first study mentioned that only involving one patient was a limitation [ 27 ] and the other stated that including three patients in the design of the tool was a strength [ 83 ]. Both studies did not critically comment or reflect on these points so that future researchers are able to avoid such problems and enhance PPI opportunities. For instance, providing the reasons/challenges behind the exclusive inclusion of a single patient and explaining how this limits the study findings and conclusion would help future researchers to address these challenges. Likewise, commenting on why incorporating three patients in the design of the study tool could be seen as a strength would have been beneficial. This could be, fostering diverse perspectives and generating novel ideas for developing the tool. Similar to our findings, Bergin et al. in their systematic review reported that 40% of the studies failed to meet this criterion [ 12 ].

Second, only 48 out of 77 articles (62.3%) reported the aim of PPI in their study, which is unlike the results of Bergin et al. where most of the studies (93.1%) in their review met this criterion [ 12 ]. Of the 29 studies which did not meet this criterion in our review, few mentioned in their objective developing a consensus-based instrument [ 41 ], reaching a consensus on the patient-reported outcomes [ 32 ], obtaining international consensus on a set of core outcome measures [ 98 ], and facilitating a multi-stakeholder dialogue [ 71 ] yet, without indicating anything in relation to patients, patient representatives, community members, or any other PPI participants. Thus, the lack of reporting the aim of PPI was clearly evident in this review. Reporting the aim of PPI in the study is crucial for promoting transparency, methodological rigor, reproducibility, and impact assessment of the PPI.

Third, 68.8% ( n  = 53) of the studies reported the extent to which PPI influenced the study overall including positive and negative effects if any. This was again similar to the findings of Bergin et al., where 38% of the studies did not meet this criterion mainly due to a failure to address PPI challenges in their respective studies [ 12 ]. Additionally, Owyang et al. in their review on the extent, and quality of PPI in orthopaedic practice, also described a poor reporting of PPI impact on research [ 13 ]. As per the GRIPP2 guidelines, both positive and negative effects of PPI on the study should be reported when applicable. Providing such information is essential as it enhances future research on PPI in terms of both practice and reporting.

Reporting a clear description of the methods used for PPI in the study was acceptable, with 79.2% of the papers meeting this criterion. Most studies provided information in the methods section of their papers on the PPI participants, their number, stages of their involvement and how they were involved. Providing clear information on the methods used for PPI is vital to give the reader a clear understanding of the steps taken to involve patients, and for other researchers to replicate these methods in future research. Additionally, reporting the results of PPI in the study was also acceptable with 81.8% of the papers reporting the outcomes of PPI in the results section. Reporting the results of PPI is important for enhancing methodological transparency, providing a more accurate interpretation for the study findings, contributing to the overall accountability and credibility of the research, and informing decision making.

Out of the 82 studies included in this review, only one study provided a plain language summary. We understand that PS research or health and medical research in general is difficult for patients and the public to understand given their diverse health literacy and educational backgrounds. However, if we expect patients and the public to be involved in research then, it is crucial to translate this research that has a huge impact on their lives into an easily accessible format. Failing to translate the benefits that such research may have on patient and public lives may result in them underestimating the value of this research and losing interest in being involved in the planning or implementation of future research [ 103 ]. Thus, providing a plain language summary for research is one way to tackle this problem. To our knowledge, only a few health and social care journals (i.e. Cochrane and BMC Research Involvement and Engagement) necessitate a plain language summary as a submission requirement. Having this as a requirement for submission is crucial in bringing the importance of this issue to researchers’ attention.

Research from recent years suggests that poor PPI reporting in articles relates to a lack of submission requirements for PPI reporting in journals and difficulties with word limits for submitted manuscripts [ 13 ]. Price et al. assessed the frequency of PPI reporting in published papers before and after the introduction of PPI reporting obligations by the British Medical Journal (BMJ) [ 104 ]. The authors identified an increase in PPI reporting in papers published by BMJ from 0.5% to 11% between the periods of 2013–2014 and 2015–2016. The study findings demonstrate the impact of journal guidelines in shaping higher quality research outputs [ 13 ]. In our review, we found a low frequency of PPI reporting in PS research using the GRIPP2 checklist, alongside sub-optimal quality of reporting following GRIPP2-SF. This could potentially be attributed to the absence of submission requirements for PPI reporting in journals following the GRIPP2 checklist, as well as challenges posed by word limits.

Strengths and limitations

This systematic review presents an overview on the frequency of PPI reporting in PS research using the GRIPP2 checklist, as well as an evaluation of the quality of reporting following the GRIPP2-SF. As the first review to focus on PS research, it provides useful knowledge on the status of PPI reporting in this field, and the extent to which researchers are adopting and adhering to PPI reporting guidelines. Despite these strengths, our review has some limitations that should be mentioned. First, only English language papers were included in this review due to being the main language of the researchers. Thus, there is a possibility that relevant articles on PPI in PS research may have been omitted. Another limitation is related to our search which was limited to papers published starting 2018 as the GRIPP2 guidelines were published in 2017. Thus it is probable that the protocols of some of these studies were developed earlier than the publication of the GRIPP2 checklist, meaning that PPI reporting following GRIPP2 was not common practice and thus not adopted by these studies. This might limit the conclusions we can draw from this review. Finally, the use of GRIPP2 to assess the quality of PPI reporting might be a limitation as usability testing has not yet been conducted to understand how the checklist works in practice with various types of research designs. However, the GRIPP2 is the first international, evidence-based, community consensus informed guideline for the reporting of PPI in health and social care research. Reflections and comments from researchers using the GRIPP2 will help improve its use in future studies.

Implications for research and practice

Lack of PPI reporting not only affects the quality of research but also implies that others cannot learn from previous research experience. Additionally, without consistent and transparent reporting it is difficult to evaluate the impact of various PPI in research [ 9 ]: “if it is not reported it cannot be assessed” ([ 105 ] p19). Enhanced PPI reporting will result in a wider range and richer high-quality evidence-based PPI research, leading to a better understanding of PPI use and effectiveness [ 103 ]. GRIPP2 reporting guidelines were developed to provide guidance for researchers, patients, and the public to enhance the quality of PPI reporting and improve the quality of the international PPI evidence-base. The guidance can be used prospectively to plan PPI or retrospectively to guide the structure or PPI reporting in research [ 9 ]. To enhance PPI reporting, we recommend the following;

Publishers and journals

First, we encourage publishers and journals to require researchers to report PPI following the GRIPP2 checklist. Utilising the short or the long version should depend on the primary focus of the study (i.e., if PPI is within the primary focus of the research then the GRIPP2-LF is recommended). Second, we recommend that journals and editorial members advise reviewers to evaluate PPI reporting within research articles following the GRIPP2 tool and make suggestions accordingly. Finally, we encourage journals to add a plain language summary as a submission requirement to increase research dissemination and improve the accessibility of research for patients and the public.

Researchers

Though there is greater evidence of PPI in research, it is still primarily the researchers that are setting the research agenda and deciding on the research questions to be addressed. Thus, significant effort is still needed to involve patients and the public in the earlier stages of the research process given the fundamental impact of PS on their lives. To enhance future PPI reporting, perhaps adding a criterion following the GRIPP2 tool to existing EQUATOR checklists for reporting research papers such as STROBE, PRISMA, CONSORT, may support higher quality research. Additionally, currently, there is no detailed explanation paper for the GRIPP2 where each criterion is explained in detail with examples. Addressing this gap would be of great benefit to guide the structure of PPI reporting and to explore the applicability of each criterion in relation to different stages of PPI in research. For instance, having a detailed explanation for each criterion across different research studies having various PPI stages would be of high value to improve future PPI reporting given the growing interest in PPI research in recent years and the relatively small PPI evidence base in health and medical research.

Funding bodies can also enhance PPI reporting by adding a requirement for researchers to report PPI following the GRIPP2 checklist. In Ireland, the National HSE has already initiated this by requiring all PPI in HSE research in Ireland to be reported following the GRIPP2 guidelines [ 10 ].

This study represents the first systematic review on the frequency and quality of PPI reporting in PS research using the GRIPP2 checklist. Most PS topics were related to medication safety, general PS, and fall prevention. The main involvement across the studies was in commenting on or developing research materials. Thus, efforts are still needed to involve patients and the public across all aspects of the research process, especially earlier stages of the research cycle. The frequency of PPI reporting following the GRIPP2 guidelines was low, and the quality of reporting following the GRIPP2-SF criteria was sub-optimal. The lowest percentages of reporting were on critically reflecting on PPI in the study so future research can learn from this experience and work to improve it, reporting the aim of the PPI in the study, and reporting the extent to which PPI influenced the study overall including positive and negative effects. Researchers, funders, publishers, journals, editorial members and reviewers have a responsibility to promote consistent and transparent PPI reporting following internationally developed reporting guidelines such as the GRIPP2. Evidence-based guidelines for reporting PPI should be supported to help future researchers plan and report PPI more effectively, which may ultimately improve the quality and relevance of research.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its Supplementary information files.

Abbreviations

• Patient safety

United Kingdom

National Institute for Health Research

Public and Patient Involvement

Health Service Executive

Guidance for Reporting Involvement of Patients and the Public

Second version of the GRIPP checklist

Long form of GRIPP2

Short form of GRIPP2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

The International Database of Prospectively Registered Systematic Reviews

British Medical Journal

Patient saftey: World Health Organisation. 2023. Available from: https://www.who.int/news-room/fact-sheets/detail/patient-safety . Updated 11 Sept 2023.

Slawomirski L, Klazinga N. The economics of patient safety: from analysis to action. Paris: Organisation for Economic Co-operation and Development; 2020.

Google Scholar  

Panagioti M, Khan K, Keers RN, Abuzour A, Phipps D, Kontopantelis E, et al. Prevalence, severity, and nature of preventable patient harm across medical care settings: systematic review and meta-analysis. Bmj-Brit Med J. 2019;366:l4185.

Article   Google Scholar  

Hodkinson A, Tyler N, Ashcroft DM, Keers RN, Khan K, Phipps D, et al. Preventable medication harm across health care settings: a systematic review and meta-analysis. Bmc Medicine. 2020;18(1):313.

Article   PubMed   PubMed Central   Google Scholar  

Park M, Giap TTT. Patient and family engagement as a potential approach for improving patient safety: A systematic review. J Adv Nurs. 2020;76(1):62–80.

Article   PubMed   Google Scholar  

Chegini Z, Janati A, Bababie J, Pouraghaei M. The role of patients in the delivery of safe care in hospital: Study protocol. J Adv Nurs. 2019;75(9):2015–23.

Chegini Z, Arab-Zozani M, Islam SMS, Tobiano G, Rahimi SA. Barriers and facilitators to patient engagement in patient safety from patients and healthcare professionals’ perspectives: A systematic review and meta-synthesis. Nurs Forum. 2021;56(4):938–49.

Going the extra mile: improving the nation’s health and wellbeing through public involvement in research. London: National Institute for Health; 2015.

Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Bmj-Brit Med J. 2017;358:j3453.

Article   CAS   Google Scholar  

Minogue V. Knowledge translation, dissemination, and impact: a practical guide for researchers. Guide No 8: patient and public involvement in HSE research. Ireland: Health Service Executive Research and Development; 2021.

Staniszewska S, Brett J, Mockford C, Barber R. The GRIPP checklist: Strengthening the quality of patient and public involvement reporting in research. Int J Technol Assess Health Care. 2011;27(4):391–9.

Bergin RJ, Short CE, Davis N, Marker J, Dawson MT, Milton S, et al. The nature and impact of patient and public involvement in cancer prevention, screening and early detection research: A systematic review. Prev Med. 2023;167:107412.

Owyang D, Bakhsh A, Brewer D, Boughton OR, Cobb JP. Patient and public involvement within orthopaedic research a systematic review. J Bone Joint Surg Am. 2021;103(13):e51.

Busch IM, Saxena A, Wu AW. Putting the patient in patient safety investigations: barriers and strategies for involvement. J Patient Saf. 2021;17(5):358–62.

Lee M, Lee NJ, Seo HJ, Jang H, Kim SM. Interventions to engage patients and families in patient safety: a systematic review. West J Nurs Res. 2021;43(10):972–83.

Ocloo J, Garfield S, Franklin BD, Dawson S. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Syst. 2021;19(1):8.

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: Systematic review and co-design pilot. Health Expect. 2019;22(4):785–801.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Plos Medicine. 2021;18(3):372.

INVOLVE. What is public involvement in research? NIHR; 2019. Available from: https://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/ .

Shahid A, Sept B, Kupsch S, Brundin-Mather R, Piskulic D, Soo A, et al. Development and pilot implementation of a patient-oriented discharge summary for critically Ill patients. World J Crit Care Med. 2022;11(4):255–68.

Bisset CN, Dames N, Oliphant R, Alasadi A, Anderson D, Parson S, et al. Exploring shared surgical decision-making from the patient’s perspective: is the personality of the surgeon important? Colorectal Dis. 2020;22(12):2214–21.

Article   CAS   PubMed   Google Scholar  

Morris RL, Ruddock A, Gallacher K, Rolfe C, Giles S, Campbell S. Developing a patient safety guide for primary care: A co-design approach involving patients, carers and clinicians. Health Expect. 2021;24(1):42–52.

Tobiano G, Marshall AP, Gardiner T, Jenkinson K, Shapiro M, Ireland M. Development and psychometric testing of the patient participation in bedside handover survey. Health Expect. 2022;25(5):2492–502.

Francis-Coad J, Farlie MK, Haines T, Black L, Weselman T, Cummings P, et al. Revising and evaluating falls prevention education for older adults in hospital. Health Educ J. 2023;82(8):878–91.

Troya MI, Chew-Graham CA, Babatunde O, Bartlam B, Higginbottom A, Dikomitis L. Patient and public involvement and engagement in a doctoral research project exploring self-harm in older adults. Health Expect. 2019;22(4):617–31.

Aharaz A, Kejser CL, Poulsen MW, Jeftic S, Ulstrup-Hansen AI, Jorgensen LM, et al. Optimization of the Danish National Electronic Prescribing System to improve patient safety: Development of a user-friendly prototype of the digital platform shared medication record. Pharmacy (Basel, Switzerland). 2023;11(2):41.

PubMed   Google Scholar  

Aho-Glele U, Bouabida K, Kooijman A, Popescu IC, Pomey MP, Hawthornthwaite L, et al. Developing the first pan-Canadian survey on patient engagement in patient safety. BMC Health Serv Res. 2021;21(1):1099.

Albutt A, O’Hara J, Conner M, Lawton R. Involving patients in recognising clinical deterioration in hospital using the patient wellness questionnaire: A mixed-methods study. J Res Nurs. 2020;25(1):68–86.

Bell SK, Bourgeois F, DesRoches CM, Dong J, Harcourt K, Liu SK, et al. Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care. BMJ Qual Saf. 2022;31(7):526–40.

Boet S, Etherington N, Lam S, Lê M, Proulx L, Britton M, et al. Implementation of the Operating Room Black Box research program at the Ottawa Hospital through patient, clinical, and organizational engagement: Case study. J Med Internet Res. 2021;23(3):e15443.

Carter J, Tribe RM, Shennan AH, Sandall J. Threatened preterm labour: Women’s experiences of risk and care management: A qualitative study. Midwifery. 2018;64:85–92.

Da Silva Lopes AM, Colomer-Lahiguera S, Mederos Alfonso N, Aedo-Lopez V, Spurrier-Bernard G, Tolstrup LK, et al. Patient-reported outcomes for monitoring symptomatic toxicities in cancer patients treated with immune-checkpoint inhibitors: A Delphi study. Eur J Cancer. 2021;157:225–37.

de Jong LD, Lavender AP, Wortham C, Skelton DA, Haines TP, Hill AM. Exploring purpose-designed audio-visual falls prevention messages on older people’s capability and motivation to prevent falls. Health Soc Care Community. 2019;27(4):e471–82.

Doucette L, Kiely BT, Gierisch JM, Marion E, Nadler L, Heflin MT, et al. Participatory research to improve medication reconciliation for older adults in the community. J Am Geriatr Soc. 2023;71(2):620–31.

Elrod CS, Pappa ST, Heyn PC, Wong RA. Using an academic-community partnership model to deliver evidence-based falls prevention programs in a metropolitan setting: A community case study. Front Public Health. 2023;11:1073520.

Feldman E, Pos FJ, Smeenk RJ, van der Poel H, van Leeuwen P, de Feijter JM, et al. Selecting a PRO-CTCAE-based subset for patient-reported symptom monitoring in prostate cancer patients: a modified Delphi procedure. ESMO Open. 2023;8(1):100775.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Francis-Coad J, Watts T, Bulsara C, Hill A-M. Designing and evaluating falls prevention education with residents and staff in aged care homes: a feasibility study. Health Educ (0965-4283). 2022;122(5):546–63.

Fuller TE, Pong DD, Piniella N, Pardo M, Bessa N, Yoon C, et al. Interactive digital health tools to engage patients and caregivers in discharge preparation: implementation study. J Med Internet Res. 2020;22(4):e15573.

Gibson B, Butler J, Schnock K, Bates D, Classen D. Design of a safety dashboard for patients. Patient Educ Couns. 2020;103(4):741–7.

Giles SJ, Lewis PJ, Phipps DL, Mann F, Avery AJ, Ashcroft DM. Capturing patients’ perspectives on medication safety: the development of a patient-centered medication safety framework. J Patient Saf. 2020;16(4):e324–39.

Gnagi R, Zuniga F, Brunkert T, Meyer-Massetti C. Development of a medication literacy assessment instrument (MELIA) for older people receiving home care. J Adv Nurs. 2022;78(12):4210–20.

Goodsmith N, Zhang L, Ong MK, Ngo VK, Miranda J, Hirsch S, et al. Implementation of a community-partnered research suicide-risk management protocol: case study from community partners in care. Psychiatr Serv (Washington, DC). 2021;72(3):281–7.

Gorman LS, Littlewood DL, Quinlivan L, Monaghan E, Smith J, Barlow S, et al. Family involvement, patient safety and suicide prevention in mental healthcare: ethnographic study. BJPsych open. 2023;9(2):e54.

Green MM, Meyer C, Hutchinson AM, Sutherland F, Lowthian JA. Co‐designing being your best program—a holistic approach to frailty in older community dwelling australians. Health Soc Care Community. 2021;30(5):e2022–32.

Guo X, Wang Y, Wang L, Yang X, Yang W, Lu Z, et al. Effect of a fall prevention strategy for the older patients: A quasi-experimental study. Nurs Open. 2023;10(2):1116–24.

Hahn-Goldberg S, Chaput A, Rosenberg-Yunger Z, Lunsky Y, Okrainec K, Guilcher S, et al. Tool development to improve medication information transfer to patients during transitions of care: A participatory action research and design thinking methodology approach. Res Social Adm Pharm. 2022;18(1):2170–7.

Harrington A, Darke H, Ennis G, Sundram S. Evaluation of an alternative model for the management of clinical risk in an adult acute psychiatric inpatient unit. Int J Ment Health Nurs. 2019;28(5):1099–109.

Harris K, Softeland E, Moi AL, Harthug S, Ravnoy M, Storesund A, et al. Development and validation of patients’ surgical safety checklist. BMC Health Serv Res. 2022;22(1):259.

Hawley-Hague H, Tacconi C, Mellone S, Martinez E, Ford C, Chiari L, et al. Smartphone apps to support falls rehabilitation exercise: app development and usability and acceptability study. JMIR Mhealth Uhealth. 2020;8(9):e15460.

Holmqvist M, Ros A, Lindenfalk B, Thor J, Johansson L. How older persons and health care professionals co-designed a medication plan prototype remotely to promote patient safety: case study. JMIR aging. 2023;6:e41950.

Jayesinghe R, Moriarty F, Khatter A, Durbaba S, Ashworth M, Redmond P. Cost outcomes of potentially inappropriate prescribing in middle-aged adults: A Delphi consensus and cross-sectional study. Br J Clin Pharmacol. 2022;88(7):3404–20.

Johannessen T, Ree E, Stromme T, Aase I, Bal R, Wiig S. Designing and pilot testing of a leadership intervention to improve quality and safety in nursing homes and home care (the SAFE-LEAD intervention). BMJ Open. 2019;9(6):e027790.

Joseph K, Newman B, Manias E, Walpola R, Seale H, Walton M, et al. Engaging with ethnic minority consumers to improve safety in cancer services: A national stakeholder analysis. Patient Educ Couns. 2022;105(8):2778–84.

Khan A, Spector ND, Baird JD, Ashland M, Starmer AJ, Rosenbluth G, et al. Patient safety after implementation of a coproduced family centered communication programme: multicenter before and after intervention study. BMJ. 2018;363:k4764.

Khazen M, Mirica M, Carlile N, Groisser A, Schiff GD. Developing a framework and electronic tool for communicating diagnostic uncertainty in primary care: a qualitative study. JAMA Network Open. 2023;6(3):e232218-e.

Knight SW, Trinkle J, Tschannen D. Hospital-to-homecare videoconference handoff: improved communication, coordination of care, and patient/family engagement. Home Healthc Now. 2019;37(4):198–207.

Lawrence V, Kimona K, Howard RJ, Serfaty MA, Wetherell JL, Livingston G, et al. Optimising the acceptability and feasibility of acceptance and commitment therapy for treatment-resistant generalised anxiety disorder in older adults. Age Ageing. 2019;48(5):741–50.

Louch G, Reynolds C, Moore S, Marsh C, Heyhoe J, Albutt A, et al. Validation of revised patient measures of safety: PMOS-30 and PMOS-10. BMJ Open. 2019;9(11):e031355.

MacDonald T, Jackson S, Charles M-C, Periel M, Jean-Baptiste M-V, Salomon A, et al. The fourth delay and community-driven solutions to reduce maternal mortality in rural Haiti: a community-based action research study. BMC Pregnancy Childbirth. 2018;18(1):254.

Mackintosh N, Sandall J, Collison C, Carter W, Harris J. Employing the arts for knowledge production and translation: Visualizing new possibilities for women speaking up about safety concerns in maternity. Health Expect. 2018;21(3):647–58.

Marchand K, Turuba R, Katan C, Brasset C, Fogarty O, Tallon C, et al. Becoming our young people’s case managers: caregivers’ experiences, needs, and ideas for improving opioid use treatments for young people using opioids. Subst Abuse Treat Prev Policy. 2022;17(1):1–15.

Mazuz K, Biswas S. Co-designing technology and aging in a service setting: Developing an interpretive framework of how to interact with older age users. Gerontechnology. 2022;21(1):1–13.

McCahon D, Duncan P, Payne R, Horwood J. Patient perceptions and experiences of medication review: qualitative study in general practice. BMC Prim Care. 2022;23(1):293.

McMullen S, Panagioti M, Planner C, Giles S, Angelakis I, Keers RN, et al. Supporting carers to improve patient safety and maintain their well-being in transitions from mental health hospitals to the community: A prioritisation nominal group technique. Health Expect. 2023;26(5):2064–74.

Morris RL, Giles S, Campbell S. Involving patients and carers in patient safety in primary care: A qualitative study of a co-designed patient safety guide. Health Expect. 2023;26(2):630–9.

Morris RL, Stocks SJ, Alam R, Taylor S, Rolfe C, Glover SW, et al. Identifying primary care patient safety research priorities in the UK: a James Lind Alliance Priority Setting Partnership. BMJ Open. 2018;8(2):e020870.

Nether KG, Thomas EJ, Khan A, Ottosen MJ, Yager L. Implementing a robust process improvement program in the neonatal intensive care unit to reduce harm. J Healthc Qual. 2022;44(1):23–30.

Powell C, Ismail H, Cleverley R, Taylor A, Breen L, Fylan B, et al. Patients as qualitative data analysts: Developing a method for a process evaluation of the “Improving the Safety and Continuity of Medicines management at care Transitions” (ISCOMAT) cluster randomised control trial. Health Expect. 2021;24(4):1254–62.

Article   PubMed Central   Google Scholar  

Powell C, Ismail H, Davis M, Taylor A, Breen L, Fylan B, et al. Experiences of patients with heart failure with medicines at transition intervention: Findings from the process evaluation of the Improving the Safety and Continuity of Medicines management at Transitions of care (ISCOMAT) programme. Health Expect. 2022;25(5):2503–14.

Radecki B, Keen A, Miller J, McClure JK, Kara A. Innovating fall safety: engaging patients as experts. J Nurs Care Qual. 2020;35(3):220–6.

Rosgen BK, Plotnikoff KM, Krewulak KD, Shahid A, Hernandez L, Sept BG, et al. Co-development of a transitions in care bundle for patient transitions from the intensive care unit: a mixed-methods analysis of a stakeholder consensus meeting. BMC Health Serv Res. 2022;22(1):10.

Schenk EC, Bryant RA, Van Son CR, Odom-Maryon T. Developing an intervention to reduce harm in hospitalized patients: patients and families in research. J Nurs Care Qual. 2019;34(3):273–8.

Spazzapan M, Vijayakumar B, Stewart CE. A bit about me: Bedside boards to create a culture of patient-centered care in pediatric intensive care units (PICUs). J Healthc Risk Manag. 2020;39(3):11–9.

Stoll JA, Ranahan M, Richbart MT, Brennan-Taylor MK, Taylor JS, Brady L, et al. Development of video animations to encourage patient-driven deprescribing: A team alice study. Patient Educ Couns. 2021;104(11):2716–23.

Subbe CP, Tomos H, Jones GM, Barach P. Express check-in: developing a personal health record for patients admitted to hospital with medical emergencies: a mixed-method feasibility study. Int J Qual Health Care. 2021;33(3):121.

Tai D, Li E, Liu-Ambrose T, Bansback N, Sadatsafavi M, Davis JC. Patient-Reported Outcome Measures (PROMs) to support adherence to falls prevention clinic recommendations: a qualitative study. Patient Prefer Adherence. 2020;14:2105–21.

Thakur T, Chewning B, Zetes N, Lee JTY. Involving caregivers in design and assessment of opioid risk and safety communication intervention in children. Patient Educ Couns. 2021;104(10):2432–6.

Thomas J, Dahm MR, Li J, Georgiou A. Can patients contribute to enhancing the safety and effectiveness of test-result follow-up? Qualitative outcomes from a health consumer workshop. Health Expect. 2021;24(2):222–33.

Tremblay MC, Bradette-Laplante M, Witteman HO, Dogba MJ, Breault P, Paquette JS, et al. Providing culturally safe care to indigenous people living with diabetes: Identifying barriers and enablers from different perspectives. Health Expect. 2021;24(2):296–306.

Troya MI, Dikomitis L, Babatunde OO, Bartlam B, Chew-Graham CA. Understanding self-harm in older adults: A qualitative study. EClinicalMedicine. 2019;12:52–61.

Tyler N, Giles S, Daker-White G, McManus BC, Panagioti M. A patient and public involvement workshop using visual art and priority setting to provide patients with a voice to describe quality and safety concerns: Vitamin B12 deficiency and pernicious anaemia. Health Expect. 2021;24(1):87–94.

Tyler N, Planner C, Shears B, Hernan A, Panagioti M, Giles S. Developing the Resident Measure of Safety in Care Homes (RMOS): A Delphi and think aloud study. Health Expect. 2023;26(3):1149–58.

Van den Bulck SA, Vankrunkelsven P, Goderis G, Van Pottelbergh G, Swerts J, Panis K, et al. Developing quality indicators for Chronic Kidney Disease in primary care, extractable from the Electronic Medical Record. A Rand-modified Delphi method. BMC Nephrol. 2020;21(1):161.

Van Strien-Knippenberg IS, Boshuizen MCS, Determann D, de Boer JH, Damman OC. Cocreation with Dutch patients of decision-relevant information to support shared decision-making about adjuvant treatment in breast cancer care. Health Expect. 2022;25(4):1664–77.

Wilson NA, Reich AJ, Graham J, Bhatt DL, Nguyen LL, Weissman JS. Patient perspectives on the need for implanted device information: Implications for a post-procedural communication framework. Health Expect. 2021;24(4):1391–402.

Winterberg AV, Lane B, Hill LM, Varughese AM. Optimizing Pediatric Induction Experiences Using Human-centered Design. J Perianesth Nurs. 2022;37(1):48–52.

Yang R, Donaldson GW, Edelman LS, Cloyes KG, Sanders NA, Pepper GA. Fear of older adult falling questionnaire for caregivers (FOAFQ-CG): Evidence from content validity and item-response theory graded-response modelling. J Adv Nurs. 2020;76(10):2768–80.

Young A, Menon D, Street J, Al-Hertani W, Stafinski T. A checklist for managed access programmes for reimbursement co-designed by Canadian patients and caregivers. Health Expect. 2018;21(6):973–80.

Yuen EYN, Street M, Abdelrazek M, Blencowe P, Etienne G, Liskaser R, et al. Evaluating the efficacy of a digital App to enhance patient-centred nursing handover: A simulation study. J Clin Nurs. 2023;32(19–20):7626–37.

Jo S, Nabatchi T. Coproducing healthcare: individual-level impacts of engaging citizens to develop recommendations for reducing diagnostic error. Public Manag Rev. 2019;21(3):354–75.

O’Hara JK, Reynolds C, Moore S, Armitage G, Sheard L, Marsh C, et al. What can patients tell us about the quality and safety of hospital care? Findings from a UK multicentre survey study. BMJ Qual Saf. 2018;27(9):673–82.

de Jong LD, Francis-Coad J, Wortham C, Haines TP, Skelton DA, Weselman T, et al. Evaluating audio-visual falls prevention messages with community-dwelling older people using a World Cafe forum approach. BMC Geriatrics. 2019;19(1):345.

O’Donnell D, Shé ÉN, McCarthy M, Thornton S, Doran T, Smith F, et al. Enabling public, patient and practitioner involvement in co-designing frailty pathways in the acute care setting. BMC Health Serv Res. 2019;19(1):797.

Russ S, Latif Z, Hazell A, Ogunmuyiwa H, Tapper J, Wachuku-King S, et al. A Smartphone app designed to empower patients to contribute toward safer surgical care: community-based evaluation using a participatory approach. Jmir Mhealth Uhealth. 2020;8(1):e12859.

Mazuz K, Biswas S, Lindner U. Developing self-management application of fall prevention among older adults: a content and usability evaluation. Front Digital Health. 2020;2:11.

Hjelmfors L, Strömberg A, Friedrichsen M, Sandgren A, Mårtensson J, Jaarsma T. Using co-design to develop an intervention to improve communication about the heart failure trajectory and end-of-life care. Bmc Palliat Care. 2018;17:17.

Horgan S, Hegarty J, Andrews E, Hooton C, Drennan J. Impact of a quality improvement intervention on the incidence of surgical site infection in patients undergoing colorectal surgery: Pre-test-post-test design. J Clin Nurs. 2023;32(15–16):4932–46.

Tyler N, Wright N, Grundy A, Waring J. Developing a core outcome set for interventions to improve discharge from mental health inpatient services: a survey, Delphi and consensus meeting with key stakeholder groups. BMJ Open. 2020;10(5):e034215.

Ward ME, De Brún A, Beirne D, Conway C, Cunningham U, English A, et al. Using Co-Design to Develop a Collective Leadership Intervention for Healthcare Teams to Improve Safety Culture. Int J Environ Res Public Health. 2018;15(6):1182.

Berthelsen DB, Simon LS, Ioannidis JPA, Voshaar M, Richards P, Goel N, et al. Stakeholder endorsement advancing the implementation of a patient-reported domain for harms in rheumatology clinical trials: outcome of the OMERACT safety working group. Semin Arthritis Rheum. 2023;63:152288.

Okkenhaug A, Tritter JQ, Landstad BJ. Developing a research tool to detect iatrogenic adverse events in psychiatric health care by involving service users and health professionals. J Psychiatr Ment Health Nurs. 2023;00:1–12.

Musbahi A, Clyde D, Small P, Courtney M, Mahawar K, Lamb PJ, et al. A systematic review of patient and public involvement (PPI) in bariatric research trials: the need for more work. Obes Surg. 2022;32(11):3740–51.

Patel VA, Shelswell J, Hillyard N, Pavitt S, Barber SK. A study of the reporting of patient and public involvement and engagement (PPIE) in orthodontic research. J Orthod. 2021;48(1):42–51.

Price A, Schroter S, Snow R, Hicks M, Harmston R, Staniszewska S, et al. Frequency of reporting on patient and public involvement (PPI) in research studies published in a general medical journal: a descriptive study. BMJ Open. 2018;8:e020452.

Amadea T, Anne-Marie B, Louise L. A researcher’s guide to patient and public involvement. 2017.

Download references

Acknowledgements

This research is funded as part of the Collective Leadership and Safety Cultures (Co-Lead) research programme which is funded by the Irish Health Research Board, grant reference number RL-2015–1588 and the Health Service Executive. The funders had no role in the study conceptualisation, design, data collection, analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

UCD Centre for Interdisciplinary Research, Education and Innovation in Health Systems (UCD IRIS), School of Nursing, Midwifery and Health Systems, Health Sciences Centre, University College Dublin, Dublin, Ireland

Sahar Hammoud, Laith Alsabek, Lisa Rogers & Eilish McAuliffe

Department of Oral and Maxillofacial Surgery, University Hospital Galway, Galway, Ireland

Laith Alsabek

You can also search for this author in PubMed   Google Scholar

Contributions

S.H and E.M.A designed the study. S.H developed the search strategies with feedback from L.A, L.R, and E.M.A. S.H conducted all searches. S.H and L.A screened the studies, extracted the data, and assessed the quality of PPI reporting. S.H analysed the data with feedback from E.M.A. S.H drafted the manuscript. All authors revised and approved the submitted manuscript. All authors agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Authors’ information

Corresponding author.

Correspondence to Sahar Hammoud .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hammoud, S., Alsabek, L., Rogers, L. et al. Systematic review on the frequency and quality of reporting patient and public involvement in patient safety research. BMC Health Serv Res 24 , 532 (2024). https://doi.org/10.1186/s12913-024-11021-z

Download citation

Received : 10 January 2024

Accepted : 21 April 2024

Published : 26 April 2024

DOI : https://doi.org/10.1186/s12913-024-11021-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Patient and public involvement
• Patient participation
• Research reporting
• Research involvement

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]