research paper for health care system

Loading metrics

Open Access

Peer-reviewed

Research Article

Assessing the impact of healthcare research: A systematic review of methodological frameworks

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing

Affiliation Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom

Roles Conceptualization, Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing – review & editing

* E-mail: [email protected]

Roles Data curation, Formal analysis, Methodology, Validation, Writing – review & editing

Roles Formal analysis, Methodology, Supervision, Validation, Writing – review & editing

• Samantha Cruz Rivera, 
• Derek G. Kyte, 
• Olalekan Lee Aiyegbusi, 
• Thomas J. Keeley, 
• Melanie J. Calvert

• Published: August 9, 2017
• https://doi.org/10.1371/journal.pmed.1002370
• Reader Comments

Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

Methods and findings

Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

Conclusions

The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Author summary

Why was this study done.

• There is a growing interest in demonstrating the impact of research in order to minimise research waste, allocate resources efficiently, and maximise the benefit of research. However, there is no consensus on which is the most appropriate tool to measure the impact of research.
• To our knowledge, this review is the first to synthesise existing methodological frameworks for healthcare research impact, and the associated impact metrics by which various authors have proposed impact should be measured, into a unified matrix.

What did the researchers do and find?

• We conducted a systematic review identifying 24 existing methodological research impact frameworks.
• We scrutinised the sample, identifying and summarising 5 proposed impact categories, 16 impact subcategories, and over 80 metrics into an impact matrix and methodological framework.

What do these findings mean?

• This simplified consolidated methodological framework will help researchers to understand how a research study may give rise to differing forms of impact, as well as in what ways and at which time points these potential impacts might be measured.
• Incorporating these insights into the design of a study could enhance impact, optimizing the use of research resources.

Citation: Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ (2017) Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 14(8): e1002370. https://doi.org/10.1371/journal.pmed.1002370

Academic Editor: Mike Clarke, Queens University Belfast, UNITED KINGDOM

Received: February 28, 2017; Accepted: July 7, 2017; Published: August 9, 2017

Copyright: © 2017 Cruz Rivera et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and supporting files.

Funding: Funding was received from Consejo Nacional de Ciencia y Tecnología (CONACYT). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript ( http://www.conacyt.mx/ ).

Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: MJC has received consultancy fees from Astellas and Ferring pharma and travel fees from the European Society of Cardiology outside the submitted work. TJK is in full-time paid employment for PAREXEL International.

Abbreviations: AIHS, Alberta Innovates—Health Solutions; CAHS, Canadian Academy of Health Sciences; CIHR, Canadian Institutes of Health Research; CINAHL+, Cumulative Index to Nursing and Allied Health Literature; EMBASE, Excerpta Medica Database; ERA, Excellence in Research for Australia; HEFCE, Higher Education Funding Council for England; HMIC, Health Management Information Consortium; HTA, Health Technology Assessment; IOM, Impact Oriented Monitoring; MDG, Millennium Development Goal; NHS, National Health Service; MEDLINE, Medical Literature Analysis and Retrieval System Online; PHC RIS, Primary Health Care Research & Information Service; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PROM, patient-reported outcome measures; QALY, quality-adjusted life year; R&D, research and development; RAE, Research Assessment Exercise; REF, Research Excellence Framework; RIF, Research Impact Framework; RQF, Research Quality Framework; SDG, Sustainable Development Goal; SIAMPI, Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society

Introduction

In 2010, approximately US$240 billion was invested in healthcare research worldwide [ 1 ]. Such research is utilised by policy makers, healthcare providers, and clinicians to make important evidence-based decisions aimed at maximising patient benefit, whilst ensuring that limited healthcare resources are used as efficiently as possible to facilitate effective and sustainable service delivery. It is therefore essential that this research is of high quality and that it is impactful—i.e., it delivers demonstrable benefits to society and the wider economy whilst minimising research waste [ 1 , 2 ]. Research impact can be defined as ‘any identifiable ‘benefit to, or positive influence on the economy, society, public policy or services, health, the environment, quality of life or academia’ (p. 26) [ 3 ].

There are many purported benefits associated with the measurement of research impact, including the ability to (1) assess the quality of the research and its subsequent benefits to society; (2) inform and influence optimal policy and funding allocation; (3) demonstrate accountability, the value of research in terms of efficiency and effectiveness to the government, stakeholders, and society; and (4) maximise impact through better understanding the concept and pathways to impact [ 4 – 7 ].

Measuring and monitoring the impact of healthcare research has become increasingly common in the United Kingdom [ 5 ], Australia [ 5 ], and Canada [ 8 ], as governments, organisations, and higher education institutions seek a framework to allocate funds to projects that are more likely to bring the most benefit to society and the economy [ 5 ]. For example, in the UK, the 2014 Research Excellence Framework (REF) has recently been used to assess the quality and impact of research in higher education institutions, through the assessment of impact cases studies and selected qualitative impact metrics [ 9 ]. This is the first initiative to allocate research funding based on the economic, societal, and cultural impact of research, although it should be noted that research impact only drives a proportion of this allocation (approximately 20%) [ 9 ].

In the UK REF, the measurement of research impact is seen as increasingly important. However, the impact element of the REF has been criticised in some quarters [ 10 , 11 ]. Critics deride the fact that REF impact is determined in a relatively simplistic way, utilising researcher-generated case studies, which commonly attempt to link a particular research outcome to an associated policy or health improvement despite the fact that the wider literature highlights great diversity in the way research impact may be demonstrated [ 12 , 13 ]. This led to the current debate about the optimal method of measuring impact in the future REF [ 10 , 14 ]. The Stern review suggested that research impact should not only focus on socioeconomic impact but should also include impact on government policy, public engagement, academic impacts outside the field, and teaching to showcase interdisciplinary collaborative impact [ 10 , 11 ]. The Higher Education Funding Council for England (HEFCE) has recently set out the proposals for the REF 2021 exercise, confirming that the measurement of such impact will continue to form an important part of the process [ 15 ].

With increasing pressure for healthcare research to lead to demonstrable health, economic, and societal impact, there is a need for researchers to understand existing methodological impact frameworks and the means by which impact may be quantified (i.e., impact metrics; see Box 1 , 'Definitions’) to better inform research activities and funding decisions. From a researcher’s perspective, understanding the optimal pathways to impact can help inform study design aimed at maximising the impact of the project. At the same time, funders need to understand which aspects of impact they should focus on when allocating awards so they can make the most of their investment and bring the greatest benefit to patients and society [ 2 , 4 , 5 , 16 , 17 ].

Box 1. Definitions

• Research impact: ‘any identifiable benefit to, or positive influence on, the economy, society, public policy or services, health, the environment, quality of life, or academia’ (p. 26) [ 3 ].
• Methodological framework: ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ].
• Pathway: ‘a way of achieving a specified result; a course of action’ [ 19 ].
• Quantitative metrics: ‘a system or standard of [quantitative] measurement’ [ 20 ].
• Narrative metrics: ‘a spoken or written account of connected events; a story’ [ 21 ].

Whilst previous researchers have summarised existing methodological frameworks and impact case studies [ 4 , 22 – 27 ], they have not summarised the metrics for use by researchers, funders, and policy makers. The aim of this review was therefore to (1) identify the methodological frameworks used to measure healthcare research impact using systematic methods, (2) summarise common impact themes and metrics in an impact matrix, and (3) provide a simplified consolidated resource for use by funders, researchers, and policy makers.

Search strategy and selection criteria

Initially, a search strategy was developed to identify the available literature regarding the different methods to measure research impact. The following keywords: ‘Impact’, ‘Framework’, and ‘Research’, and their synonyms, were used during the search of the Medical Literature Analysis and Retrieval System Online (MEDLINE; Ovid) database, the Excerpta Medica Database (EMBASE), the Health Management Information Consortium (HMIC) database, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL+) database (inception to May 2017; see S1 Appendix for the full search strategy). Additionally, the nonindexed Journal of Research Evaluation was hand searched during the same timeframe using the keyword ‘Impact’. Other relevant articles were identified through 3 Internet search engines (Google, Google Scholar, and Google Images) using the keywords ‘Impact’, ‘Framework’, and ‘Research’, with the first 50 results screened. Google Images was searched because different methodological frameworks are summarised in a single image and can easily be identified through this search engine. Finally, additional publications were sought through communication with experts.

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see S1 PRISMA Checklist ), 2 independent investigators systematically screened for publications describing, evaluating, or utilising a methodological research impact framework within the context of healthcare research [ 28 ]. Papers were eligible if they included full or partial methodological frameworks or pathways to research impact; both primary research and systematic reviews fitting these criteria were included. We included any methodological framework identified (original or modified versions) at the point of first occurrence. In addition, methodological frameworks were included if they were applicable to the healthcare discipline with no need of modification within their structure. We defined ‘methodological framework’ as ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ], whereas we defined ‘pathway’ as ‘a way of achieving a specified result; a course of action’ [ 19 ]. Studies were excluded if they presented an existing (unmodified) methodological framework previously available elsewhere, did not explicitly describe a methodological framework but rather focused on a single metric (e.g., bibliometric analysis), focused on the impact or effectiveness of interventions rather than that of the research, or presented case study data only. There were no language restrictions.

Data screening

Records were downloaded into Endnote (version X7.3.1), and duplicates were removed. Two independent investigators (SCR and OLA) conducted all screening following a pilot aimed at refining the process. The records were screened by title and abstract before full-text articles of potentially eligible publications were retrieved for evaluation. A full-text screening identified the publications included for data extraction. Discrepancies were resolved through discussion, with the involvement of a third reviewer (MJC, DGK, and TJK) when necessary.

Data extraction and analysis

Data extraction occurred after the final selection of included articles. SCR and OLA independently extracted details of impact methodological frameworks, the country of origin, and the year of publication, as well as the source, the framework description, and the methodology used to develop the framework. Information regarding the methodology used to develop each methodological framework was also extracted from framework webpages where available. Investigators also extracted details regarding each framework’s impact categories and subgroups, along with their proposed time to impact (‘short-term’, ‘mid-term’, or ‘long-term’) and the details of any metrics that had been proposed to measure impact, which are depicted in an impact matrix. The structure of the matrix was informed by the work of M. Buxton and S. Hanney [ 2 ], P. Buykx et al. [ 5 ], S. Kuruvila et al. [ 29 ], and A. Weiss [ 30 ], with the intention of mapping metrics presented in previous methodological frameworks in a concise way. A consensus meeting with MJC, DGK, and TJK was held to solve disagreements and finalise the data extraction process.

Included studies

Our original search strategy identified 359 citations from MEDLINE (Ovid), EMBASE, CINAHL+, HMIC, and the Journal of Research Evaluation, and 101 citations were returned using other sources (Google, Google Images, Google Scholar, and expert communication) (see Fig 1 ) [ 28 ]. In total, we retrieved 54 full-text articles for review. At this stage, 39 articles were excluded, as they did not propose new or modified methodological frameworks. An additional 15 articles were included following the backward and forward citation method. A total of 31 relevant articles were included in the final analysis, of which 24 were articles presenting unique frameworks and the remaining 7 were systematic reviews [ 4 , 22 – 27 ]. The search strategy was rerun on 15 May 2017. A further 19 publications were screened, and 2 were taken forward to full-text screening but were ineligible for inclusion.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pmed.1002370.g001

Methodological framework characteristics

The characteristics of the 24 included methodological frameworks are summarised in Table 1 , 'Methodological framework characteristics’. Fourteen publications proposed academic-orientated frameworks, which focused on measuring academic, societal, economic, and cultural impact using narrative and quantitative metrics [ 2 , 3 , 5 , 8 , 29 , 31 – 39 ]. Five publications focused on assessing the impact of research by focusing on the interaction process between stakeholders and researchers (‘productive interactions’), which is a requirement to achieve research impact. This approach tries to address the issue of attributing research impact to metrics [ 7 , 40 – 43 ]. Two frameworks focused on the importance of partnerships between researchers and policy makers, as a core element to accomplish research impact [ 44 , 45 ]. An additional 2 frameworks focused on evaluating the pathways to impact, i.e., linking processes between research and impact [ 30 , 46 ]. One framework assessed the ability of health technology to influence efficiency of healthcare systems [ 47 ]. Eight frameworks were developed in the UK [ 2 , 3 , 29 , 37 , 39 , 42 , 43 , 45 ], 6 in Canada [ 8 , 33 , 34 , 44 , 46 , 47 ], 4 in Australia [ 5 , 31 , 35 , 38 ], 3 in the Netherlands [ 7 , 40 , 41 ], and 2 in the United States [ 30 , 36 ], with 1 model developed with input from various countries [ 32 ].

https://doi.org/10.1371/journal.pmed.1002370.t001

Methodological framework development

The included methodological frameworks varied in their development process, but there were some common approaches employed. Most included a literature review [ 2 , 5 , 7 , 8 , 31 , 33 , 36 , 37 , 40 – 46 ], although none of them used a recognised systematic method. Most also consulted with various stakeholders [ 3 , 8 , 29 , 31 , 33 , 35 – 38 , 43 , 44 , 46 , 47 ] but used differing methods to incorporate their views, including quantitative surveys [ 32 , 35 , 43 , 46 ], face-to-face interviews [ 7 , 29 , 33 , 35 , 37 , 42 , 43 ], telephone interviews [ 31 , 46 ], consultation [ 3 , 7 , 36 ], and focus groups [ 39 , 43 ]. A range of stakeholder groups were approached across the sample, including principal investigators [ 7 , 29 , 43 ], research end users [ 7 , 42 , 43 ], academics [ 3 , 8 , 39 , 40 , 43 , 46 ], award holders [ 43 ], experts [ 33 , 38 , 39 ], sponsors [ 33 , 39 ], project coordinators [ 32 , 42 ], and chief investigators [ 31 , 35 ]. However, some authors failed to identify the stakeholders involved in the development of their frameworks [ 2 , 5 , 34 , 41 , 45 ], making it difficult to assess their appropriateness. In addition, only 4 of the included papers reported using formal analytic methods to interpret stakeholder responses. These included the Canadian Academy of Health Sciences framework, which used conceptual cluster analysis [ 33 ]. The Research Contribution [ 42 ], Research Impact [ 29 ], and Primary Health Care & Information Service [ 31 ] used a thematic analysis approach. Finally, some authors went on to pilot their framework, which shaped refinements on the methodological frameworks until approval. Methods used to pilot the frameworks included a case study approach [ 2 , 3 , 30 , 32 , 33 , 36 , 40 , 42 , 44 , 45 ], contrasting results against available literature [ 29 ], the use of stakeholders’ feedback [ 7 ], and assessment tools [ 35 , 46 ].

Major impact categories

1. primary research-related impact..

A number of methodological frameworks advocated the evaluation of ‘research-related impact’. This encompassed content related to the generation of new knowledge, knowledge dissemination, capacity building, training, leadership, and the development of research networks. These outcomes were considered the direct or primary impacts of a research project, as these are often the first evidenced returns [ 30 , 62 ].

A number of subgroups were identified within this category, with frameworks supporting the collection of impact data across the following constructs: ‘research and innovation outcomes’; ‘dissemination and knowledge transfer’; ‘capacity building, training, and leadership’; and ‘academic collaborations, research networks, and data sharing’.

1 . 1 . Research and innovation outcomes . Twenty of the 24 frameworks advocated the evaluation of ‘research and innovation outcomes’ [ 2 , 3 , 5 , 7 , 8 , 29 – 39 , 41 , 43 , 44 , 46 ]. This subgroup included the following metrics: number of publications; number of peer-reviewed articles (including journal impact factor); citation rates; requests for reprints, number of reviews, and meta-analysis; and new or changes in existing products (interventions or technology), patents, and research. Additionally, some frameworks also sought to gather information regarding ‘methods/methodological contributions’. These advocated the collection of systematic reviews and appraisals in order to identify gaps in knowledge and determine whether the knowledge generated had been assessed before being put into practice [ 29 ].

1 . 2 . Dissemination and knowledge transfer . Nineteen of the 24 frameworks advocated the assessment of ‘dissemination and knowledge transfer’ [ 2 , 3 , 5 , 7 , 29 – 32 , 34 – 43 , 46 ]. This comprised collection of the following information: number of conferences, seminars, workshops, and presentations; teaching output (i.e., number of lectures given to disseminate the research findings); number of reads for published articles; article download rate and number of journal webpage visits; and citations rates in nonjournal media such as newspapers and mass and social media (i.e., Twitter and blogs). Furthermore, this impact subgroup considered the measurement of research uptake and translatability and the adoption of research findings in technological and clinical applications and by different fields. These can be measured through patents, clinical trials, and partnerships between industry and business, government and nongovernmental organisations, and university research units and researchers [ 29 ].

1 . 3 . Capacity building , training , and leadership . Fourteen of 24 frameworks suggested the evaluation of ‘capacity building, training, and leadership’ [ 2 , 3 , 5 , 8 , 29 , 31 – 35 , 39 – 41 , 43 ]. This involved collecting information regarding the number of doctoral and postdoctoral studentships (including those generated as a result of the research findings and those appointed to conduct the research), as well as the number of researchers and research-related staff involved in the research projects. In addition, authors advocated the collection of ‘leadership’ metrics, including the number of research projects managed and coordinated and the membership of boards and funding bodies, journal editorial boards, and advisory committees [ 29 ]. Additional metrics in this category included public recognition (number of fellowships and awards for significant research achievements), academic career advancement, and subsequent grants received. Lastly, the impact metric ‘research system management’ comprised the collection of information that can lead to preserving the health of the population, such as modifying research priorities, resource allocation strategies, and linking health research to other disciplines to maximise benefits [ 29 ].

1 . 4 . Academic collaborations , research networks , and data sharing . Lastly, 10 of the 24 frameworks advocated the collection of impact data regarding ‘academic collaborations (internal and external collaborations to complete a research project), research networks, and data sharing’ [ 2 , 3 , 5 , 7 , 29 , 34 , 37 , 39 , 41 , 43 ].

2. Influence on policy making.

Methodological frameworks addressing this major impact category focused on measurable improvements within a given knowledge base and on interactions between academics and policy makers, which may influence policy-making development and implementation. The returns generated in this impact category are generally considered as intermediate or midterm (1 to 3 years). These represent an important interim stage in the process towards the final expected impacts, such as quantifiable health improvements and economic benefits, without which policy change may not occur [ 30 , 62 ]. The following impact subgroups were identified within this category: ‘type and nature of policy impact’, ‘level of policy making’, and ‘policy networks’.

2 . 1 . Type and nature of policy impact . The most common impact subgroup, mentioned in 18 of the 24 frameworks, was ‘type and nature of policy impact’ [ 2 , 7 , 29 – 38 , 41 – 43 , 45 – 47 ]. Methodological frameworks addressing this subgroup stressed the importance of collecting information regarding the influence of research on policy (i.e., changes in practice or terminology). For instance, a project looking at trafficked adolescents and women (2003) influenced the WHO guidelines (2003) on ethics regarding this particular group [ 17 , 21 , 63 ].

2 . 2 . Level of policy impact . Thirteen of 24 frameworks addressed aspects surrounding the need to record the ‘level of policy impact’ (international, national, or local) and the organisations within a level that were influenced (local policy makers, clinical commissioning groups, and health and wellbeing trusts) [ 2 , 5 , 8 , 29 , 31 , 34 , 38 , 41 , 43 – 47 ]. Authors considered it important to measure the ‘level of policy impact’ to provide evidence of collaboration, coordination, and efficiency within health organisations and between researchers and health organisations [ 29 , 31 ].

2 . 3 . Policy networks . Five methodological frameworks highlighted the need to collect information regarding collaborative research with industry and staff movement between academia and industry [ 5 , 7 , 29 , 41 , 43 ]. A policy network emphasises the relationship between policy communities, researchers, and policy makers. This relationship can influence and lead to incremental changes in policy processes [ 62 ].

3. Health and health systems impact.

A number of methodological frameworks advocated the measurement of impacts on health and healthcare systems across the following impact subgroups: ‘quality of care and service delivering’, ‘evidence-based practice’, ‘improved information and health information management’, ‘cost containment and effectiveness’, ‘resource allocation’, and ‘health workforce’.

3 . 1 . Quality of care and service delivery . Twelve of the 24 frameworks highlighted the importance of evaluating ‘quality of care and service delivery’ [ 2 , 5 , 8 , 29 – 31 , 33 – 36 , 41 , 47 ]. There were a number of suggested metrics that could be potentially used for this purpose, including health outcomes such as quality-adjusted life years (QALYs), patient-reported outcome measures (PROMs), patient satisfaction and experience surveys, and qualitative data on waiting times and service accessibility.

3 . 2 . Evidence-based practice . ‘Evidence-based practice’, mentioned in 5 of the 24 frameworks, refers to making changes in clinical diagnosis, clinical practice, treatment decisions, or decision making based on research evidence [ 5 , 8 , 29 , 31 , 33 ]. The suggested metrics to demonstrate evidence-based practice were adoption of health technologies and research outcomes to improve the healthcare systems and inform policies and guidelines [ 29 ].

3 . 3 . Improved information and health information management . This impact subcategory, mentioned in 5 of the 24 frameworks, refers to the influence of research on the provision of health services and management of the health system to prevent additional costs [ 5 , 29 , 33 , 34 , 38 ]. Methodological frameworks advocated the collection of health system financial, nonfinancial (i.e., transport and sociopolitical implications), and insurance information in order to determine constraints within a health system.

3 . 4 . Cost containment and cost-effectiveness . Six of the 24 frameworks advocated the subcategory ‘cost containment and cost-effectiveness’ [ 2 , 5 , 8 , 17 , 33 , 36 ]. ‘Cost containment’ comprised the collection of information regarding how research has influenced the provision and management of health services and its implication in healthcare resource allocation and use [ 29 ]. ‘Cost-effectiveness’ refers to information concerning economic evaluations to assess improvements in effectiveness and health outcomes—for instance, the cost-effectiveness (cost and health outcome benefits) assessment of introducing a new health technology to replace an older one [ 29 , 31 , 64 ].

3 . 5 . Resource allocation . ‘Resource allocation’, mentioned in 6frameworks, can be measured through 2 impact metrics: new funding attributed to the intervention in question and equity while allocating resources, such as improved allocation of resources at an area level; better targeting, accessibility, and utilisation; and coverage of health services [ 2 , 5 , 29 , 31 , 45 , 47 ]. The allocation of resources and targeting can be measured through health services research reports, with the utilisation of health services measured by the probability of providing an intervention when needed, the probability of requiring it again in the future, and the probability of receiving an intervention based on previous experience [ 29 , 31 ].

3 . 6 . Health workforce . Lastly, ‘health workforce’, present in 3 methodological frameworks, refers to the reduction in the days of work lost because of a particular illness [ 2 , 5 , 31 ].

4. Health-related and societal impact.

Three subgroups were included in this category: ‘health literacy’; ‘health knowledge, attitudes, and behaviours’; and ‘improved social equity, inclusion, or cohesion’.

4 . 1 . Health knowledge , attitudes , and behaviours . Eight of the 24 frameworks suggested the assessment of ‘health knowledge, attitudes, behaviours, and outcomes’, which could be measured through the evaluation of levels of public engagement with science and research (e.g., National Health Service (NHS) Choices end-user visit rate) or by using focus groups to analyse changes in knowledge, attitudes, and behaviour among society [ 2 , 5 , 29 , 33 – 35 , 38 , 43 ].

4 . 2 . Improved equity , inclusion , or cohesion and human rights . Other methodological frameworks, 4 of the 24, suggested capturing improvements in equity, inclusion, or cohesion and human rights. Authors suggested these could be using a resource like the United Nations Millennium Development Goals (MDGs) (superseded by Sustainable Development Goals [SDGs] in 2015) and human rights [ 29 , 33 , 34 , 38 ]. For instance, a cluster-randomised controlled trial in Nepal, which had female participants, has demonstrated the reduction of neonatal mortality through the introduction of maternity health care, distribution of delivery kits, and home visits. This illustrates how research can target vulnerable and disadvantaged groups. Additionally, this research has been introduced by the World Health Organisation to achieve the MDG ‘improve maternal health’ [ 16 , 29 , 65 ].

4 . 3 . Health literacy . Some methodological frameworks, 3 of the 24, focused on tracking changes in the ability of patients to make informed healthcare decisions, reduce health risks, and improve quality of life, which were demonstrably linked to a particular programme of research [ 5 , 29 , 43 ]. For example, a systematic review showed that when HIV health literacy/knowledge is spread among people living with the condition, antiretroviral adherence and quality of life improve [ 66 ].

5. Broader economic impacts.

Some methodological frameworks, 9 of 24, included aspects related to the broader economic impacts of health research—for example, the economic benefits emerging from the commercialisation of research outputs [ 2 , 5 , 29 , 31 , 33 , 35 , 36 , 38 , 67 ]. Suggested metrics included the amount of funding for research and development (R&D) that was competitively awarded by the NHS, medical charities, and overseas companies. Additional metrics were income from intellectual property, spillover effects (any secondary benefit gained as a repercussion of investing directly in a primary activity, i.e., the social and economic returns of investing on R&D) [ 33 ], patents granted, licences awarded and brought to the market, the development and sales of spinout companies, research contracts, and income from industry.

The benefits contained within the categories ‘health and health systems impact’, ‘health-related and societal impact’, and ‘broader economic impacts’ are considered the expected and final returns of the resources allocated in healthcare research [ 30 , 62 ]. These benefits commonly arise in the long term, beyond 5 years according to some authors, but there was a recognition that this could differ depending on the project and its associated research area [ 4 ].

Data synthesis

Five major impact categories were identified across the 24 included methodological frameworks: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These major impact categories were further subdivided into 16 impact subgroups. The included publications proposed 80 different metrics to measure research impact. This impact typology synthesis is depicted in ‘the impact matrix’ ( Fig 2 and Fig 3 ).

CIHR, Canadian Institutes of Health Research; HTA, Health Technology Assessment; PHC RIS, Primary Health Care Research & Information Service; RAE, Research Assessment Exercise; RQF, Research Quality Framework.

https://doi.org/10.1371/journal.pmed.1002370.g002

AIHS, Alberta Innovates—Health Solutions; CAHS, Canadian Institutes of Health Research; IOM, Impact Oriented Monitoring; REF, Research Excellence Framework; SIAMPI, Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society.

https://doi.org/10.1371/journal.pmed.1002370.g003

Commonality and differences across frameworks

The ‘Research Impact Framework’ and the ‘Health Services Research Impact Framework’ were the models that encompassed the largest number of the metrics extracted. The most dominant methodological framework was the Payback Framework; 7 other methodological framework models used the Payback Framework as a starting point for development [ 8 , 29 , 31 – 35 ]. Additional methodological frameworks that were commonly incorporated into other tools included the CIHR framework, the CAHS model, the AIHS framework, and the Exchange model [ 8 , 33 , 34 , 44 ]. The capture of ‘research-related impact’ was the most widely advocated concept across methodological frameworks, illustrating the importance with which primary short-term impact outcomes were viewed by the included papers. Thus, measurement of impact via number of publications, citations, and peer-reviewed articles was the most common. ‘Influence on policy making’ was the predominant midterm impact category, specifically the subgroup ‘type and nature of policy impact’, in which frameworks advocated the measurement of (i) changes to legislation, regulations, and government policy; (ii) influence and involvement in decision-making processes; and (iii) changes to clinical or healthcare training, practice, or guidelines. Within more long-term impact measurement, the evaluations of changes in the ‘quality of care and service delivery’ were commonly advocated.

In light of the commonalities and differences among the methodological frameworks, the ‘pathways to research impact’ diagram ( Fig 4 ) was developed to provide researchers, funders, and policy makers a more comprehensive and exhaustive way to measure healthcare research impact. The diagram has the advantage of assorting all the impact metrics proposed by previous frameworks and grouping them into different impact subgroups and categories. Prospectively, this global picture will help researchers, funders, and policy makers plan strategies to achieve multiple pathways to impact before carrying the research out. The analysis of the data extraction and construction of the impact matrix led to the development of the ‘pathways to research impact’ diagram ( Fig 4 ). The diagram aims to provide an exhaustive and comprehensive way of tracing research impact by combining all the impact metrics presented by the different 24 frameworks, grouping those metrics into different impact subgroups, and grouping these into broader impact categories.

NHS, National Health Service; PROM, patient-reported outcome measure; QALY, quality-adjusted life year; R&D, research and development.

https://doi.org/10.1371/journal.pmed.1002370.g004

This review has summarised existing methodological impact frameworks together for the first time using systematic methods ( Fig 4 ). It allows researchers and funders to consider pathways to impact at the design stage of a study and to understand the elements and metrics that need to be considered to facilitate prospective assessment of impact. Users do not necessarily need to cover all the aspects of the methodological framework, as every research project can impact on different categories and subgroups. This review provides information that can assist researchers to better demonstrate impact, potentially increasing the likelihood of conducting impactful research and reducing research waste. Existing reviews have not presented a methodological framework that includes different pathways to impact, health impact categories, subgroups, and metrics in a single methodological framework.

Academic-orientated frameworks included in this review advocated the measurement of impact predominantly using so-called ‘quantitative’ metrics—for example, the number of peer-reviewed articles, journal impact factor, and citation rates. This may be because they are well-established measures, relatively easy to capture and objective, and are supported by research funding systems. However, these metrics primarily measure the dissemination of research finding rather than its impact [ 30 , 68 ]. Whilst it is true that wider dissemination, especially when delivered via world-leading international journals, may well lead eventually to changes in healthcare, this is by no means certain. For instance, case studies evaluated by Flinders University of Australia demonstrated that some research projects with non-peer-reviewed publications led to significant changes in health policy, whilst the studies with peer-reviewed publications did not result in any type of impact [ 68 ]. As a result, contemporary literature has tended to advocate the collection of information regarding a variety of different potential forms of impact alongside publication/citations metrics [ 2 , 3 , 5 , 7 , 8 , 29 – 47 ], as outlined in this review.

The 2014 REF exercise adjusted UK university research funding allocation based on evidence of the wider impact of research (through case narrative studies and quantitative metrics), rather than simply according to the quality of research [ 12 ]. The intention was to ensure funds were directed to high-quality research that could demonstrate actual realised benefit. The inclusion of a mixed-method approach to the measurement of impact in the REF (narrative and quantitative metrics) reflects a widespread belief—expressed by the majority of authors of the included methodological frameworks in the review—that individual quantitative impact metrics (e.g., number of citations and publications) do not necessary capture the complexity of the relationships involved in a research project and may exclude measurement of specific aspects of the research pathway [ 10 , 12 ].

Many of the frameworks included in this review advocated the collection of a range of academic, societal, economic, and cultural impact metrics; this is consistent with recent recommendations from the Stern review [ 10 ]. However, a number of these metrics encounter research ‘lag’: i.e., the time between the point at which the research is conducted and when the actual benefits arise [ 69 ]. For instance, some cardiovascular research has taken up to 25 years to generate impact [ 70 ]. Likewise, the impact may not arise exclusively from a single piece of research. Different processes (such as networking interactions and knowledge and research translation) and multiple individuals and organisations are often involved [ 4 , 71 ]. Therefore, attributing the contribution made by each of the different actors involved in the process can be a challenge [ 4 ]. An additional problem associated to attribution is the lack of evidence to link research and impact. The outcomes of research may emerge slowly and be absorbed gradually. Consequently, it is difficult to determine the influence of research in the development of a new policy, practice, or guidelines [ 4 , 23 ].

A further problem is that impact evaluation is conducted ‘ex post’, after the research has concluded. Collecting information retrospectively can be an issue, as the data required might not be available. ‘ex ante’ assessment is vital for funding allocation, as it is necessary to determine the potential forthcoming impact before research is carried out [ 69 ]. Additionally, ex ante evaluation of potential benefit can overcome the issues regarding identifying and capturing evidence, which can be used in the future [ 4 ]. In order to conduct ex ante evaluation of potential benefit, some authors suggest the early involvement of policy makers in a research project coupled with a well-designed strategy of dissemination [ 40 , 69 ].

Providing an alternate view, the authors of methodological frameworks such as the SIAMPI, Contribution Mapping, Research Contribution, and the Exchange model suggest that the problems of attribution are a consequence of assigning the impact of research to a particular impact metric [ 7 , 40 , 42 , 44 ]. To address these issues, these authors propose focusing on the contribution of research through assessing the processes and interactions between stakeholders and researchers, which arguably take into consideration all the processes and actors involved in a research project [ 7 , 40 , 42 , 43 ]. Additionally, contributions highlight the importance of the interactions between stakeholders and researchers from an early stage in the research process, leading to a successful ex ante and ex post evaluation by setting expected impacts and determining how the research outcomes have been utilised, respectively [ 7 , 40 , 42 , 43 ]. However, contribution metrics are generally harder to measure in comparison to academic-orientated indicators [ 72 ].

Currently, there is a debate surrounding the optimal methodological impact framework, and no tool has proven superior to another. The most appropriate methodological framework for a given study will likely depend on stakeholder needs, as each employs different methodologies to assess research impact [ 4 , 37 , 41 ]. This review allows researchers to select individual existing methodological framework components to create a bespoke tool with which to facilitate optimal study design and maximise the potential for impact depending on the characteristic of their study ( Fig 2 and Fig 3 ). For instance, if researchers are interested in assessing how influential their research is on policy making, perhaps considering a suite of the appropriate metrics drawn from multiple methodological frameworks may provide a more comprehensive method than adopting a single methodological framework. In addition, research teams may wish to use a multidimensional approach to methodological framework development, adopting existing narratives and quantitative metrics, as well as elements from contribution frameworks. This approach would arguably present a more comprehensive method of impact assessment; however, further research is warranted to determine its effectiveness [ 4 , 69 , 72 , 73 ].

Finally, it became clear during this review that the included methodological frameworks had been constructed using varied methodological processes. At present, there are no guidelines or consensus around the optimal pathway that should be followed to develop a robust methodological framework. The authors believe this is an area that should be addressed by the research community, to ensure future frameworks are developed using best-practice methodology.

For instance, the Payback Framework drew upon a literature review and was refined through a case study approach. Arguably, this approach could be considered inferior to other methods that involved extensive stakeholder involvement, such as the CIHR framework [ 8 ]. Nonetheless, 7 methodological frameworks were developed based upon the Payback Framework [ 8 , 29 , 31 – 35 ].

Limitations

The present review is the first to summarise systematically existing impact methodological frameworks and metrics. The main limitation is that 50% of the included publications were found through methods other than bibliographic databases searching, indicating poor indexing. Therefore, some relevant articles may not have been included in this review if they failed to indicate the inclusion of a methodological impact framework in their title/abstract. We did, however, make every effort to try to find these potentially hard-to-reach publications, e.g., through forwards/backwards citation searching, hand searching reference lists, and expert communication. Additionally, this review only extracted information regarding the methodology followed to develop each framework from the main publication source or framework webpage. Therefore, further evaluations may not have been included, as they are beyond the scope of the current paper. A further limitation was that although our search strategy did not include language restrictions, we did not specifically search non-English language databases. Thus, we may have failed to identify potentially relevant methodological frameworks that were developed in a non-English language setting.

In conclusion, the measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise benefit, and to help minimise research waste. This review provides a collective summary of existing methodological impact frameworks and metrics, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Supporting information

S1 appendix. search strategy..

https://doi.org/10.1371/journal.pmed.1002370.s001

S1 PRISMA Checklist. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.

https://doi.org/10.1371/journal.pmed.1002370.s002

Acknowledgments

We would also like to thank Mrs Susan Bayliss, Information Specialist, University of Birmingham, and Mrs Karen Biddle, Research Secretary, University of Birmingham.

• View Article
• PubMed/NCBI
• Google Scholar
• 3. HEFCE. REF 2014: Assessment framework and guidance on submissions 2011 [cited 2016 15 Feb]. Available from: http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf .
• 8. Canadian Institutes of Health Research. Developing a CIHR framework to measure the impact of health research 2005 [cited 2016 26 Feb]. Available from: http://publications.gc.ca/collections/Collection/MR21-65-2005E.pdf .
• 9. HEFCE. HEFCE allocates £3.97 billion to universities and colleges in England for 2015–1 2015. Available from: http://www.hefce.ac.uk/news/newsarchive/2015/Name,103785,en.html .
• 10. Stern N. Building on Success and Learning from Experience—An Independent Review of the Research Excellence Framework 2016 [cited 2016 05 Aug]. Available from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf .
• 11. Matthews D. REF sceptic to lead review into research assessment: Times Higher Education; 2015 [cited 2016 21 Apr]. Available from: https://www.timeshighereducation.com/news/ref-sceptic-lead-review-research-assessment .
• 12. HEFCE. The Metric Tide—Report of the Independent Review of the Role of Metrics in Research Assessment and Management 2015 [cited 2016 11 Aug]. Available from: http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/The,Metric,Tide/2015_metric_tide.pdf .
• 14. LSE Public Policy Group. Maximizing the impacts of your research: A handbook for social scientists. http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf . London: LSE; 2011.
• 15. HEFCE. Consultation on the second Research Excellence Framework. 2016.
• 18. Merriam-Webster Dictionary 2017. Available from: https://www.merriam-webster.com/dictionary/methodology .
• 19. Oxford Dictionaries—pathway 2016 [cited 2016 19 June]. Available from: http://www.oxforddictionaries.com/definition/english/pathway .
• 20. Oxford Dictionaries—metric 2016 [cited 2016 15 Sep]. Available from: https://en.oxforddictionaries.com/definition/metric .
• 21. WHO. WHO Ethical and Safety Guidelines for Interviewing Trafficked Women 2003 [cited 2016 29 July]. Available from: http://www.who.int/mip/2003/other_documents/en/Ethical_Safety-GWH.pdf .
• 31. Kalucy L, et al. Primary Health Care Research Impact Project: Final Report Stage 1 Adelaide: Primary Health Care Research & Information Service; 2007 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3338.pdf .
• 33. Canadian Academy of Health Sciences. Making an impact—A preferred framework and indicators to measure returns on investment in health research 2009 [cited 2016 26 Feb]. Available from: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf .
• 39. HEFCE. RAE 2008—Guidance in submissions 2005 [cited 2016 15 Feb]. Available from: http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf .
• 41. Royal Netherlands Academy of Arts and Sciences. The societal impact of applied health research—Towards a quality assessment system 2002 [cited 2016 29 Feb]. Available from: https://www.knaw.nl/en/news/publications/the-societal-impact-of-applied-health-research/@@download/pdf_file/20021098.pdf .
• 48. Weiss CH. Using social research in public policy making: Lexington Books; 1977.
• 50. Kogan M, Henkel M. Government and research: the Rothschild experiment in a government department: Heinemann Educational Books; 1983.
• 51. Thomas P. The Aims and Outcomes of Social Policy Research. Croom Helm; 1985.
• 52. Bulmer M. Social Science Research and Government: Comparative Essays on Britain and the United States: Cambridge University Press; 2010.
• 53. Booth T. Developing Policy Research. Aldershot, Gower1988.
• 55. Kalucy L, et al Exploring the impact of primary health care research Stage 2 Primary Health Care Research Impact Project Adelaide: Primary Health Care Research & Information Service (PHCRIS); 2009 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_8108.pdf .
• 56. CHSRF. Canadian Health Services Research Foundation 2000. Health Services Research and Evidence-based Decision Making [cited 2016 February]. Available from: http://www.cfhi-fcass.ca/migrated/pdf/mythbusters/EBDM_e.pdf .
• 58. W.K. Kellogg Foundation. Logic Model Development Guide 2004 [cited 2016 19 July]. Available from: http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf .
• 59. United Way of America. Measuring Program Outcomes: A Practical Approach 1996 [cited 2016 19 July]. Available from: https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf .
• 60. Nutley S, Percy-Smith J and Solesbury W. Models of research impact: a cross sector review of literature and practice. London: Learning and Skills Research Centre 2003.
• 61. Spaapen J, van Drooge L. SIAMPI final report [cited 2017 Jan]. Available from: http://www.siampi.eu/Content/SIAMPI_Final%20report.pdf .
• 63. LSHTM. The Health Risks and Consequences of Trafficking in Women and Adolescents—Findings from a European Study 2003 [cited 2016 29 July]. Available from: http://www.oas.org/atip/global%20reports/zimmerman%20tip%20health.pdf .
• 70. Russell G. Response to second HEFCE consultation on the Research Excellence Framework 2009 [cited 2016 04 Apr]. Available from: http://russellgroup.ac.uk/media/5262/ref-consultation-response-final-dec09.pdf .

• Open access
• Published: 16 June 2020

Defining and classifying public health systems: a critical interpretive synthesis

• Tamika Jarvis   ORCID: orcid.org/0000-0002-2823-2319 1 ,
• Fran Scott 1   na1 ,
• Fadi El-Jardali 1 , 2   na1 &
• Elizabeth Alvarez 1   na1  

Health Research Policy and Systems volume  18 , Article number:  68 ( 2020 ) Cite this article

24k Accesses

24 Citations

18 Altmetric

Metrics details

The introduction of the determinants of health has caused a shift towards understanding health from a holistic perspective as well as increased recognition of public health’s contributions to the health of the population. Several frameworks exist to conceptualise healthcare systems, highlighting the stark contrast of frameworks unique to public health systems. The objectives of this study were to define public health systems and assess differences between healthcare systems and public health systems within established health systems frameworks.

A critical interpretive synthesis was conducted. Databases searched included EBSCOhost, OVID, Scholars Portal, Web of Science, Cochrane Library and Health Systems Evidence. Data extraction, coding and analysis followed a best-fit framework analysis method. Initial codes were based on a current leading health systems and policy classification scheme – health systems arrangements (governance, financial and delivery arrangements).

A total of 5933 unique documents were identified and 67 were included in the analysis. Definitions of public health and public health systems varied significantly as did their roles and functions across jurisdictions. Public health systems arrangements generally followed those of health systems, with the addition of partnerships (community and inter-sectoral) and communication playing a larger role in public health. A public health systems framework and conceptualisation of how public health currently fits within health systems are presented.

Conclusions

Public health systems are unique and vital entities within health systems. In addition to examining how public health and public health systems have been defined within the literature, this review suggests that establishing the scope of public health is crucial to understanding its role within the larger health system and adds to the discourse around the relationship between public health, healthcare and population health. More broadly, this study addresses an important gap in understanding public health systems and provides conceptual and practical contributions as well as areas for future research.

Peer Review reports

Public health is generally understood to engage in population rather than in individual health activities and to undertake a population health approach recognising that genetic, behavioural and socio-economic factors (e.g. housing, social networks, education) influence health and well-being [ 1 , 2 ]. The introduction of the determinants of health has caused a shift towards understanding health from a holistic perspective as well as increased recognition of public health’s contributions to the health of the population [ 3 ]. Outside of global public health emergencies, such as Ebola or Zika Virus, attention to the role that public health plays in the protection and advancement of health has often taken a backseat to discussions of healthcare reform [ 4 , 5 ]. For many, health and the health system equate to healthcare, namely clinics and hospitals. In the context of the United States and Canada, considering, for example, that medical care consumes the largest amounts of the health budget in Canada and the United States, it is unsurprising that there is generally little public or political interest in strengthening or investing in public health systems [ 6 , 7 ]. For example, amid concerns that public health across Canada continues to be weakened through budgetary cuts and lack of investment in public health infrastructure, there remains little evidence related to understanding public health systems or what is currently done in practice in a comparative fashion [ 6 , 8 ]. Most public health research has focused on the evaluation of programmes aimed at individual or population-level interventions and understanding the causes and patterns of risk of ill health and disease rather than informing broader questions about the organisation, delivery or funding mechanisms of public health systems [ 7 , 9 , 10 , 11 ]. Amid the current novel coronavirus disease (COVID-19) pandemic, understanding how public health and broader health systems function, is crucial.

Health services and system researchers have not adequately acknowledged public health as a vital component and contributor to health systems, and achievements made by public health activities, such as communicable and non-communicable disease control, are often attributed to the delivery of primary healthcare services and advances in biomedical interventions [ 12 ]. While multiple health systems frameworks identify the components, functions and goals of healthcare systems, no clear or consistent definition of public health systems exist [ 13 , 14 ].

Defining public health systems can help determine how to best design systems and deliver programmes and services to support public health within the larger health system and other key institutions and move discussions about the relationship between public health and healthcare forward. As a first step, this paper addresses a priority research area that called for the development of a framework describing the key elements of public health systems [ 3 , 15 , 16 ]. A qualitative synthesis of the current literature was completed to investigate how public health systems have been defined and classified as well as the differences between healthcare systems and public health systems within established conceptual frameworks for health systems.

This qualitative review adopted the critical interpretive synthesis (CIS) approach as the overarching methodology while using a second and complimentary qualitative strategy, the best-fit framework synthesis (BFF), to guide structured data extraction and analysis. CIS differs from traditional systematic reviews in several ways; namely, (1) it is an iterative process that explicitly allows for the critical re-interpretation of existing literature and filling of conceptual gaps, and (2) it prioritises the inclusion of papers based on relevance to the research question, including grey literature, increasing the likelihood of capturing relevant documents [ 17 , 18 , 19 ]. The BFF is used to test, refine and/or generate relevant frameworks, theories or conceptual models using systematically retrieved empirical data. For this study, BFF was determined to be useful for the organisation, extraction and analysis of large amounts of data, as a priori or pre-identified codes allows researchers to utilise and generate codes and interpret themes but not be restricted by an existing framework, model or theory [ 20 ]. This study conformed with the recommended PRISMA guidelines [ 21 ] and was registered in PROSPERO (CRD42016049967).

Search strategy

Following pilot testing, the databases searched included EBSCOhost, OVID, Scholars Portal, Web of Science, Cochrane Library and Health Systems Evidence. The final search was conducted on 25 October 2016. Studies were not limited to date, language or study design. Additional sources were identified through reviewing the references of included documents to find relevant material and through internet searching to fill conceptual gaps using non-systematic searching. The initial search strategy can be found in Additional file  1 . As public health systems have not been clearly defined, our search strategy sought to include terms that may be used interchangeably within the literature but warrant clear definitions. For the purpose of this study, we define a system as “ a set of inter-connected parts that have to function together to be effective ” [ 22 ], a framework as “ a basic conceptual structure ” [ 23 ], a model as “ a standard or example for imitation for comparison ” [ 24 ], and classification as “ an arrangement of people or things into groups based on ways that they are alike ” [ 25 ].

Study selection

Records identified were screened, duplicates were removed, and titles and abstracts were independently reviewed for exclusion by two reviewers. Records were excluded that (1) did not describe local, state/provincial/territorial, or national public health systems, frameworks or critical components, (2) addressed publicly funded healthcare systems, unless it also addressed the role of public health, and (3) were in languages other than English, French or Spanish. Records were not restricted by date or country as we wanted to obtain a general picture of public health systems globally. To help address the compass question and maximise the diversity of papers, potentially relevant documents were purposively sampled and prioritised for inclusion if they were clearly relevant to the research topic, offered conceptual insights about full frameworks, and were able to provide a cross section of different jurisdictions [ 19 , 26 ]. Full-text documents were retrieved and assessed for eligibility with additional documents found through reference chaining of all included studies or internet searches to help fill conceptual gaps.

Data abstraction

A data extraction tool was developed to organise the key themes of relevant documents and bibliographic information, including title of document, author(s), study type, context of study, key topic areas, and further relevant references from paper. Documents were imported into NVivo 11 software to facilitate the coding and organisation of data. Seven documents were randomly selected and coded independently by two reviewers to ensure consistency. Disagreements were resolved by consensus.

Data analysis and synthesis

A current leading health systems and policy classification scheme, the health systems arrangements framework (based on the three key building blocks of governance, financial and delivery arrangements), was used to form the initial a priori codes [ 27 ]. Originally developed as a taxonomy of health system topics to classify documents for Health Systems Evidence, this framework was chosen as the BFF theoretical framework because it is comprehensive (e.g. includes the essential components of WHO’s health system building blocks [ 22 ]), easy to understand, and has been used in various international contexts for health systems and policy research and applied work, for example, to develop health systems guidance documents and to contextualise research for evidence-based decision-making in Peru and Uganda [ 28 , 29 , 30 ]. Codes were added inductively from the data and were linked into themes. Data analysis continued until there was data saturation and conceptual gaps were addressed [ 31 ].

Search results and study selection

Electronic database searches identified 5933 unique citations, 338 of which met inclusion criteria after title and abstract exclusion; 81 of these documents were purposively sampled and full-text review excluded 23 records. Nine additional documents were attained through reference chaining and internet searches. In total, 67 documents were included (Fig.  1 ). A description of these documents can be found in Additional file  2 .

PRISMA flow chart for inclusion/exclusion of documents [ 21 ]

The results are presented in four sections – defining public health and public health systems; roles and functions of public health; public health systems; and public health within health systems. High-level findings are presented in the text and more details are provided in the Additional files.

Defining public health and public health systems

Most documents defined public health via its functions; therefore, separate sections were created for defining public health and public health systems and for describing the roles and functions of public health.

• Public health

Seven definitions of public health were found (Additional file  3 ). Public health was described as a multidisciplinary area of practice, concept and set of values that engaged in a larger population health perspective. Eight documents used the definition of public health provided by WHO as an art and science whose organised efforts aim to prevent illness and disease as well as to protect and promote health within society [ 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 ]. Five other definitions expanded or emphasised various priorities within public health practice and included values of equity and equality.

Public health systems

Public health systems were defined in 20 documents, with 10 unique definitions identified (Additional file 3 ). Eleven documents defined public health systems as all levels of governmental and non-governmental entities which share in the responsibility for ensuring healthy social and physical environments, and consist of a variety of organisations that contribute to the core functions of public health to protect and promote health within the community [ 12 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ]. Public health systems were also defined based on their composition, level of service, contributing actors, mission and activities, or a combination of these. Public health was largely seen as a governmental responsibility and included partnerships between formal (government) and informal (private sectors, volunteer) organisations.

Roles and functions of public health

Subsystem models of delivery, governance, finance, and roles and functions were identified, for example, Mays et al.’s [ 45 ] typology of public health delivery systems, but none provided a comprehensive public health system framework. Several frameworks identified essential public health functions (Additional file  4 ). The most frequent frameworks were the Institute of Medicine’s three ‘core’ public health functions of assessment, policy development and assurance [ 12 , 38 , 42 , 47 , 48 , 50 ] and the ‘Ten Essential Public Health Services’, which were developed to further refine the specific set of functions and services within public health systems in the United States but have been adapted elsewhere [ 40 , 42 , 44 , 48 , 51 , 52 , 53 , 54 , 55 , 56 ].

Roles and functions

Thirty-nine documents defined or highlighted what they identified as the ‘essential’ functions of public health. Additional file 4 provides a table as a way of organising what functions were found within the literature across a variety of countries, following the three core public health functions and 10 essential services. The following were listed as public health functions and services in more than half of the documents: health promotion ( n  = 30); health protection, which included air, water, and food quality and inspection as well as environmental and occupational health activities ( n  = 26); investigation and surveillance ( n  = 25); emergency planning, preparedness and response ( n  = 25); health assessment and monitoring ( n  = 24); injury and chronic disease prevention and management ( n  = 23); and linking with and providing personal clinical services, which included maternal and child health services, minority, rural, indigent, mental, clinical and community health improvement activities, to targeted and/or vulnerable populations ( n  = 22). Communicable disease control ( n  = 18); research ( n  = 16); regulation and enforcement ( n  = 16); resource and organisational management, including leadership, governance capacity, resource management and development of organisational structures ( n  = 14); establishment of partnerships and advocacy in communities ( n  = 13); evaluation of health services ( n  = 11); policy development and planning ( n  = 11); workforce strengthening ( n  = 9); programme implementation ( n  = 4); laboratory services ( n  = 3); hospital and long-term care facility licensing ( n  = 2); and vital statistics ( n  = 2) were also identified as being the responsibility of public health in various jurisdictions. Functions and services had to be interpreted and summarised, as there were different terms being used to represent the same activities between jurisdictions. For example, health protection and environmental health were both used to describe the responsibility for testing and monitoring the quality of air, food and water, and population health assessment was used to describe monitoring, surveillance or epidemiological activities.

It was found that public health system descriptions fit the health system arrangements framework well, with the addition of partnerships and communication, which affected each of the other parts of the system (Additional file  5 ). The health systems arrangements framework was refined to highlight differences between terms and components of public health systems (Table  1 ). While little to no evidence on certain features, such as commercial authority, remunerating providers and incentivising consumers, was available within the literature, these arrangements are still applicable to public health, and thus remain within the public health systems framework.

Governance arrangements

Policy authority.

Four levels of policy authority were identified within the public sector, namley international, national, state/provincial/territorial, and local. The degree of decentralisation within a country or state/province determined the responsibilities and structural organisation of agencies within public health systems [ 4 , 12 , 32 , 34 , 37 , 44 , 46 , 49 , 57 , 58 , 59 , 60 ]. Most national public health agencies were primarily responsible for providing guidance and acting as a source of expertise while giving states/provinces authority to organise public health [ 4 , 32 , 39 , 48 , 54 , 57 , 61 , 62 , 63 , 64 ]. Many state/provincial governments established overall priorities, strategic direction, policies, strategies, standards, and funding models for local public health agencies [ 4 , 34 ].

Organisational authority

Regional or local health units planned and implemented the majority of services, developed policies and communicated legislation [ 34 , 57 , 61 , 63 , 65 ].

Consumer and stakeholder involvement

In public health systems, consumers most often referred to targeted populations and communities rather than individuals, as is more common in healthcare systems. Stakeholders included other public sectors, communities, service providers in and outside of the health system, the private sector, and individuals [ 63 ]. Community partnerships and public engagement were identified as being important for individual and community health, accountability, and an influential factor in the operation of local public health agencies [ 33 , 41 , 47 , 56 , 61 , 63 , 66 , 67 ].

Delivery arrangements

In public health systems, the terms ‘programmes’ or ‘services’ seemed to better reflect the wide range of activities and roles of public health within the larger health system than the term ‘care’.

How are programmes and services designed to meet consumers’ needs

Public health functions were carried out by all levels of government, including federal, state/provincial/territorial and local, but most activities remained organised at the state/provincial level or locally in many countries [ 2 , 38 , 39 , 40 , 48 , 58 , 59 , 62 , 63 , 68 , 69 , 70 , 71 ]. Delivery of public health services often rested at the local level but, in some instances, were delivered at the state/provincial level or through separate government or private organisations [ 38 , 48 , 59 , 62 ]. In one United States-based example, although public health and healthcare were largely independent of one another, public health increasingly provided personal health services for pre-identified or vulnerable groups [ 72 ].

By whom are programmes and services provided

Most public health programmes/services were provided by public sector employees as part of a public health unit, as well as faith-based groups, private businesses, social services agencies, schools, workplaces and healthcare providers [ 46 , 73 ]. Healthcare and other sectors support public health in its missions by participating in surveillance, health protection and emergency planning activities [ 37 , 73 , 74 ]. Because of the diversity in the organisations and people involved in providing public health programmes and services, the size of a public health workforce is difficult to determine [ 5 , 57 , 63 , 73 , 75 ].

Where are programmes and services provided

Delivery of public health programmes and services occurs in multiple public and private settings, including schools, homes of private citizens, workplaces, clinics, public health laboratories, local public health agencies and offices, and various indoor and outdoor spaces within the community [ 57 , 61 ]. Partnerships and contracts with non-governmental and community organisations in public and private sectors have often been established to circumvent barriers to service provision (e.g. due to geographical location or size of jurisdictions) [ 40 , 49 , 71 ].

With what supports are programmes and services provided

Support was often referred to as capacity in human health resources [ 2 , 40 , 51 , 57 , 59 ] and information technology [ 62 , 75 ]. A few articles discussed the use of technology as a tool used to deliver and support public health activities and messaging, and included services such as eHealth, web portals, mobile phone applications and social media [ 66 ].

Financial arrangements

It was difficult to estimate the direct and indirect financial contributions by public and private sectors given the diversity in public health activities [ 39 ].

Financing systems

Several sources estimated that, on average, public health systems received between 3% and 8% of the national health budget [ 5 , 32 , 38 , 39 , 44 , 51 , 57 , 72 ]. Like healthcare, public and private funding sources existed in these systems, with many being publicly financed through general taxation, including federal, state/provincial and local taxes such as income, property and sales taxes [ 32 , 39 , 47 , 52 , 57 , 58 , 59 , 62 , 69 , 76 ]. Private sector financing included out-of-pocket service fees and for-profit and non-profit organisations [ 44 , 47 , 52 , 59 ]. A significant part of public health funding is derived from external donors, particularly for disease-specific initiatives, in low-income countries [ 58 ].

Funding organisations

Revenue transfers from national governments to state/provincial or local public health agencies, with funding being distributed to local health agencies to deliver services, were most prevalent. Funding was largely allocated by funding formulas; however, a combination of funding mechanisms, such as activity- and standard-specific funding and reimbursements, per capita allocations, competitive and needs-based grants, and performance-based funding were also reported [ 38 , 46 , 47 , 55 , 58 , 59 ]. Other sources of funding originated from other public sector partners and from collaborations between public and private sectors [ 39 , 47 , 59 , 65 ]. In some instances, external donors allocated funds to community-based organisations to target specific community health needs or provided informal funding for ‘non-essential’ public health programmes [ 4 , 12 , 39 , 47 , 63 , 65 ].

Purchasing products and services

Funding organisations and purchasing products and services were strongly linked. Many federal and state/provincial governments allocated funds for specific public health activities, which influenced the availability of services [ 38 , 46 , 47 , 58 , 59 ]. Generally, there was a trend towards a substantial portion of public health funding directed at individual clinical services (e.g. maternal and child health, mental health, prenatal visits, family planning) [ 12 , 60 ].

Partnerships and communication

Partnerships were identified as an essential way to extend the reach of programmes to target population health issues and to share expertise, information and resources [ 2 , 4 , 39 , 40 , 46 , 56 , 59 , 65 , 66 , 68 , 77 , 78 , 79 , 80 , 81 ]. Partners included other local, national and international government agencies, the healthcare system, academic centres, private sector businesses, faith groups, foundations, service organisations and communities [ 32 , 33 , 34 , 40 , 45 , 46 , 48 , 52 , 54 , 56 , 62 , 63 , 64 , 72 , 79 , 82 , 83 ]. Within governance arrangements, the goal of partnerships was identified as community empowerment and capacity-building for successful interventions [ 4 , 56 , 61 , 74 , 79 , 82 , 84 ]. Engagement within communities reportedly increased stakeholder involvement in policy and decision-making [ 82 , 84 ].

Public health is an information-dependent sector that requires constant information exchange in order to support public health functions, activities and policies, especially in emergency planning and response [ 2 , 58 ]. Communication not only improves surveillance and response systems between all levels of government and internationally but is necessary for effective knowledge translation activities [ 5 , 84 , 85 ]. Clear, consistent and timely communication is essential for delivering messages to the public, preventing mixed messages and encouraging public engagement. Current and evolving technology, such as the internet and other mass media sources, are tools that support this effort by improving health literacy and outreach [ 66 ] but can also spread misinformation.

Public health within health systems

In framing how public health is seen as part of a larger health system, the literature pointed to two related but separate concepts – that of system integration and the role of public health in promoting a population health approach.

Sofaer [ 79 ] states that the best way to judge how effective a health system is, is by how well it can improve the health of individuals and populations. Interest in integrating public health and healthcare systems is not new [ 34 , 48 , 76 ]. While definitions of integration vary, integration in this report is the relationship between public health and healthcare and the extent to which services are provided to promote and achieve health. Integration is believed to bring the two systems closer together to provide a seamless service delivery within the larger health system and better respond to the needs of both individuals and communities [ 33 , 65 ]. Potential benefits of integration include bringing a population health perspective to the healthcare system, increased access to care, and the reduction of direct and indirect healthcare costs [ 4 ]. However, the literature also points to various challenges regarding integration and what it might mean for the future of the public health system. These include the loss of public health authority and expertise, capacity and management of competing priorities, consequently linked to adverse health outcomes [ 58 ]. Over time, the diversion of public health resources to primary care, loss of positions in public health units, and loss of linkages to community partners and communities would hinder public health from being able to extend the reach of its activities and lead to fragmentation in programme delivery and in the services necessary to protect the health of the population such as community health assessments, programme planning, and disease control and surveillance [ 9 , 58 , 86 ].

From the literature, healthcare and public health are separate systems, often with their own aims and functions, governance and financial systems, and ways of delivering services, although significant overlap has been observed, particularly within delivery arrangements. The health system is separate from but influenced by the larger political and social systems. Health, within this health system, is determined by individual factors and access to and use of public health and healthcare services. Yet, the wider determinants of health recognise the importance of social and political factors on health. Figure  2 aims to organise how public health currently fits within a health system. It is important to note that public health is often equated to and touted as a steward of population health. Population health is the driving force behind public health. Its upstream focus, following an ecological model of health, is concerned with how individual, social and environmental determinants influence health outcomes [ 53 , 55 ]. A strength of the population health approach is that it recognises that people are not passive but are active participants in their own health outcomes. Individual health is supported by both public health and healthcare activities and by how individuals interact with these systems and their larger social environments. There is a constant exchange between individuals, healthcare, public health, and the political and social systems they are embedded in, with more resources, programmes and services targeted towards those identified as vulnerable to try and establish a level of equity in health outcomes. It could be argued that, while activities in public health are population based, the ultimate target of public health is still to support individual health within the larger community. For example, although health promotion messages and activities are delivered to the population, the goal of these activities is to encourage individuals within communities towards healthier lifestyles (e.g. tobacco cessation, vaccinations), whose health statistics are then tracked (e.g. surveillance) and regarded as the population’s health.

Conceptual fit of public health systems within current health systems

Population health is conceptualised as extending far beyond the health system to include the political and societal contexts. While policies outside of the public health system may not be implemented to directly impact population health, they often do. For example, taxes on carbon emissions have short- and long-term effects on population-level health outcomes. Similarly, public health systems affect, and are affected by, many sectors. As broader determinants of health are becoming increasingly recognised as influential, there has been an increased emphasis on holistic approaches to healthy public policies [ 9 ]. Recent work has focused on holistic approaches to health such as Health in All Policies and One Health. Health in All Policies refers to the intersectoral aim of integrating health considerations into the actions, interventions and policies outside of the health sector, and One Health refers to the approach that recognises that human health is influenced by both animal and environmental health [ 87 , 88 ]. Figure 2 highlights the gaps that exist between what we currently have, at least in high-income countries, and paradigms of population health, Health in All Policies and, especially, One Health. For example, public health is often separate from healthcare and from the political process (i.e. healthy public policies). Population health spans further than public health’s reach and integration may need to be reconceptualised to align with a broader vision of health.

Main findings

The synthesis suggests that public health systems have not been clearly defined because (1) public health systems have been conceptualised in various ways and (2) there is overlap in terminology with publicly funded healthcare systems. One further potential reason for the lack of clarity regarding definitions and the change in the use of terms over time could be related to funder and publication preferences. No comprehensive public health system frameworks were identified within the literature although there was significant emphasis on defining the essential roles and functions of public health. These are broad and consensus on essential functions is often absent between jurisdictions, which made comparisons challenging. Services not provided by healthcare systems are often taken up by public health, increasing pressures on already limited budgets. In addition, response to health emergencies appears to have largely been adopted by public health systems because they are most likely to possess the capacity and expertise to organise and respond to large-scale events or threats to health. We found that, while many components of public health systems fit under the governance, delivery and financial arrangements of traditional healthcare systems, there are noted differences, specifically related to the role of partnerships and communication within public health. Partnerships provided the structure for multi-sectoral collaboration and facilitated communication and information exchange to accomplish the core functions of public health. A proposed framework for public health systems is presented in this paper.

The argument is made that public health and healthcare share the common goal of supporting the health of individuals within populations. Integration, the intersection of public health and healthcare, is believed to bring the two systems closer together to provide seamless service delivery within the broader health system that better responds to the needs of both individuals and communities [ 33 , 65 ]. The challenge with developing these integrated health systems is determining how to best align financial, governance and delivery arrangements, ensuring both complementarity and positive health outcomes. As population health extends beyond the health system to include the political, environmental and societal contexts, as such, it is important to understand these larger contexts within which health systems operate [ 89 ]. The conceptualisation of the current fit of public health systems within health systems has two important ramifications. The first being that integration of healthcare and public health will be difficult at best given that the aims, governance, finance and service delivery are not often aligned. There would have to be significant incentives for integration to happen and even with that, there may not be a shared vision of health to drive collaboration between these systems. The second ramification is that public health will not be able to inform healthy public policies unless they have a seat at the decision-making table for policies outside of the public health realm. There may be arguments on both sides as to whether this is practical or desired, but the case can be made that population health, encompassing the broader social determinants of health, will not occur within the current paradigm and structures.

Strengths and limitations

The flexibility of the CIS approach allows for a broad sampling frame and iterative filling of conceptual gaps. The BFF approach provides a structured approach to data analysis but also for change if a more applicable model is identified. The combination of these two qualitative approaches allowed for a broad research question in an area that is not well defined and helped bring a lot of data together in an efficient manner. The study was informed by a diverse team of experts in public health, health systems research and qualitative research methodology. The search strategy may not have captured all terms and concepts regarding public health systems. To try and mitigate this, a search string was developed with broad search terms to identify as much relevant literature as possible. Additionally, as the reviewed literature mainly covers the period from 2000 to 2016, the search strategy may not have captured all relevant documents such as recent institutional reports. While literature addressing health systems have origins before 2000, more recent documents were purposively sampled and prioritised in the document selection process as they would expectedly include earlier relevant works. This also presents an opportunity to review works after 2016, such as the 2018 report on the organisation and financing of public health services in Europe, and more recent documents that highlight the importance of improving population health through the Sustainable Development Goals (SDGs) [ 90 , 91 ]. CIS requires constant reflexive analysis by the principal investigator and results may vary if another person were to replicate this study; however, the use of a priori codes was used to increase transparency. Finally, although public health systems from various countries were reviewed, almost all documents were from high-income countries, mostly originating from the United States and Canada. While some of the results may be equally relevant to systems outside of these Western contexts, the findings highlight the need for future research outside of these jurisdictions, for example, in low- and middle-income countries, particularly in light of the SDGs.

Placing this work within the literature

This study is a first attempt at defining a holistic public health systems framework and highlighting the differences and similarities between public health and healthcare system arrangements. We have specifically addressed a priority research theme proposed by Canadian and United States federal agencies, such as the Canadian Institutes of Health Research Institute of Population and Public Health, the Centers for Disease Control and Prevention, and other stakeholders, to describe dimensions of public health systems and conceptualise a framework for public health systems.

Practice and policy implications

This study suggests five considerations for practice and policy. First, defining public health systems solidifies and challenges public health’s role to encourage political interest to secure the investment necessary to improve health system capacity. The COVID-19 pandemic has demonstrated the importance of understanding the role of public health within the larger health system, particularly with respect to the capacity to respond to public health crises efficiently and effectively. As was observed following the 2004 SARS pandemic, it is expected that there will be an increase in renewed calls and discussion around public health systems strengthening. Second, the discourse around establishing essential functions of public health is enhanced. Our synthesis has identified a growing concern that public health is currently filling gaps within the healthcare system by providing clinical services to targeted or vulnerable groups, consuming vast amounts of both human and financial resources from already under-resourced public health systems [ 5 , 60 , 69 ]. Thus, defining public health and the boundaries of public health systems could be an important step towards measuring performance and preventing public health systems from becoming too overburdened from the increasing scope of public health clinical activities [ 48 , 54 ]. Third, this study has reinforced the importance of partnerships in the work of public health. Partnerships have the potential to form and navigate systems in contexts challenged by limited resources. For example, the United Nations Millennium Development Goals, aimed to tackle societal issues influencing health, such as poverty, education, and gender equality and, while substantial progress was made, the state of many health systems revealed barriers to reaching specific targets and delivering services to the most vulnerable, particularly for those in many low- and middle-income countries. The SDGs, whose agenda is broader and more ambitious, explicitly recognise the broader determinants of health, by establishing social, economic and environmental objectives such as climate action, sustainable cities and communities, economic development, and social inclusion [ 92 ]. Defining public health systems serves as a building block for under-developed or transitioning public health systems and services, whereby determining roles and functions of public health systems allows practitioners to identify areas that require strengthening. Fourth, this synthesis suggests that the critical differences between public health and healthcare systems need to be acknowledged and negotiated for integration to be successful. The gaps that exist between the public health and healthcare subsystems have been highlighted. Lastly, the idea that public health is the champion of population health is presented as a challenge. Population health is influenced by political and social factors outside of the public health system. The idea of public health as the steward of population health requires serious consideration, especially if public health continues to be excluded from the decision-making process and its role in protecting and promoting health is relegated to the background.

Research implications

By developing a method that combined the best aspects of two qualitative systematic review methods, CIS and BFF, we were able to bring substantial amounts of data together in a timely manner, while simultaneously testing and refining a well-known framework in a critical way. The use of other frameworks, such as the performance-based conceptual framework by Handler et al. [ 42 ], or an assessment framework identified by Martin-Moreno et al. [ 53 ], provide other ways to examine public health systems and could be a way to validate the findings of this study or illustrate different health system ideas. The performance of public health systems cannot be measured if definitions, functions and key components are not well defined. The numerous variations in terminology make it difficult to perform a comparative analysis of public health systems across jurisdictions. Similarly, the differences in defined functions, or a lack thereof, limits our ability to monitor quality indicators between systems. Furthermore, the lack of research in public health and public health systems hampers both interest and investment in public health and limits the development of recommendations for evidence-based practice. Further research is required to determine what integration might look like and at what systems levels integration might work best. This presents additional opportunities for future research, particularly with respect to the gaps highlighted within financial arrangements such as remunerating providers and incentivising consumers. Lastly, this study is a first attempt at trying to understand how public health systems have been conceptualised. A public health systems framework (Table 1 ) and conceptualisation of how public health currently fits within the larger health system (Fig.  2 ) are proposed and can be applied and tested in real life settings as well as used to guide further research and practice in public health systems.

The aim of this paper was to examine the literature on how public health systems have been defined and classified and to illustrate how current public health systems align within established conceptual frameworks for health systems. Defining the scope of public health systems is crucial to solidify public health’s role as part of the larger health system and the degree to which public health and healthcare systems are different should be understood if public health is to be able to attend to its primary mandate within integrated health systems. While there is increased movement towards health systems focused on population health, specifically the increased focus on Health in All Policies and One Health, many gaps exist to reach those aims.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its additional files.

Abbreviations

Best-fit framework synthesis

Critical interpretive synthesis

Sustainable Development Goals

Levesque J-F, Bergeron P, Roy DA. The interaction of public health and primary care: functional roles and organizational models that bridge individual and population perspectives. Public Health Rev. 2013;35(1):1.

Article   Google Scholar  

Moloughney BW. The renewal of public health in Nova Scotia: building a public health system to meet the needs of Nova Scotians: Health Promotion and Protection; 2006. http://novascotia.ca/dhw/publichealth/documents/Renewal-of-Public-Health-Report.pdf . Accessed 7 June 2020.

Scutchfield FD, Ingram RC. Public health systems and services research: building the evidence base to improve public health practice. Public Health Rev. 2013;8:35.

Google Scholar  

Lozon JC, Alikhan LM. Canada’s public health system: is the pace of progress sufficient? Healthc Pap. 2007;7(3):52–9.

Article   PubMed   Google Scholar  

Kruk ME. Emergency preparedness and public health systems lessons for developing countries. Am J Prev Med. 2008;34(6):529–34.

Article   PubMed   PubMed Central   Google Scholar  

Guyon A, Hancock T, Kirk M, MacDonald M, Neudorf C, Sutcliffe P, et al. The weakening of public health: a threat to population health and health care system sustainability. Can J Public Health. 2017;108(1):e1–6.

Scutchfield FD, Marks JS, Perez DJ, Mays GP. Public Health Services and Systems Research. Am J Prev Med. 2007;33(2):169–71.

Guyon A, Perreault R. Public health systems under attack in Canada: evidence on public health system performance challenges arbitrary reform. Can J Public Health. 2016;107(3):e326–9.

Breton M, Denis J-L, Lamothe L. Incorporating public health more closely into local governance of health care delivery: lessons from the quebec experience. Can J Public Health. 2010;101(4):314–7.

Kaluzny AB, Halverson PK, Miller CA, Fried B, Schenck SE, Richards TB. Performing public health functions: the perceived contribution of public health and other community agencies. J Health Hum Serv Adm. 1996;18:288–303.

PubMed   Google Scholar  

Moloughney BW. Public health medicine, public health practice, and public health systems. Can J Public Health. 2013;104(2):e115–6.

Beitsch L, Brooks R, Menachemi N, Libbey PM. Public health at center stage: new roles, old props. Health Aff. 2006;25(4):911–22.

Shakarishvili G, Atun R, Berman P, Hsiao W, Burgess C, Lansang MA. Converging health systems frameworks: towards a concepts-to-actions roadmap for health systems strengthening in low and middle income countries. Glob Health Gov. 2010;3(2):1-17.

Hoffman, Røttingen J-A, Bennett S, Lavis JN, Edge JS, Frenk J. A review of conceptual barriers and opportunities facing health systems research to inform a strategy from the World Health Organization. Geneva: Alliance HPSR, WHO; 2012.

Di Ruggiero E, Frank J, Moloughney B. Strengthen Canada’s public health system now. Can J Public Health. 2004;95(1):5.

Lenaway D, Halverson P, Sotnikov S. Public health systems research: setting a national agenda. Am J Public Health. 2006;96(3):410–3.

Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11:29. https://doi.org/10.1186/1471-2288-11-29 .

Kastner M, Tricco AC, Soobiah C, Lillie E, Perrier L, Horsley T, et al. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12:114.

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35.

Carroll C, Booth A, Leaviss J, Rick J. ‘Best fit’ framework synthesis: refining the method. BMC Med Res Methodol. 2013;13:37.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097 .

World Health Organization. Everybody’s business: strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: WHO; 2007.

Framework. The Merriam-Webster.com Dictionary. http://www.merriam-webster.com/dictionary/framework . Accessed 7 June 2020.

Model. The Merriam-Webster.com Dictionary. https://www.merriam-webster.com/dictionary/model . Accessed 7 June 2020.

Classification. The Merriam-Webster.com Dictionary. http://www.merriam-webster.com/dictionary/classification . Accessed 7 June 2020.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Lavis JN, Røttingen J-A, Bosch-Capblanch X, Atun R, El-Jardali F, Gilson L, et al. Guidance for evidence-informed policies about health systems: linking guidance development to policy development. PLoS Med. 2012;9(3):e1001186.

Ciapponi A, Lewin S, Herrera CA, Opiyo N, Pantoja T, Paulsen E, et al. Delivery arrangements for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9(9):CD011083 http://doi.wiley.com/10.1002/14651858.CD011083.pub2 .

Alvarez E, Lavis JN, Brouwers M, Schwartz L. Developing a workbook to support the contextualisation of global health systems guidance: a case study identifying steps and critical factors for success in this process at WHO. Health Res Policy Syst. 2018;16:19.

Alvarez E, Lavis JN, Brouwers M, Carmona Clavijo G, Sewankambo N, Solari L, et al. Developing evidence briefs for policy: a qualitative case study comparing the process of using a guidance-contextualization workbook in Peru and Uganda. Health Res Policy Syst. 2019;17:89. https://doi.org/10.1186/s12961-019-0488-0 .

Dey I. Qualitative data analysis: a user-friendly guide for social scientists. London; New York: Routledge; 1993.

Public Health Agency of Canada. Learning from SARS: renewal of Public Health in Canada: Health Canada; 2004. http://www.phac-aspc.gc.ca/publicat/sars-sras/naylor/3-eng.php . Accessed 7 June 2020.

White F. Primary health care and public health: foundations of universal health systems. Med Princ Pract. 2015;24(2):103–16.

Ministry of Health and Long-Term Care. Initial Report on Public Health. Ministry of Health and Long-Term Care. Ontairo; 2009. https://collections.ola.org/mon/23008/295012.pdf . Accessed 7 June 2020.

Knight TM, Birt CA, Bocsan I, Armitage LE. The public health function in central and eastern Europe. Public Health. 2003;117(2):98–105.

Article   CAS   PubMed   Google Scholar  

Alwan A, Puska P, Siddiqi S. Essential public health functions for countries of the Eastern Mediterranean Region: what are they and what benefits do they offer? East Mediterr Health J. 2015;21(12):859–60.

Moore S, Mawji A, Shiell A, Noseworthy T. Public health preparedness: a systems-level approach. J Epidemiol Community Health. 2007;61(4):282–6.

Wall S. Transformations in public health systems. Health Aff. 1998;17(3):64–80.

Article   CAS   Google Scholar  

Allin S, Mossialos E, McKee M, Holland WW, World Health Organization. Making decisions on public health: a review of eight countries. Copenhagen: WHO Regional Office for Europe; 2004.

Centers for Disease Control and Prevention. Public health’s infrastructure: a status report. Bethesda: Department of Health and Human Services; 2001.

Ellison JH. National Public Health Performance Standards: are they a means of evaluating the local public health system? J Public Health Manag Pract. 2005;11(5):433–6.

Handler A, Issel M, Turnock B. A conceptual framework to measure performance of the public health system. Am J Public Health. 2001;91(8):1235–9.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Hunter D. The state of the public health system in England. Public Health. 2008;122(10):1042–6.

Hyde JK, Shortell SM. The structure and organization of local and state public health agencies in the U.S. Am J Prev Med. 2012;42(5):S29–41.

Mays GP, Scutchfield FD, Bhandari MW, Smith SA. Understanding the organization of public health delivery systems: an empirical typology. Milbank Q. 2010;88(1):81–111.

Mays GP, Smith SA, Ingram RC, Racster LJ, Lamberth CD, Lovely ES. Public health delivery systems evidence, uncertainty, and emerging research needs. Am J Prev Med. 2009;36(3):256–65.

Roper WL, Baker EL, Dyal W, Nicola R. Strengthening the public health system. Public Health Rep. 1992;107(6):609–15.

CAS   PubMed   PubMed Central   Google Scholar  

Scutchfield FD, Miron E, Ingram RC. From service provision to function based performance - perspectives on public health systems from the USA and Israel. Isr J Health Policy Res. 2012;1:46.

Wholey DR, Gregg W, Moscovice I. Public health systems: a social networks perspective. Health Serv Res. 2009;44(5):1842–62.

Welton WE, Kantner TA, Katz SM. Developing tomorrow’s integrated community health systems: a leadership challenge for public health and primary care. Milbank Q. 1997;75(2):261–88.

Centers for Disease Control and Prevention. Changes in the public health system. MMWR Morb Mortal Wkly Rep. 1999;48(50):1141–7.

Halverson PK, Berkowitz B. Public health finance: a conceptual framework. J Public Health Manag Pract. 2004;10(5):377–82.

Martin-Moreno JM, Harris M, Jakubowski E, Kluge H. Defining and assessing public health functions: a global analysis. Annu Rev Public Health. 2016;37:335–55.

Mays GP, McHugh M, Shim K, Perry N, Halverson PK, Lenaway D, et al. Identifying dimensions of performance in local public health systems: results from the National Public Health Performance Standards Program. J Public Health Manag Pract. 2004;10(3):193–203.

Moulton AD, Baker EL, Blumenstock JS, Jensen J, Morris RD. Building the legal foundation for an effective public health system. J Law Med Ethics. 2002;30(3):48–51.

Zahner SJ. Local public health system partnerships. Public Health Rep. 2005;120(1):76–83.

Lavis JN, editor. Ontario’s health system: key insights for engaged citizens, professionals and policymakers. Hamilton: McMaster Health Forum; 2016.

World Health Organization. Essential public health functions: a three-country study in the Western Pacific Region. Geneva: WHO; 2003. https://iris.wpro.who.int/handle/10665.1/5437 . Accessed 12 June 2020.

Potter M, Fitzpatrick T. State funding for local public health: observations from six case studies. J Public Health Manag Pract. 2007;13(2):163–8.

Beitsch L, Grigg M, Menachemi N, Brooks R. Roles of local public health agencies within the state public health system. J Public Health Manag Pract. 2006;12(3):232–41.

Sutcliffe PA, Deber RB, Pasut G. Public health in Canada: a comparative study of six provinces. Can J Public Health. 1997;88(4):246–9.

Regan S, MacDonald M, Allan DE, Martin C, Peroff-Johnston N. Public health human resources: a comparative analysis of policy documents in two Canadian provinces. Hum Resour Health. 2014;12:13–24.

Kothari A, Edwards N. Who is Protecting the Canadian Public Health System? Home Health Care Manag Pract. 2003;15(5):391–8.

Das Gupta M, Rani M. India’s public health system: how well does it function at the national level? In: World Bank Policy Research Working Paper No. 3447; 2004. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=625325 . Accessed 7 June 2020.

Williams D. Infectious diseases prevention and control in Ontario: continuing the investment in public health in 2008. Ministry of Health and Long-Term Care; 2008.  https://collections.ola.org/ser/258918/2008.pdf . Accessed 7 June 2020.

Taylor G. The chief public health officer’s report on the state of public health in Canada, 2014: public health in the future: Public Health Agency of Canada; 2014. http://publichealth.gc.ca/CPHOReport . Accessed 7 June 2020.

Lenihan P. The public health system: an idea whose time has come. J Public Health Manag Pract. 2005;11(2):165–7.

Beitsch L, Brooks R, Grigg M, Menachemi N. Structure and functions of state public health agencies. Am J Public Health. 2006;96(1):167–72.

Fooks C, Maslove L. A public health wake-up call for Canada. Health Policy Monit. 2003.

Madamala K, Sellers K, Beitsch LM, Pearsol J, Jarris PE. Structure and functions of state public health agencies in 2007. Am J Public Health. 2011;101(7):1179–86.

Parker AM, Shelton SR, Morganti KG, Nelson C. Assessing relationships between state and local public health organizations: evidence from the NACCHO 2008 profile of local health departments. J Public Health Manag Pract. 2012;18(2):156–9.

Fielding J, Luck J, Tye G. Reinvigorating public health core functions: restructuring Los Angeles county’s public health system. J Public Health Manag Pract. 2003;9(1):7–15.

Basrur S. Building the foundation of a strong public health system for Ontarians. Ministry of Health and Long-Term Care; 2005. https://collections.ola.org/ser/258918/2005.pdf .

Hunter M, Hunter JC, Yang JE, Crawley AW, Aragon TJ. Public health system response to extreme weather events. J Public Health Manag Pract. 2016;22(1):E1–10.

Guyon A. Canada’s public health systems: Quebec. CPHA-PHPC webinar on Canadian Public Health Systems. Vancouver; 2015. https://www.cpha.ca/canadas-public-health-system . Accessed 10 June 2020.

Baird J, Carlson K. National Public Health Performance Standards Assessment: first steps in strengthening North Dakota’s public health system. J Public Health Manag Pract. 2005;11(5):422–7.

Ministry of Health and Long-Term Care. Standards for public health programs and services: consultation document. Ontario: Ministry of Health and Long-Term Care; 2017.

Mays GP, Scutchfield FD. Improving public health system performance through multiorganizational partnerships. Prev Chronic Dis. 2010;7(6):A116.

PubMed   PubMed Central   Google Scholar  

Sofaer S. Challenges for the public in negotiating the health system in the 21st century. J Urban Health. 1999;76(2):211–28.

Rodriguez HP, Chen J, Owusu-Edusei K, Suh A, Bekemeier B. Local public health systems and the incidence of sexually transmitted diseases. Am J Public Health. 2012;102(9):1773–81.

Moore D, Shiell A, Noseworthy T, Russell M, Predy G. Public health preparedness in Alberta: a systems-level study. BMC Public Health. 2006;6:313.

Carlson V, Chilton MJ, Corso LC, Beitsch LM. Defining the functions of public health governance. Am J Public Health. 2015;105:S159–66.

Griffiths S, Jewell T, Donnelly P. Public health in practice: the three domains of public health. Public Health. 2005;119(10):907–13.

Padgett S, Bekemeier B, Berkowitz B. Collaborative partnerships at the state level: promoting systems changes in public health in infrastructure. J Public Health Manag Pract. 2004;10(3):251–7.

Canadian Coalition for Public Health in the 21st Century. Consultation on strengthening the Pan-Canadian public health system and meeting with the Minister of State (public health). Can J Public Health. 2004;95(4):1–15.

Hunter D, Marks L, Smith K. The public health system in England: a scoping study. 2007. https://core.ac.uk/download/pdf/28964369.pdf . Accessed 7 June 2020.

Ståhl T, Wismar M, Ollila E, Lahtinen E, Leppo K. Health in all policies. Prospects and potentials. Geneva: European Observatory of Health Systems and Policies; 2006.

Destoumieux-Garzón D, Mavingui P, Boetsch G, Boissier J, Darriet F, Duboz P, et al. The one health concept: 10 years old and a long road ahead. Front Vet Sci. 2018;5. https://doi.org/10.3389/fvets.2018.00014 .

Fleck F. Randall Packard: learning to learn from global health history. Bull World Health Organ. 2018;96(4):231–2.

Byskov J, Maluka S, Marchal B, Shayo EH, Blystad A, Bukachi S, et al. A systems perspective on the importance of global health strategy developments for accomplishing today’s sustainable development goals. Health Policy Plann. 2019;34(9):635–45.

Rechel B, Jakubowski E, McKee M, Nolte E, editors. Organization and financing of public health services in Europe. Copenhagen: WHO Regional Office for Europe; 2018.

United Nations. Transforming our world: the 2030 Agenda for sustainable development. Sustainable Development Knowledge Platform. https://sustainabledevelopment.un.org/?menu=1300 . Accessed 7 June 2020.

Download references

Acknowledgements

We would like to acknowledge Dr. Steven J. Hoffman for his input on an earlier version of this manuscript.

Not applicable.

Author information

Fran Scott, Fadi El-Jardali and Elizabeth Alvarez contributed equally to this work.

Authors and Affiliations

Department of Health Research Methods, Evidence and Impact (HEI), McMaster University, 1280 Main Street West, Hamilton, ON, L8S 4K1, Canada

Tamika Jarvis, Fran Scott, Fadi El-Jardali & Elizabeth Alvarez

Faculty of Health Sciences, American University of Beirut, Beirut, Lebanon

Fadi El-Jardali

You can also search for this author in PubMed   Google Scholar

Contributions

T. Jarvis and E. Alvarez conceived and designed the presented study. E. Alvarez, F. Scott and F. El-Jardali verified the analytical methods. T. Jarvis extracted, analysed and interpreted the data. E. Alvarez supervised the findings of this work. All authors discussed the results and contributed to the final manuscript. The author(s) read and approved the final manuscript.

Authors’ information

This work was completed as part of a Master’s thesis and includes material from a thesis published on MacSphere: http://hdl.handle.net/11375/22170 .

Jarvis, T. (2017). Defining public health systems: A critical interpretive synthesis of how public health systems are defined and classified (Master’s dissertation).

Corresponding author

Correspondence to Tamika Jarvis .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

The initial search strategy and databases. Additional file 1 provides search strings and detailed database search strategy.

Additional file 2.

Characteristics of documents reviewed for this study. Additional file 2 provides a description of the characteristics of the documents included in this study.

Additional file 3.

Definitions of entities and systems. Additional file 3 provides the definitions found within the literature regarding public health and public health systems.

Additional file 4.

Public Health Functions and Purpose. Additional file 4 provides a table as a way of organising what functions were found within the literature across a variety of countries, following the three core public health functions and 10 essential services.

Additional file 5.

Aligning public health systems into the health system arrangements framework. This additional file includes a summary of the coded data sources used to align public health within the health system arrangements framework.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Jarvis, T., Scott, F., El-Jardali, F. et al. Defining and classifying public health systems: a critical interpretive synthesis. Health Res Policy Sys 18 , 68 (2020). https://doi.org/10.1186/s12961-020-00583-z

Download citation

Received : 13 January 2020

Accepted : 27 May 2020

Published : 16 June 2020

DOI : https://doi.org/10.1186/s12961-020-00583-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health systems
• Population health
• Health services

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Research article
• Open access
• Published: 15 March 2016

Patient satisfaction with the healthcare system: Assessing the impact of socio-economic and healthcare provision factors

• Sofia Xesfingi 1 &
• Athanassios Vozikis 1  

BMC Health Services Research volume  16 , Article number:  94 ( 2016 ) Cite this article

47k Accesses

113 Citations

24 Altmetric

Metrics details

Patient satisfaction is an important measure of healthcare quality as it offers information on the provider’s success at meeting clients’ expectations and is a key determinant of patients’ perspective behavioral intention. The aim of this paper is first to assess the degree of patient satisfaction, and second, to study the relationship between patient satisfaction of healthcare system and a set of socio-economic and healthcare provision indicators.

This empirical analysis covers 31 countries for the years 2007, 2008, 2009 and 2012. The dependent variable, the satisfaction index, is defined as the patient satisfaction of their country’s health system. We first construct an index of patient satisfaction and then, at a second stage, this index is related to socio-economic and healthcare provision variables.

Our findings support that there is a strong positive association between patient satisfaction level and healthcare provision indicators, such as nurses and physicians per 100,000 habitants, with the latter being the most important contributor, and a negative association between patient satisfaction level and number of hospital beds. Among the socio-economic variables, public health expenditures greatly shape and positive relate to patient satisfaction, while private spending on health relates negatively. Finally, the elder a patient is, the more satisfied with a country’s healthcare system appears to be.

Conclusions

We conclude that there is a strong positive association between patient satisfaction and public health expenditures, number of physicians and nurses, and the age of the patient, while there is a negative evidence for private health spending and number of hospital beds.

Peer Review reports

Quality of care is a dominant concept in quality assurance and quality improvement programs in the healthcare sector. The importance of quality in the healthcare sector has been recognized, but it has been accelerated over the last decade through the development of quality insurance, quality improvement programs and patients’ agendas [ 1 ]. While quality of care, rather than price, is the main concern in healthcare [ 2 ], the service provider’s technical competence, as well as the immediate results from many treatments, is very difficult for a patient to evaluate [ 3 ].

It has been proposed that we can measure the quality of healthcare by observing its structure, its processes and its outcomes [ 4 ]. Whereas the aims of effectiveness and safety of healthcare are nearly universal, societies and cultures around the world differ in how much they emphasize the additional aims of patient-centeredness, timeliness, efficiency and equity. Healthcare measures –including process measures– are developed for varied audiences who may wish to use them for healthcare purchasing, utilization, or performance improvement [ 5 ]. For all these purposes it is imperative that are meaningful, scientifically sound, generalizable, and interpretable [ 6 ].

Patient satisfaction is an important measure of healthcare quality as it offers information on the provider’s success at meeting the expectations of most relevance to the client [ 7 ] and a key determinant of patients’ perspective behavioral intention [ 8 ]. Patient satisfaction is correlated with important outcomes, such as superior compliance, decreased utilization of medical services, less malpractice litigation and better prognosis [ 7 ]. The absence of a solid conceptual basis and consistent measurement tool for consumer satisfaction has led, over the past ten years, to a proliferation of surveys that focus exclusively on patient experience, i.e., aspects of the care experience such as waiting times, the quality of basic amenities, and communication with healthcare providers, all of which help identify tangible priorities for quality improvement [ 9 ]. Some researchers have suggested that defining quality improvement from patients’ perspective provides better value for their money with improved safety, accessibility, equity, and comprehensiveness of care, while from a provider’s point of view, quality improvement may be more efficient, providing more effective services to a greater number of consumers with a reasonable level of satisfaction, with the latter being enough for customer retention [ 10 ].

A handful of studies have attempted to relate patient’s health status to factors such as the performance of healthcare system [ 9 ] or other demographic and economic factors [ 11 – 13 ]. For a comprehensive review on patient satisfaction, see Pascoe [ 14 ] and Naidu [ 15 ].

More specifically, Bleich et al. [ 9 ] find that with respect to patient satisfaction and for 21 EU countries for the year 2003, about a quarter of the variation is attributed to healthcare system itself and to patient expectations, health status, type of care and immunization coverage. Furthermore, the study of Mummalaneni and Gopalakrishna [ 11 ] examines socio-demographic factors such as age, gender, occupation, employment status, education and income and reveals that income is the only socio-demographic factor found to have an influence on patient satisfaction. In addition, Gordo [ 12 ] examines data from the German Socio-Economic Panel and finds a strong association between long-term unemployment and patient satisfaction, while a weak association is documented for the short-term unemployment and patient satisfaction depending on the gender. Lastly, the study of Popescu et al. [ 13 ] investigates health status in relation to expenditures on health along with healthcare provisions (hospital beds and physicians per person) and finds a strong relationship between reporting a good or bad health status and health expenditures and provisions. A relevant study, that of Zhao et al. [ 16 ], examines instead the willingness to pay (WTP) per Quality-Adjusted Life Year (QALY) for a sample of chronic prostates patients. The WTP is associated with demographic factors of patients such as age, gender, education, marital status and with economic factors such as employment and level of income.

The purpose of this paper is first, to map the degree of the patient satisfaction in relevance with the healthcare system of their country during the years 2007, 2008, 2009 and 2012 in a panel of 31 countries using a satisfaction index, and second, to assess the impact of socio-economic and healthcare provision factors on the degree of patient satisfaction.

The contribution of this study is twofold. First, the hospital performance is transformed into a satisfaction index based on the patient’s perceptions about their country healthcare system. The latter, consists the first attempt in the literature. Second, the degree of patient satisfaction is examined along with a set of socio-economic and healthcare provision indicators. This is the first time in the literature as the majority of relevant studies explore only some indicators and for a limited number of countries and years.

The remaining of this paper is organized as follows: Section 2 presents our framework of analysis, data and model. Section 3 presents and discusses our findings. Finally, Section 4 concludes.

Quality of care from the patient’s perspective and patient satisfaction are two major multidimensional concepts that are used several times interchangeably [ 1 ]. Within this framework, every consumer or citizen may be a potential patient. This empirical analysis covers 31 countries: 28 EU Member States, Iceland, Norway, and Switzerland. The dependent variable, the satisfaction index, is defined as the patient satisfaction with respect to the country’s healthcare system, for the years 2007, 2008, 2009 and 2012. Information for the years 2010 and 2011 was not available. Furthermore, some of the countries did not provide data at the time of our research, especially with respect to healthcare provision factors, and, more particularly, for the number of nurses corresponding per 1000 habitants. Therefore, our final data set consists of 88 observations (instead of 124 = 31*4) as, for robustness purposes, we exclude the countries with limited data.

For the construction of the satisfaction index, data for the corresponding years were used from the Euro Health Consumer Powerhouse, and particularly, from the Euro Health Consumer Indexes (EHCI), where the performance of a country’s health system is evaluated through personal interviews and an active feedback from national healthcare agencies and institutions [ 17 ]. The built of EHCI is based on several indicators grouped in seven sub-disciplines, namely “Patient Rights and Information (PRI)”, “Accessibility (ACC)”, “Outcomes (OUT)”, “Range (RAN)”, “Pharmaceuticals (PHA)”, “Prevention (PRE)” and “E-Health (E-HEA)”. The performance of the respective national healthcare system is graded on a three-grade scale for each indicator, where the grades have the rather obvious meaning of “Good” = 3 points, “So-so” = 2 points and “Not so good” = 1 point. For each of the sub-disciplines, the country score is calculated as a percentage of the maximum possible, and then multiplied by weight coefficients since certain indicators are being more important than others.

For example, the sub-discipline “ACC” for the year 2012 consists of five indicators. Therefore, the maximum possible score is 5*3 = 15. The weight coefficient for this sub-discipline, along with the suggestions of expert panels and experience from a number of patient survey studies, is 250. Therefore, the points taken for every “Good” evaluation is 50. Consequently, the maximum score attainable for a national healthcare system, adding up all sub-disciplines scores after multiplying them with the weight coefficients is 1000 and the lowest possible score is 333.

We used the sub-disciplines’ total scores for each country to construct the Satisfaction Index. A country’s satisfaction index is a dummy and takes the value of 1 if its satisfaction index value is above the sample average; otherwise is 0.

Figure  1 shows the distribution of each one of seven sub-disciplines of the satisfaction index.

Share of the sub-disciplines of the satisfaction index

A number of (macro)economic indicators were employed, such as Gross Domestic Product (GDP) per capita (measured in constant 2005 US$), Health Expenditures as a percentage of GDP, Public Health Expenditures as a percentage of GDP, Private Health Expenditures as a percentage of GDP, Unemployment rate as percentage of total labor force, and Population Aging as a percentage of the total population above the age of 65 years, obtained from World Bank [ 18 ]. Furthermore, we also include some healthcare provision indicators, namely Number of Physicians per 100,000 habitants, Number of Nurses per 100,000 habitants and Number of Hospital Beds per 100,000 habitants, from Eurostat [ 19 ].

Table  1 presents the summary statistics of all variables.

As Table  1 shows, countries in our sample spend about 8.4 % of their GDP on health. Specifically, public health expenditures are three times larger than private ones. Moreover, it appears that per 1000 habitants (i.e. patients), on average, correspond almost 3 physicians, 8 nurses, and 1 hospital bed (323.635/100 ≈ 3, 856.691/100 ≈ 8, and 93.117/100 ≈ 1).

The likelihood of a certain patient being satisfied of a country’s healthcare system can be described by a logit model defined as follows:

where the endogenous variable Y is the degree of patient’s satisfaction and takes the value 1, if the patient is satisfied with his/her country’s healthcare system, and 0 otherwise; F is the standard logistic cumulative distribution function and Χ i is a set of covariates. The model is defined as:

where GDPcapita is gross domestic product (GDP) per capita, HealthExpenditures is public and private expenditures on health (%GDP), Unemployment is the unemployment rate, PopulationAging is the people above the age of 65 years old (%total population), Physicians is the number of physicians per 100,000 habitants, Nurses is the number of nurses per 100,000 habitants and HospitalBeds is the number of hospital beds per 100,000 habitants. The first four variables capture socio-economic conditions, whereas the remaining three proxy healthcare provision.

Table  2 presents the odds ratios for all specifications. The odd ratios can be interpreted as follows: if the odd ratios >1, then the probability of a patient being satisfied, i.e. Y it  = 1, increases, while decreases if odd ratios <1. Column (1) presents estimates of the baseline model, where health expenditures are aggregated into public and private spending. Column (2) splits the health expenditures into two categories, public and private health expenditures. For robustness purposes, columns (3) and (4), re-estimate specifications (1) and (2), but this time countries are classified as “high-income” and “low-income”. In doing so, a new variable, Dummy , is defined as follows: if a country’s GDP per capita is above sample average, then Dummy is one; otherwise is zero.

As Table  2 demonstrates, in specification (1), where one does not account for different type of health expenditures, i.e. public vs. private, the logit estimates are consistent with the theory and carry the right sign. Among the socio-economic variables, GDPcapita, HealthExpenditures , Unemployment and PopulationAging , only the latter appears to be statistically significant. More specifically, if population aging increases, the probability of a patient being satisfied increases by 340.3 % [(4.403-1)*100]. With respect to the healthcare provision indicators, all of them seem to be statistically significant. The number of physicians and the number of nurses associate positively with the patient satisfaction level (an increase in those indicators leads to an increase of the satisfaction level by 1.8 and 1.2 %, respectively) while the number of hospital beds associates negatively with patient’s satisfaction (an increase in number of hospital beds leads to a decrease of the patient satisfaction level by 1.3 %).

In column (2), once aggregated health expenditures are decomposed into public and private, findings appear somewhat different. Particularly, public health spending appears to be positively and statistically associated with patient satisfaction, that is, if public health expenditures increase, the probability of a patient being satisfied increases tremendously about 3500 times.

In order to capture the income differences across countries, the Dummy variable is introduced in the model in columns (3) and (4). The estimates of the baseline model still carry the right sign while the statistical significance pertains. Independently of a country’s income level, we find that the same set of variables associates in shaping patient satisfaction degree.

With respect to the overall performance of our specifications, correlations between patient satisfaction ( Y it ) and predicted patient satisfaction ( Ŷ it ) range from 84.5 to 92 % (at 5 % level of significance), indicating that the fitness of our specifications is satisfactory. The likelihood ratios from the diagnostics (bottom of Table  2 ), further confirm the goodness of the fit of our model.

Many studies have analyzed the relationship between GDP per capita and the health spending. These studies led to the extremely robust conclusion that even after statistical control for many other factors, the effect of GDP per capita (income) on expenditure is clearly positive and significant [ 20 ]. Public health expenditures play an important role for the patient satisfaction. Strong primary care has on better population health, fewer health disparities and lower rates of unnecessary hospitalizations [ 21 ]. Some countries are wealthy enough and they can afford to gear their governance, healthcare workforce, and funding arrangements towards expensive specialized care to satisfy public expectations [ 22 ].

If a patient is the habitant of a high-income country, his/her probability of being satisfied with the country’s health system is about 3400 times higher compared to a patient satisfaction level from a low-income country. This dramatic difference between “high-income” vs. “low-income” countries reflects the different perceptions existing among patients from different countries, implying that patients who reside in wealthier countries are more satisfied in general with the healthcare system compared to patients from less wealthy economies. It seems that wealthier countries are able to keep their patients more satisfied than poor ones, as expected.

The public spending on health has a large impact on patient satisfaction simply because health services are perceived to be provided free of charge by the state. The latter is more important for countries which are less wealthy. The important role of public health spending is also documented in numerous studies [ 23 , 22 ]. In contrast, private health spending appears to be negatively correlated with patient satisfaction as an increase of private health expenditures decreases patient satisfaction by 98.7 %. The negative relation between private health spending and patient satisfaction seems reasonable if one takes into consideration that citizens of all countries, although contribute to public health expenditures through taxation, they pay out of their pockets to receive (better) private healthcare when public healthcare fails. This is also consistent with other studies findings [ 24 ].

According to Kotzian [ 23 ], the patient satisfaction with the healthcare system might be influenced by other economic factors and properties of the healthcare system. As pointed out in the same article, the healthcare system might work well, but the distribution of the financial burden for its financing might be considered unfair by the patients. For example, as habitants of each country are getting older, we expect them to spend more money for their health status. Therefore, it seems natural to conclude that a nation’s per capita health spending will rise significantly as the average age of its population rises and that cross-national variations in health spending per capita are driven significantly by cross-national variations in the percentage of the population that is age sixty-five and older [ 24 ]. This is in line with other studies showing that elderly patients are more likely to express satisfaction with their healthcare than other sections of the patient population [ 25 ].

When it comes to healthcare provision, the literature finds that patient-to-nurse workloads were significantly associated with patients’ ratings and recommendation of the hospital to others, and therefore with their satisfaction when discharged [ 26 ]. Furthermore, Kotzian [ 23 ] suggested that a relatively low level of physicians per capita indicates a relative shortage of medical staff, and this might lower the satisfaction in the sense that there are not enough personnel to deliver beyond-health outputs. In the study of Ghose and Adhish [ 27 ], it was observed that patient satisfaction was greatly influenced by timing of admission, medical research and development, pharmacy, pantry services, nursing care and doctor’s care. More specifically, a very high percentage of the patients were satisfied with the physician services like availability of the doctor, doctor’s care and the treatment given by them.

All healthcare provision indicators seem to be statistically significant, but their relationships with patient satisfaction level do not carry the same sign for all of them. Particularly, the positive associations between the number of physicians and the satisfaction level, as well as the number of nurses and the satisfaction level, have also been documented to have similar effects in other studies. These findings are consistent with the studies of Kutney-Lee et al. [ 26 ] and Kotzian [ 23 ]. However, this increase of doctors and/or nurses could lead to higher public expenses and in countries with high debt/deficit could be challenging. In addition, if the number of hospital beds increases, the probability of a patient being satisfied with the healthcare system decreases by 1.3 %. This finding may reflect the unsolved issue of overcapacity which is documented in several studies, such as Kosnik [ 28 ] and Fidler et al. [ 29 ].

Since there is the first time to our knowledge that the healthcare systems’ performance is transformed into patient satisfaction, it is worth to further evaluate the construction of the satisfaction index. There was no methodology to base upon and, in addition, there were missing data for some of the countries (for the variables or for the years). Possible changes or errors with respect to the way data were collected could influence the satisfaction index and, in consequence, the results. Our methodology though, for the construction of the satisfaction index, is based first on the sample average and second on the ranking of each country with respect to the aforementioned average. We try with alternative indices constructed with higher deviations with respect to the proposed one. Results do not change significantly. Rankings of each country do not change dramatically across the years, so we may assume that even with the addition of some data, a country would probably stay above or under the sample average. If one takes into consideration that all finding refer to the sum of all countries in question, it is quite difficult to interpret since countries are grouped in different categories according to their national health care system, and face different difficulties with respect to their public spending or their health care provision indicators.

Finally, there might be several confounding factors that have contributed to these findings. Although it has been demonstrated that educational level plays an important role in shaping patient satisfaction, the data did not provided this information. Therefore, further research could focus on a country-level analysis, taking into consideration omitted factors, and evaluate the use of the questionnaire as well as the possible ceiling effect.

Overall, a key factor of patient satisfaction seems to be the responsiveness of the national healthcare system and the strategic changes’ implementation. It is a general belief that the relatively richer countries, and with full-coverage healthcare systems, are the best performers but the true problem lies on communicating the considerable improvements to the wide public.

This paper studied the relationship between patient satisfaction of a country’s healthcare system and a set of socio-economic and healthcare provision indicators.

Based on 31 countries and four years, our findings document the significant role of healthcare provision indicators such as the number of physicians and nurses provided in the healthcare system and support that there is a strong association between patient satisfaction level and number of hospital beds, nurses and physicians per 100,000 habitants, with the latter being the most important contributor. Among the socio-economic variables, public spending on health plays prominent role on patient satisfaction as greatly shapes and positive relates to patient satisfaction, while private spending on health related negatively. Finally, the elder a patient is, the more satisfied with a country’s healthcare system appears to be, exhibiting higher satisfaction from countries’ healthcare system.

A policy implication of our findings is that the role of government on health spending is highly important for patient satisfaction with respect to a healthcare system performance. Future research should control also for the type and quality of public as well spending in health.

Abbreviations

Accessibility

Euro Health Consumer Index

European Union

Gross Domestic Product

Pharmaceuticals

Patient Rights and Information

Quality-Adjusted Life Year

Willingness to pay

Raftopoulos V. A grounded theory for patients’ satisfaction with quality of hospital care. ICU Nurs Web J. 2005;22:1–15.

Google Scholar  

Cheng SH, Wei YJ, Chang HJ. Quality competition among hospitals: The effects of perceived quality and perceived expensiveness on health care consumers. Am J Med Qual. 2006;21(1):68–75.

Article   PubMed   Google Scholar  

Asubonteng P, McCleary KJ, Swan JE. SERVQUAL revisited: a critical review of service quality. J Serv Mark. 1996;10(6):62–81.

Article   Google Scholar  

Donabedian A. The quality of care, how can it be assessed? J Am Med Assoc. 1988;260(12):1743–8.

Article   CAS   Google Scholar  

Rubin HR, Pronovost P, Diette G. The advantages and disadvantages of process-based measures of health care. Int J Qual Health Care. 2001;13(6):469–74.

Article   CAS   PubMed   Google Scholar  

McGlynn EA. Choosing and evaluating clinical performance measures. Jt Comm J Qual Improv. 1998;24(9):470–9.

CAS   PubMed   Google Scholar  

Huang JA, Lai CS, Tsai WC, Hu WH, Yang DW. Determining factors of patient satisfaction for frequent users of emergency services in a medical center. J Chin Med Assoc. 2004;67(8):403–10.

PubMed   Google Scholar  

Al-Refaie A. A structural model to investigate factors affect patient satisfaction and revisit intention in Jordanian hospitals. Int J Artif Life Res. 2011;2(4):43–56.

Bleich SN, Özaltin E, Murray CJL. How does satisfaction with the health-care system relate to patient experience? Bull World Health Organ. 2009;87(4):271–8.

Article   PubMed   PubMed Central   Google Scholar  

Patwardhan A, Spencer CH. Are patient surveys valuable as a service-improvement tool in health services? An overview. J Healthc Leadersh. 2012;4:33–46.

Mummalaneni V, Gopalakrishna P. Mediators vs. Moderators of patient satisfaction. J Health Care Mark. 1995;15(4):16–22.

Gordo LR. Effects of short- and long-term unemployment on health satisfaction: evidence from German data. Appl Econ. 2006;38(20):2335–50.

Popescu L, Rat C, Rebeleanu A. Self-assessed health status and satisfaction with health care services in the new member states of the European Union. Studia UBB Sociologia. 2007;52(2):125–50.

Pascoe GC. Patient satisfaction in primary health care: A Literature Review and Analysis. Eval Program Plann. 1983;6(3–4):185–210.

Naidu A. Factors affecting patient satisfaction and healthcare quality. Int J Health Care Qual Assur. 2009;22(4):366–81.

Zhao FL, Yue M, Yang H, Wang T, Wu JH, Li SC. Willingness to pay per quality-adjusted life year: is one threshold enough for decision-making?: results from a study in patients with chronic prostatitis. Med Care. 2011;49(3):267–72.

Health Consumer Powerhouse. Euro Health Consumer Index, 2007, 2008, 2009, 2012 reports. http://www.healthpowerhouse.com/index.php?option=com_content&view=category&layout=blog&id=36&Itemid=55 . Assessed 1 October 2014.

The World Bank Group. Online database. http://data.worldbank.org/indicator . Assessed 1 October 2014.

Statistical Office of the European Communities. European Regional and Urban Statistics. Luxembourg: Eurostat; 2007. Reference guide. Assessed 1 October 2014. http://ec.europa.eu/eurostat/data/database .

Gerdtham UG, Jönsson B. International comparison of health expenditures: Theory, data and econometric analysis. Handb Health Econ. 2000;1(A):11–53.

Kringos DS, Boerma WGW, Van der Zee J, Groenewegen PP. Europe’ s strong primary care systems are linked to better population health, but also to higher health spending. Health Aff. 2013;32(4):686–94.

Kringos DS, Boerma WGW, Van der Zee J, Groenewegen PP. Political, cultural and economic foundations of primary care in Europe. Soc Sci Med. 2013;99(C):9–17.

Kotzian P. Determinants of satisfaction with health care system. Open Polit Sci J. 2009;2(1):47–58.

Reinhardt UE, Hussey PS, Anderson GF. Cross-national comparisons of health systems using OECD data. Health Aff. 2002;21(3):169–81.

Owens DJ, Batchelor C. Patient satisfaction and the elderly. Soc Sci Med. 1996;42(11):1483–91.

Kutney Lee A, McHugh MD, Sloane DM, Cimiotti JP, Flynn L, Felber Neff D, Aiken LH. Nursing: A key to patient satisfaction. Health Aff. 2009;28(4):669–77.

Ghose A, Adhsih VS. Patient satisfaction with medical services: Hospital-based study. Health Popul. 2011;34(4):232–42.

Kosnik L. Patient flow: Reducing delay in healthcare delivery. Int Ser Oper Res Manag Sci. 2006;91:101–22.

Fidler AH, Haslinger RR, Hofmarcher MM, Jesee M, Palu T. Incorporation of public hospitals: A “silver bullet” against overcapacity, managerial bottlenecks and resource constraints?: Case studies from Austria and Estonia. Health Policy. 2007;81(2–3):328–38.

Download references

Acknowledgements

We are grateful to Kyriakos Drivas, Claire Economidou, Dimitris Karamanis, Eugene Moustaferi and Georgia Papadopoulou for providing useful comments. The publication of this paper has been partly supported by the University of Piraeus Research Center.

Author information

Authors and affiliations.

Department of Economics, University of Piraeus, 18534, Piraeus, Greece

Sofia Xesfingi & Athanassios Vozikis

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Athanassios Vozikis .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

AV conceived of the study, and participated to its design and coordination. SX participated in the design of the study, and performed and interpreted the econometric analysis. Both authors participated equally in the sequence alignment and drafted, read, and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Xesfingi, S., Vozikis, A. Patient satisfaction with the healthcare system: Assessing the impact of socio-economic and healthcare provision factors. BMC Health Serv Res 16 , 94 (2016). https://doi.org/10.1186/s12913-016-1327-4

Download citation

Received : 19 June 2015

Accepted : 02 March 2016

Published : 15 March 2016

DOI : https://doi.org/10.1186/s12913-016-1327-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Patient satisfaction
• Healthcare provision
• Public health
• Private health
• Healthcare quality

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Help | Advanced Search

Computer Science > Cryptography and Security

Title: s3pher: secure and searchable system for patient-driven health data sharing.

Abstract: Healthcare data contains some of the most sensitive information about an individual, yet sharing this data with healthcare practitioners can significantly enhance patient care and support research efforts. However, current systems for sharing health data between patients and caregivers do not fully address the critical security requirements of privacy, confidentiality, and consent management. Furthermore, compliance with regulatory laws such as GDPR and HIPAA is often deficient, largely because patients typically are asked to provide general consent for healthcare entities to access their data. Recognizing the limitations of existing systems, we present S3PHER, a novel approach to sharing health data that provides patients with control over who accesses their data, what data is accessed, and when. Our system ensures end to end privacy by integrating a Proxy ReEncryption Scheme with a Searchable Encryption Scheme, utilizing Homomorphic Encryption to enable healthcare practitioners to privately search and access patients' documents. The practicality and benefits of S3PHER are further validated through end to end deployment and use case analyses, with tests on real datasets demonstrating promising execution times.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

This is a member publication of University College London (Jisc)

Twitter Analysis of Health Care Workers’ Sentiment and Discourse Regarding Post–COVID-19 Condition in Children and Young People: Mixed Methods Study

Authors of this article:

Original Paper

• Macarena Chepo 1 * , RN, BSN, MPH, PhD   ; 
• Sam Martin 2, 3 * , MSc, PhD   ; 
• Noémie Déom 2 , MSc   ; 
• Ahmad Firas Khalid 4 , MD, PhD   ; 
• Cecilia Vindrola-Padros 2 , BA, MA, PhD  

1 School of Nursing, Universidad Andrés Bello, Santiago, Chile

2 Department of Targeted Intervention, University College London, London, United Kingdom

3 Oxford Vaccine Group, Churchill Hospital, University of Oxford, Oxford, United Kingdom

4 Canadian Institutes of Health Research Health System Impact Fellowship, Centre for Implementation Research, Ottawa Hospital Research Institute, Otawa, ON, Canada

*these authors contributed equally

Corresponding Author:

Sam Martin, MSc, PhD

Department of Targeted Intervention

University College London

Charles Bell House 43-45

Foley Street

London, W1W 7TY

United Kingdom

Phone: 44 (0)20 3108 3232

Email: [email protected]

Background: The COVID-19 pandemic has had a significant global impact, with millions of cases and deaths. Research highlights the persistence of symptoms over time (post–COVID-19 condition), a situation of particular concern in children and young people with symptoms. Social media such as Twitter (subsequently rebranded as X) could provide valuable information on the impact of the post–COVID-19 condition on this demographic.

Objective: With a social media analysis of the discourse surrounding the prevalence of post–COVID-19 condition in children and young people, we aimed to explore the perceptions of health care workers (HCWs) concerning post–COVID-19 condition in children and young people in the United Kingdom between January 2021 and January 2022. This will allow us to contribute to the emerging knowledge on post–COVID-19 condition and identify critical areas and future directions for researchers and policy makers.

Methods: From a pragmatic paradigm, we used a mixed methods approach. Through discourse, keyword, sentiment, and image analyses, using Pulsar and InfraNodus, we analyzed the discourse about the experience of post–COVID-19 condition in children and young people in the United Kingdom shared on Twitter between January 1, 2021, and January 31, 2022, from a sample of HCWs with Twitter accounts whose biography identifies them as HCWs.

Results: We obtained 300,000 tweets, out of which (after filtering for relevant tweets) we performed an in-depth qualitative sample analysis of 2588 tweets. The HCWs were responsive to announcements issued by the authorities regarding the management of the COVID-19 pandemic in the United Kingdom. The most frequent sentiment expressed was negative. The main themes were uncertainty about the future, policies and regulations, managing and addressing the COVID-19 pandemic and post–COVID-19 condition in children and young people, vaccination, using Twitter to share scientific literature and management strategies, and clinical and personal experiences.

Conclusions: The perceptions described on Twitter by HCWs concerning the presence of the post–COVID-19 condition in children and young people appear to be a relevant and timely issue and responsive to the declarations and guidelines issued by health authorities over time. We recommend further support and training strategies for health workers and school staff regarding the manifestations and treatment of children and young people with post–COVID-19 condition.

Introduction

More than 3 years after the outbreak of the COVID-19 pandemic [ 1 ], the social, political, and economic impact of this phenomenon has been more than significant, considering >700 million worldwide cases and nearly 7 million people’s deaths [ 2 ]. Given the scale of the phenomenon, it is imperative for all countries to thoroughly examine the lessons gleaned from the pandemic, particularly regarding a matter that has raised significant concern among the populace: the long-term effects experienced by individuals who have had COVID-19, spanning weeks, months, or even years after their initial infection [ 3 ]. This phenomenon, referred to as post–COVID-19 condition (or more commonly “long COVID”), warrants careful consideration and analysis [ 4 ].

There is increasing information regarding the clinical manifestation of this condition, particularly in the adult population. The worldwide prevalence has been estimated at approximately 50% to 70% in individuals hospitalized during acute COVID-19 infection and 10% to 12% in vaccinated cases [ 5 ]. While children and young people have a low likelihood of severe COVID-19 infection [ 6 ], the information available to date indicates that the presence of post–COVID-19 condition in this group may be as disabling as in adults, reaching a prevalence rate of 23.4% (range 3.7%-66.5%) [ 7 ].

An agreed definition by the World Health Organization indicates that post–COVID-19 condition in children and young people is a condition that occurs “in individuals with a history of confirmed or probable SARS-CoV-2 infection when experiencing symptoms lasting at least two months which initially occurred within three months of acute COVID-19” [ 8 ]. Post–COVID-19 condition strongly impacts daily functioning and can develop or continue after COVID-19 infection and may fluctuate or relapse over time [ 4 , 8 , 9 ].

Among the symptoms most frequently attributable to post–COVID-19 condition in children and young people are fatigue, altered smell or anosmia, and anxiety [ 8 ]. However, other symptoms have also been reported, such as sleep disturbances, difficulty in concentrating, abdominal pain, myalgia or arthralgia, earache or ringing in ears, mood swings, persistent chest pain, stomach pain, light sensitivity, diarrhea, heart palpitations, and skin lesions [ 8 , 10 ]. One of England’s most significant studies is the Children and Young People With Long COVID study by Stephenson et al [ 11 ]. This national research matched longitudinal and cohort studies in adolescent individuals aged 11 to 17 years and found the presence of symptoms in 35.4% of the adolescent individuals who tested positive at baseline and 8.3% who of the adolescent individuals who tested negative at baseline. A total of 3 months after testing, 66.5% of those who tested positive and 53.3% of those who tested negative had any symptoms [ 11 ]. However, Stephenson et al [ 12 ] recently indicated that in a 6-month follow-up, the prevalence of specific symptoms reported at the time of the polymerase chain reaction testing decreased over time, where, for example, the prevalence of chills, fever, myalgia, cough, and sore throat among those who tested positive decreased from 10% to 25% to <3%.

As research on the symptoms, prevalence, and treatment of post–COVID-19 condition in children and young people continues, it is essential to add to the literature by developing studies that determine the condition’s impact on this group, considering that they are experiencing a range of unwanted symptoms that disrupt their quality of life and that of their families.

Considering that listening to the voices of families and health workers could be helpful to broaden the knowledge achieved in post–COVID-19 condition in children and young people, a powerful tool could be social media, such as Twitter (subsequently rebranded as X). With >3729 million daily active users, Twitter has become one of the most important social platforms in the world [ 13 ]. People used Twitter during the COVID-19 pandemic for different purposes, such as world leaders communicating with citizens [ 14 , 15 ], organizations monitoring movement [ 16 ], scientists studying public discourse around the pandemic [ 17 , 18 ], and researchers performing sentiment analysis [ 19 - 21 ]. In the case of physicians and health care workers (HCWs), Twitter has been used to share and evaluate scientific evidence, guidelines, and technical advice [ 22 - 24 ] and track the course and burden of disease [ 25 ].

Using the social media monitoring platform Pulsar [ 26 ], we aimed to explore HCWs’ perceptions concerning post–COVID condition in children and young people in the United Kingdom between January 2021 and January 2022. We aimed to contribute to the emerging knowledge on post–COVID-19 condition in children and young people and identify critical areas and future directions for researchers and policy makers.

We considered a mixed methods approach to be a pragmatic research paradigm. We analyzed data by conducting a Collaborative and Digital Analysis of Big Qualitative Data in Time Sensitive Contexts (LISTEN) [ 27 ]. This mixed methods analysis consisted of iterative cycles intercalating team discussion and using digital text and discourse analytics tools to analyze related social media data [ 27 ]. We used the LISTEN method to perform quantitative and qualitative analyses of Twitter posts, extracted through the Pulsar platform [ 26 ], related to the experience of post–COVID-19 condition in children and young people in the United Kingdom (eg, phrases, words, hashtags, videos, and images), published between January 1, 2021, and January 31, 2022. We created an advanced Boolean search for keywords mentioning “long COVID” and corelated words, hashtags, and symptoms; furthermore, we filtered for user accounts who identified as HCWs in their Twitter biography description ( Multimedia Appendix 1 ).

Quantitative analysis of all tweets included the following: (1) engagement analysis, where we specifically measured reactions to posts, for example, a retweet, a share, or a comment or quote made toward a tweet; (2) sentiment and emotion analysis, where we measured the positive or negative sentiment in the words and tone of each post within the context of post–COVID-19 condition and HCW’s roles ( Multimedia Appendix 2 ); (3) emotion analysis, where we measured the emotions expressed in the tweets, classified as sadness, anger, disgust, fear, and joy; (4) frequency analysis, where we observed the frequency of keywords and themes in the data set; (5) segmentation analysis, where we measured the key connections or relationships between keywords and their frequent use in the same context; (6) demographic analysis, where we measured the occupation, gender (man or woman or nonbinary or unknown), and city of origin related to the users posting tweets; and (7) analyses, where we evaluated the most influential accounts and the most mentioned websites.

Big qualitative analysis was carried out through thematic discourse analysis of the data sample, using InfraNodus [ 28 ], specifically analyzing the key themes and topics of concern expressed throughout the data set. A codebook was constructed based on the mapping of themes agreed upon by 3 researchers (ND, SM, and MC; Multimedia Appendix 3 ).

The principal investigators (ND, AFK, SM, and MC) interpreted and analyzed the data collected, following the recommendations for rigorous research provided by Creswell and Poth [ 29 ]. Using the LISTEN method [ 27 ], we aimed to show that the integration of qualitative insights through thematic analysis with the quantitative backing of topic modeling can offer a comprehensive view of the discourse. This mixed methods approach allows us to capture the richness of qualitative data while leveraging the objectivity of quantitative measures. Our initial data harvest of the larger corpus data from the Pulsar platform captured 300,000 tweets; this data harvest helped to underpin the software’s sentiment analysis modeling of this specific data set, providing a robust quantitative foundation. The addition of further qualitative data analyses from a smaller qualitative sample allowed for an in-depth understanding of nuanced conversations, particularly when exploring new or complex phenomena such as post–COVID-19 condition in children and young people, with the provision of insights into the context, subtext, and sentiment behind the tweets offering valuable snapshots of public perception and discourse. We used an iterative mixed methods approach, iterating between team discussions and using digital analytics tools to discern relevant themes from the Twitter data corpus. Specifically, we used InfraNodus for thematic analysis, which incorporates a topic modeling script for analyzing and identifying key topics of concern with a data set and provides a structured and objective interpretation of the data. The coding process involved 3 independent researchers (MC, SM, and ND), each with expertise in health care, social network analysis, and digital global health. When initial coding disagreements arose, we meticulously tagged any queries and discussed the posts in question. These instances led to 3 structured meetings wherein the research team deliberated collaboratively to resolve conflicting interpretations. This approach resulted in an 81.99% (2122/2588) initial intercoder agreement rate for the tweets analyzed. For the remaining instances where consensus was not initially reached, the majority rule was applied to finalize theme codings. To quantify the reliability of our coding procedure, with 81.99% (2122/2588) of the tweets coded identically, we used the Cohen κ score, which provides a measure of interrater agreement adjusted for chance. Including the calculation of all variations, this score was calculated to be approximately κ=0.70, indicating good agreement among the coders.

Ethical Considerations

The study only collected data from publicly accessible social networks that have been anonymized by various means, particularly by replacing all usernames and links with anonymous text and summaries of tweets that have been edited, retaining the original message, avoiding direct quotations being identifiable, and ensuring that no information is provided on the identity of the individuals who posted the content studied on the platform.

Internet research requires researchers to carefully consider guidelines to determine whether ethics approval and informed consent are needed [ 30 ]. On the basis of the terms set out by the Research Ethics Committee at the University College London [ 31 ], the study was considered exempt from formal ethics approval for the following reasons: (1) study involving information freely available in the public domain, such as published biographies, newspaper accounts of an individual’s activities, and published minutes of a meeting, that although is considered personal under the Data Protection Act, would not require ethics review; and (2) study involving anonymized records and data sets in the public domain, such as data sets available through the Office for National Statistics or the UK Data Archive where appropriate permissions have already been obtained and it is not possible to identify individuals from the information provided.

Therefore, we anonymized all records and data sets collected during the study to make identification impossible. We removed social media usernames from the data samples. No direct or easily traceable quotes have been included. These measures align with best practices [ 32 - 35 ]. While this study was beyond the scope of the human ethics committee, we adhered to the principles of ethics: beneficence, nonmaleficence, autonomy, and justice [ 36 ]. We collected and analyzed data through secure encrypted servers via the Meltwater and InfraNodus platforms.

Audience Analysis

During the period from January 2021 to January 2022, we obtained 300,000 tweets from 936 accounts. After filtering for relevant posts (refer to inclusion and exclusion criteria in Multimedia Appendix 1 ), we analyzed a sample of 2588 tweets using mixed methods analysis. In terms of gender (man, woman, nonbinary, or unknown), 32.88% (851/2588) were female individuals, 23.49% (608/2588) were male individuals, and 43.59% (1128/2588) were unknown. According to the description given in the user’s biography, the most frequently self-reported terms were “NHS” (582/2588, 22.49%), “health” (230/2588, 8.89%), “medical” (168/2588, 6.49%), “nurse” (166/2588, 6.41%), “clinical” (160/2588, 6.18%), “mum” (158/2588, 6.11%), “doctor” (145/2588, 5.6%), and “GP” (145/2588, 5.6%). In terms of city, tweets came mainly from London (958/2588, 37.02%), Newcastle upon Tyne (326/2588, 12.6%), Redcar (160/2588, 6.18%), Manchester (140/2588, 5.41%), and Bradford (111/2588, 4.29%).

Regarding profession described in the user’s biography, the most frequently mentioned roles were nurses (176/2588, 6.8%); medical roles, for example, paramedic and nursing assistant (173/2588, 6.68%); clinical roles, for example, surgeon, physiotherapist, and anesthesiologist (160/2588, 6.18%); general practitioners (GPs), for example, hospital GP or local surgery GP (142/2588, 5.49%); and physician (140/2588, 5.41%). The most frequent organization affiliated with was the National Health Service (587/2588, 22.68%).

Most Influential Accounts

One of the accounts that generated the highest number of mentions and, therefore, some of the most influence, as they were the ones that talked the most about post–COVID-19 condition in children and young people, was the account for @longcovidkids (593/2588, 22.91% tweets), related to the most shared website longcovidkids.org [ 37 ] , an international UK-based charity for families and children living with post–COVID-19 condition. Although the account was created in October 2020, it was first mentioned in our data collection timeline on January 1, 2021. It offers web support services, funding, and research participation and represents children and young people living with post–COVID-19 condition in expert forums, research panels, health organizations, and parliamentary groups. The other most shared web pages were theguardian.com (the United Kingdom) [ 38 ], bbc.co.uk (the United Kingdom) [ 39 ], peoplewith.com (the United States) [ 40 ], and ncbi.nlm.nih.gov (the United States) [ 41 ]. This shows that in the United Kingdom, there was a mixed influence of UK and US link resources linked to HCW Twitter users in the United Kingdom.

Keyword Analysis

The volume of social media engagement in the discussion about the post–COVID-19 condition experience in children and young people in the United Kingdom reached 1400 posts, 1550 engagements, and 1.9 million impressions. Overall, comments were very responsive to government decisions regarding the vaccination program and school closures ( Multimedia Appendix 4 ). During the first peak of comments in January 2021, the amount of discourse expanded leading up to March 2021, when there were different announcements of school closures, and the guidelines were delivered regarding the priority groups of the vaccination program (frontline HCW and people aged >80 years first). The highest engagement was between June and July 2021, which coincides with the government announcement regarding the availability of vaccines for people aged >18 years. The third peak of comments occurred in September 2021, the same month the authorities announced the extension of the vaccination program to children aged 12 to 15 years.

Top Keywords Analysis

The top words in posts associated with children and young people’s experience of post–COVID-19 condition in the United Kingdom were “Children” (352/2588, 13.6%), “kids” (160/2588, 6.18%), “people” (158/2588, 6.11%), “Young” (148/2588, 5.72%), and “schools” (83/2588, 3.21%). The top hashtags were #longcovid (1387/2588, 53.59%), #longcovidkids (448/2588, 17.31%), #covid19 (370/2588, 14.3%), and #covid (176/2588, 6.8%).

Sentiment and Emotions Analysis

According to sentiment analysis, 99.38% (2572/2588) of the posts reflected negative sentiments and 0.62% (16/2588) reflected positive sentiments. Negative sentiments were mainly associated with comments on hospitalization figures related to the COVID-19 pandemic, criticism of pandemic mitigation policies, and vaccination of children and young people. Furthermore, positive sentiments mainly concerned acknowledgments around decreasing numbers of community support groups.

The primary emotions identified were as follows:

• Sadness (1752/2588, 67.7%), such as in the following tweet:

@[Username] Really upset, after my tough on-call last night. Hospitalisations are still going up, and Gov announcement minismises the effect of long-COVID in adults and children. It’s so hard to keep spirits up today. But we’ll try and continue doing our best in the NHS.

• Joy (367/2588, 14.18%), such as in the following tweet:

@[Username] It’s been an amazing day! [...] I’ve been able to share the experience I’ve gained treating children and adolescents with Long COVID over the last year.

• Fear (233/2588, 9%), as seen in the following tweet:

@[Username] It’s really urgent that young people get the message that they need to get vaccinated. Long COVID is ruining many people’s lives! It’s not a lie or hypochondria, there are real, physiological changes, please understand!

Segmentation Analysis

This analysis revealed the critical clusters of conversation around the main topics of concern within the discourse network around post–COVID-19 condition. Comments were distributed in 4 key conversation segments as follows:

• People, schools, and prevention (1734/2588, 67%): Most of the comments related to measures taken in terms of COVID-19 prevention in schools, concern about the risk of exposure, and sharing experiences of infection in schools.
• Health, adults, and impact (401/2588, 15.49%): Comments mainly reflected concerns and uncertainty about the long-term effect of post–COVID-19 condition on both children and young people and adults.
• Cases, virus, and risk (326/2588, 12.6%): Comments reflected worries about the associated risks and long-term consequences attributable to post–COVID-19 condition (in both adults and children and young people) and the constant mutation of the virus, which will create a permanent risk in the population.
• Months, distress, and symptoms (106/2588, 4.1%): Some HCWs used Twitter to share how children and young people experience post–COVID-19 condition and the extent of these symptoms. Some HCWs exemplified certain typical manifestations, such as fatigue.

Discourse Analysis by Theme

To better understand the topics discussed from the segmentation analysis, we performed a discourse analysis of the key co-occurring themes and topics of concern shared within discussions regarding post–COVID-19 condition in children and young people. The following themes emerged ( Textbox 1 ): concern or uncertainty for the future, school attendance, mask protection from COVID-19, vaccine uptake, infection rates, policy (support or skepticism), understanding and visualizing symptoms, child mental health, access to care, community support, and research ( Figures 1 and 2 ).

• Concern for the future or uncertainty (615/2588, 23.76% tweets): Most comments showed a concern for the future, focusing on shared statistics regarding the rate and spread of infection in children and young people and how this would affect future health outcomes. Furthermore, this group expressed concern regarding political decisions; the presence of illness in loved ones; the eventual overload and response capacity of the health system in the face of an increase in post–COVID-19 condition cases; and the need for training of health care workers (HCWs) to deal with comorbid, potentially long-term symptoms ( Figure 1 A).
• Schools (460/2588, 17.77% tweets): Comments aimed to promote vaccination policies for schoolchildren and flexible measures regarding teachers’ work and attendance, considering cases of people with prolonged symptoms. In addition, several tweets expressed dissatisfaction with school risk mitigation measures, such as the use of face masks and air filters ( Figure 1 B).
• Vaccine (386/2588, 14.9% tweets): Most tweets from this group showed their disapproval of the constant changes in the government’s decisions regarding schools and priority groups for vaccination. Between March and June 2021, the first set of tweets criticized the lack of priority in the vaccination program for schoolchildren and other at-risk groups (such as teachers). Once the authorities announced a vaccination program for schoolchildren aged 12 to 15 years ( Multimedia Appendix 4 ), most comments promoted vaccination for this group. A few comments (78/2588, 3.01%) shared concerns about the vaccine’s efficacy for children, based on the experiences of COVID-19 reinfection in adults despite having received the recommended initial doses. However, to a lesser extent (26/2588, 1%), there was a refusal to vaccinate children, citing fear of possible adverse effects. Nonetheless, it is worth noting that the community frequently refuted such comments ( Figure 1 C).
• Share statistics (334/2588, 12.91% tweets): Frequently, HCWs shared statistical data, such as the number of affected children and young people, the number of post–COVID-19 condition cases, and hospital admissions and deaths. Some of these data were used to validate the existence of the post–COVID-19 phenomenon or to express concern about it ( Figure 1 D).
• Policy (316/2588, 12.21% tweets): The comments were responsive to the policies emanating from the authorities over time ( Multimedia Appendix 4 ). There were 5 main criticisms, including changes in school closure or opening policies; HCWs question why the authorities ignore the evidence of post–COVID-19 cases in children and young people, leading them to question whether decision makers have sufficient training to control the pandemic adequately; the failure to include teachers and school workers in the COVID-19 vaccination program as well as the younger population; the lack of mitigation measures in schools, such as improvements in ventilation systems and mandatory use of masks; and the herd immunity as a plan in the government’s hidden agenda , that is, to promote work and activate the economy ( Figure 1 E).
• “Proof” (280/2588, 10.82% tweets): Most tweets in this group argued regarding the existence of children and young people with post–COVID-19 condition through pictures; statistics; scientific papers; and personal, family, and professional experiences ( Figure 1 F).
• Signs and symptoms (189/2588, 7.3% tweets): Among the symptoms described, chronic fatigue and exhaustion were the most frequent symptoms, which prevent normal activities. Other symptoms were respiratory: dyspnea, chronic cough, and shortness of breath; gastrointestinal: acute or intense abdominal pain, nausea, bloating, gastroparesis, and change in smell or taste; muscular: severe joint pain, “painful foot” and difficulty with physical activity; mental health: anxiety and low mood; topical: rash, skin rashes, and redness and pain in the eyes; and nonspecific symptoms, such as chest pain, heart palpitations, constant high body temperature, precocious puberty, hormonal changes, and erectile dysfunction ( Figure 2 A).
• Face masks (119/2588, 4.6% tweets): Face masks were widely promoted, especially in schools, because HCWs considered them as a practical and straightforward strategy to control the pandemic ( Figure 2 B).
• Skepticism (101/2588, 3.9% tweets): Comments showed reticence toward post–COVID-19 condition in children and young people. Some of the arguments focused on a perceived lack of clarity in the clinical manifestations and stressed the need to better differentiate the post–COVID-19 condition from other related symptomatologies, such as mood disorders (eg, depression and anxiety due to confinement). In contrast, several arguments agreed on the need for more scientific evidence, arguing that post–COVID-19 condition in children and young people are isolated. Other users claimed not to know of such cases instead of calling post–COVID-19 condition in children and young people SMS text message an exaggeration. In addition, several arguments favored releasing restrictions for children and young people, particularly arguments related to the use of masks, because of possible associated risks, for example, hypoxia ( Figure 2 C).
• Mental health (54/2588, 2.09% tweets): Symptoms attributable to mental health problems in children and young people were also a concern. For instance, HCWs mentioned sadness, fear of infecting their family, anxiety regarding sick parents, stress, night terrors, self-harm, and suicidal ideation. Furthermore, users discussed a perceived lack of specific support for children and young people and their families in situations such as hospitalization; prolonged COVID-19 condition; admission to intensive care; and death of a family member, schoolmate, or teacher, all situations that triggered permanent stress in these groups ( Figure 2 D).
• Community support or asking for advice (93/2588, 3.59% tweets): Some HCWs used Twitter to ask for guidance on a specific issue or share experiences of having post–COVID-19 condition or caring for children and young people or family members. Furthermore, they shared informative infographics provided by experts regarding post–COVID-19 condition in children and young people ( Figure 2 E).
• Access to health care or treatment (72/2588, 2.78% tweets): Some HCWs mentioned the lack of specialist (cardiology) support, concerns regarding prolonged National Health Service burnout, and criticisms regarding how follow-up was carried out concerning the relative symptomatology of children and young people with post–COVID-19 condition. At the same time, opening new centers for children and young people with post–COVID-19 condition generated different reactions. On the one hand, some HCWs recognized it as a substantial development, but on the other hand, some HCWs recognized it as proof of the existence of post–COVID-19 condition in children and young people, which raised concerns for the future ( Figure 2 F).
• Research (52/2588, 2% tweets): Under this theme, tweets largely promoted study on post–COVID-19 condition in children and young people or highlighted the need for further study on the subject ( Figure 2 G).
• Images (57/2588, 2.2% tweets): Images shared were primarily from scientific studies, including infographics (from organizations such as National Health Service or @LongCovidKids) and visualization of children and young people’s symptoms, such as rashes, COVID-19 toe, and joint pain. Most infographics shared by organizations (and not individuals), such as the organization LongCovidKids, were related to statistics, such as the number of children and young people with post–COVID-19 condition or the quantification of the type of symptoms experienced. Shared photographs tended to show the more “visually recognizable” symptoms of post–COVID-19 condition, such as skin lesions, rashes, or inflammation. The less visible symptoms, such as chronic fatigue and neurological issues, were represented with photographs of children and young people lying, sleeping under blankets, or duvets or on hospital beds ( Figure 2 H).

Principal Findings

Our primary objective was to explore HCWs’ perceptions concerning post–COVID-19 condition in children and young people in the United Kingdom between January 2021 and January 2022. Our findings indicated that comments made by HCWs on Twitter were responsive to announcements issued by authorities regarding the management of the COVID-19 pandemic in the United Kingdom and associated regulations on the operation of schools. The most frequent feelings and emotions were negative, mainly sadness. In turn, we identified relevant themes for HCWs, such as uncertainty or concern about the future; policies; and regulations for the prevention, management, and addressing both COVID-19 and post–COVID-19 condition in children and young people; vaccination; and the use of Twitter as a strategy to share scientific literature, management strategies, and clinical and personal experiences.

Concern from HCWs regarding the policies for addressing the COVID-19 pandemic in the children and young people in the United Kingdom (including vaccination and schools) was a recurring theme in our findings. Furthermore, concern regarding the side effects of the COVID-19 vaccine and how the vaccine might interact with preexisting physiological symptoms of post–COVID-19 condition in children and young people was a topic of discussion. Similarly, the constant change in policy making in the United Kingdom, as public health bodies and governments have tried to understand and adapt to the emergence of post–COVID-19 condition, have added to the strength of this ongoing debate [ 42 ]. The lack of up-to-date evidence on post–COVID-19 condition in children and young people prompted HCWs to rely on Twitter during the pandemic to communicate relevant information. Twitter has a broad audience reach; is used as a communication tool by politicians, health bodies, and other key influences; and facilitates real-time updates [ 43 ]. During the pandemic, HCWs, primarily those in frontline roles and local response coordination, have often been challenged to become credible spokespersons for pandemic information [ 44 ]. Such credibility directly influences public confidence and decision-making, ultimately determining the success or failure of a public health intervention [ 43 ].

Furthermore, failures in risk communication could explain the presence of uncertainty and negative feelings associated with school regulations. When people are upset, distressed, or fearful, they often do not trust the authority, decrease the perceived validity of the communication received, and find information processing difficult [ 45 ]. In this regard, Fotheringham et al [ 46 ] indicated that during 2020, school leaders in the United Kingdom faced pressures and challenges related to translating and enacting school policies, particularly with the perceived lack of agency shared by the government concerning being able to translate centrally issued guidelines. In turn, Tomson et al [ 47 ] reported that the pandemic has negatively impacted the well-being of leaders in all types of schools and across all demographic groups, affecting their ability to think clearly and solve work-related problems. Given that the protection and care of children and young people health during the COVID-19 pandemic ultimately rests with school leaders, the search for support strategies that focus on the needs of these groups becomes an urgent necessity.

Findings in Relation to Other Studies

Using Twitter’s information, this is one of the first studies to capture health professionals’ perceptions of prolonged COVID-19 in the children and young people in the United Kingdom. However, other studies have addressed post–COVID-19 condition on this social network. Callard and Peregov [ 48 ] reviewed how, through social platforms such as Twitter, patients made the persistence and heterogeneity of COVID-19 symptoms visible, thus catapulting the inclusion of post–COVID-19 condition as a relevant phenomenon in clinical and policy debates. In contrast, other authors in the last 2 years have explored on various platforms (including Twitter) the persistence of symptoms and emotional impact after months of suspected and confirmed diagnosis of COVID-19 [ 49 - 55 ], including the period of vaccination. Furthermore, others have explored web discussions regarding this phenomenon [ 56 ]. Several of these authors agree on a perceived lack of support and specific resources from governmental bodies, a lack of information or clarity in the instructions given, and the absence of formal mechanisms to allow the voices of patients and the community to be heard. The above point is critical as it highlights the gap between the needs of the population and the response provided by policy makers, which not only translates into a gap in access to health services but also limits citizen participation in decision-making on the issues that affect their own health and increases distrust toward regulations and instructions issued by the government.

Implications for Policy and Practice

Several policy recommendations and implications are targeted at various stakeholders to consider while implementing future policy guidelines to address post–COVID-19 health care delivery. First, policy makers should consider investing appropriate resources to collect data regarding post–COVID-19 condition in children and young people, specifically on the impact of COVID-19 on the mental health of children and young people. This implies working closely with researchers to streamline data collection and reporting on post–COVID-19 condition. Second, policy makers should consider providing a basic level of psychosocial support with access to quality mental health and psychosocial support services for HCWs, school staff, parents, and children and young people experiencing post–COVID-19 condition. This implies strengthening health systems, community-based programming, and mobilization. Policies must include documenting the impact of mental health and psychosocial support interventions and innovative approaches to be more widely disseminated and scaled up across different contexts and target population groups. Third, to address the criticism around frequent changes in school closure and opening policies, decision makers should develop clear, easy-to-understand school mitigation plans informed by the best available evidence. The plans should incorporate teachers, school workers, and parents to ensure all voices are included in the policy plan. Fourth, policy makers should adopt a shared decision-making approach incorporating HCWs in the decision-making process for managing the COVID-19 pandemic. Finally, government decision makers should set post–COVID-19 pandemic recovery policies informed from a health equity perspective and how this affects children and young people living with post–COVID-19 condition, factoring in childhood, family income, housing, domestic violence, access to health care, and racism.

In terms of the needed clearer road map for recommendations to support training strategies for HCWs and school staff regarding post–COVID-19 condition in children and young people, we have outlined the following 10 steps.

Step 1: Data Collection and Analysis

Our study underlines the critical need for comprehensive data on post–COVID-19 condition’s impact on the mental health of children and young people. As a first step, it is recommended that policy makers should allocate resources for the systematic collection and analysis of data on post–COVID-19 condition in children and young people, particularly focusing on mental health outcomes. These data should be used to identify the most prevalent symptoms and the most effective treatment strategies. In this context, it is recommended that experts emphasize the importance of early detection and medical consultation for mental health issues in children and young people diagnosed with post–COVID-19 condition, including mood changes, irritability, social withdrawal, memory problems, difficulty in concentrating, anxiety, depression, posttraumatic stress, school absenteeism, and suicidal ideation [ 57 , 58 ]. This entails working closely with researchers to streamline data collection and reporting on post–COVID-19 condition.

Step 2: Psychosocial Support Framework

It has been noted that globally, programs for managing post–COVID-19 condition in children and young people are heterogeneous, ranging from the use of physiotherapy, pediatric occupational therapy, and psychological support to interventions aimed at lifestyle modifications [ 59 ]. This diversity could impact differential outcomes in the treatment, recovery, and timely and effective rehabilitation of children and young people with post–COVID-19 condition. Upon analyzing the wider literature and the social media data in this study, it is recommended that a basic level of psychosocial support should be established. This would involve ensuring access to quality mental health services for HCWs, school staff, parents, and children and young people with post–COVID-19 condition. This framework should be integrated into the health system and community-based programming, emphasizing the mobilization of resources and strengthening of support networks. It is suggested that the psychosocial support framework should facilitate access to quality mental health services and support networks that are robust and responsive. Community engagement gleaned from further Twitter discourse analysis should be a helpful guide in the development of these services to ensure they meet the real and expressed needs of children and young people with post–COVID-19 condition. Practical examples of basic psychosocial support include using web support services; individual or group therapy sessions; school-based emotional support programs; and counseling sessions aimed at parents, family members, or school staff.

Step 3: Educational Mitigation Plans

The frequent policy changes around school closures highlight the necessity for stable and clear educational mitigation plans. It is recommended that these plans should be directly informed by the evidence collected and further analysis of sentiments and emotions surrounding post–COVID-19 condition in schools. Incorporating the viewpoints of teachers, parents, and school staff, as identified in our thematic analysis, will ensure that the mitigation strategies are comprehensive, feasible, and sensitive to the psychosocial impact on children and young people. School staff and policy makers should collaborate to develop clear, evidence-informed educational mitigation plans. These plans should be straightforward and involve teachers, school workers, and parents in their creation, ensuring a unified approach that considers the voices of all stakeholders.

Step 4: Shared Decision-Making in Health Care

In health care settings, the adoption of a shared decision-making model is crucial, enabling HCWs to actively contribute to the formulation of COVID-19 and post–COVID-19 policies. This inclusive approach ensures that frontline workers can provide valuable insights toward policy development. To facilitate this, the establishment of advisory committees composed of representatives from HCWs is recommended. This committee can convene regularly to deliberate on key decisions pertaining to the COVID-19 pandemic management, including prevention measures, resource distribution, and vaccination strategies. Such collaborative groups have demonstrated effectiveness in identifying priority needs within the context of a pandemic [ 60 ].

Step 5: Health Equity in Policy Setting

Post–COVID-19 recovery policies should be set with a health equity lens. This means considering factors such as family income, housing, domestic violence, access to health care, and racism and how these factors affect children and young people living with post–COVID-19 condition. Our findings emphasize the importance of framing post–COVID-19 recovery policies through a lens of health equity. The concerns raised by HCWs regarding the socioeconomic impacts, such as family income and access to health care, underline the need for policies that address not just the medical aspects of post–COVID-19 condition but also the social determinants of health. An equitable approach will ensure that children and young people from diverse backgrounds receive appropriate support.

Step 6: Documenting and Disseminating Interventions

It is vital to document the impact of mental health and psychosocial support interventions. In this context, it is crucial to implement innovative strategies to disseminate unbiased information about post–COVID-19 condition among health care professionals and educators working with children and young people, ensuring it reaches different contexts and populations. These strategies may include creating interactive multimedia resources, such as videos and mobile apps; organizing webinars; actively using social media; and forming web support groups. These groups will provide a space where patients, health care professionals, and educators can share their experiences and knowledge regarding post–COVID-19 condition. These actions will not only help reduce isolation and social stigma but also strengthen support for these groups considered vulnerable [ 61 ].

Step 7: Developing a Clear Communication Strategy

Policy makers must develop a clear communication strategy to address frequent policy changes and mitigate confusion. This strategy should be informed by the data collected and analysis conducted in Step 1. The data reveal a palpable sense of uncertainty and frustration due to frequent policy shifts, underscoring the need for a clear and consistent communication strategy. This strategy should be grounded in the evidence gathered from the health care community’s discourse and aim to minimize confusion by providing timely, transparent, and reliable information regarding post–COVID-19 policies and support services.

Step 8: Training and Support Strategies

On the basis of the findings of the comprehensive data analysis, specific training and support strategies should be developed for HCWs and school staff. These strategies should be focused on the practical aspects of identifying and managing post–COVID-19 condition in children and young people. For instance, training sessions could include practical workshops on recognizing post–COVID-19 symptoms in children and adolescents, conducting diagnostic assessments, and implementing appropriate treatment and support interventions.

Step 9: Continuous Feedback and Policy Adaptation

The continuous evolution of the post–COVID-19 phenomenon demands an iterative approach to policy making. On the basis of our study, we recommend establishing feedback mechanisms with HCWs and school staff to monitor the reception and effectiveness of implemented policies. This feedback, coupled with ongoing research, should inform policy adaptations to ensure they remain aligned with the evolving landscape of post–COVID-19 condition and its impact on children and young people.

Step 10: Evaluation and Research

Finally, there should be a commitment to ongoing evaluation and research. This will involve not only monitoring the implementation of the abovementioned steps but also supporting new research to fill any remaining gaps in understanding the long-term effects of COVID-19 on children and young people.

This sequence of steps is designed to be iterative and responsive, ensuring that the recommendations from the study are translated into concrete actions that adapt to emerging data and research findings.

Strengths and Limitations

A key strength of this study is that our social media analysis of post–COVID-19 condition contributes toward an emerging understanding of reported experiential, emotional, and practical dimensions of post–COVID-19 condition in children and young people specifically and questions of vaccine hesitancy in children and young people with post–COVID-19 condition. This is one of the few studies to collect HCWs’ perceptions regarding post–COVID-19 condition in children and young people in the United Kingdom using information from Twitter. We identify key areas that need considering attention and focus, such as the provision of psychosocial support with access to quality mental health resources to alleviate the impact of post–COVID-19 condition in children and young people and the development of clear post–COVID-19 pandemic recovery guidelines that are informed by health equity perspective, and how this affects children and young people living with post–COVID-19 condition.

One of the limitations this study acknowledges is the definition of post–COVID-19 condition in children and young people. When data were collected, the lack of consensus on the definition of post–COVID-19 condition in children and young people forced us to formulate a definition of post–COVID-19 condition in children and young people based on the available literature. Furthermore, this study is limited to the perceptions of people who used descriptors in their web biography attributable to HCWs; therefore, our results only represent some HCWs in the United Kingdom and those in other countries. In turn, this research collected data from Twitter only; therefore, further inquiry into HCWs’ perceptions of post–COVID-19 condition in children and young people required expanding to other data sources or social networks and including languages other than English. We acknowledge that demographic factors, geographic location, and individual daily activities of social media users can significantly influence language use and word choice, introducing potential biases in tweet-based data. Such biases are inherent in any analysis of social media content and can affect the generalizability of findings. For instance, our study relies on Twitter data, which do not encompass the full spectrum of global or the UK public opinion on post–COVID-19 condition in children and young people. While Twitter serves as a valuable platform for capturing real-time sentiments and experiences, it is not fully representative of all demographics and geographic regions. Our results may reflect the perspectives of more vocal or active social media users, which may not correspond to the silent majority or those without access to social media. In addition, the absence of geotagged information for many users limits our ability to conduct a more nuanced spatial analysis of the sentiments expressed.

Furthermore, our study is built upon the recognition that social media data may overrepresent certain demographic groups while underrepresenting others, such as the older population or those without reliable internet access. This skew can influence the apparent prevalence of certain views or experiences of post–COVID-19 condition. Moreover, individuals’ patterns of daily life, reflected in their social media use and content, contribute additional layers of complexity and potential bias to the discourse analyzed.

Consistent with scholarly precedents on the subject [ 62 , 63 ], our study acknowledges these biases as intrinsic limitations of social media–based research. Although our analysis did not control for these factors, we recognize their potential impact on our results. Future studies would benefit from incorporating a broader array of data sources, including interviews or focus groups, to provide a more representative and comprehensive understanding of post–COVID-19 condition in children and young people. This approach would complement our Twitter-based findings and help mitigate the biases inherent in social media data.

Conclusions

More than a year after the start of the COVID-19 pandemic, the perceptions described on Twitter by HCWs concerning the presence of post–COVID-19 condition in children and young people appear to be a relevant and timely issue as well as very responsive to the declarations and guidelines issued by the health authorities over time. The most prominent group within the discourse studied was the activist or lobbying organization @LongCovidKids, which shared the most tweets and images over the period studied. We recommend that future research focus on how web health activism is organized and carried out for children and young people with post–COVID-19 condition. Such a strategy would allow for a better understanding of the scope and impact of this phenomenon and how it can influence decision-making. Furthermore, we suggest different mitigation strategies, support, and training of HCWs and school staff regarding manifestations and treatment of post–COVID-19 condition in children and young people across all demographic areas.

Acknowledgments

The authors would like to thank the Rapid Research Evaluation and Assessment Lab, Department of Targeted Intervention, University College London, London, United Kingdom, whose support has been essential for developing this project.

Conflicts of Interest

None declared.

Filters used for the search strategy on Twitter.

Sentiment analysis framework: attitudes toward post–COVID-19 condition in children and young people.

Theme codebook: examples of tweets that fit into main themes tagged for mention of children and young people with post–COVID-19 condition.

Timeline of national governmental policies and guidelines regarding children and young people.

• World Health Organization. COVID-19 Public Health Emergency of International Concern (PHEIC) Global research and innovation forum. Global Research Collaboration for Infectious Disease Preparedness. Feb 12, 2020. URL: https:/​/www.​who.int/​publications/​m/​item/​covid-19-public-health-emergency-of-international-concern-(pheic)-global-research-and-inno vation-forum [accessed 2022-06-06]
• Johns Hopkins University of Medicine. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Coronavirus Resource Center. Jan 22, 2020. URL: https://coronavirus.jhu.edu/map.html [accessed 2023-06-09]
• Nature Editorial. Long COVID and kids: more research is urgently needed. Nature. Feb 08, 2022;602(7896):183. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Long COVID or post-COVID conditions. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/coronavirus/2019-ncov/long-term-effects/index.html [accessed 2023-06-09]
• Davis HE, McCorkell L, Vogel JM, Topol E. Long COVID: major findings, mechanisms and recommendations. Nat Rev Microbiol. Mar 13, 2023;21(3):133-146. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zimmermann P, Curtis N. Why is COVID-19 less severe in children? A review of the proposed mechanisms underlying the age-related difference in severity of SARS-CoV-2 infections. Arch Dis Child. Dec 01, 2020;106(5):429-439. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zheng YB, Zeng N, Yuan K, Tian SS, Yang YB, Gao N, et al. Prevalence and risk factor for long COVID in children and adolescents: a meta-analysis and systematic review. J Infect Public Health. Mar 07, 2023;16(5):660-672. [ FREE Full text ] [ CrossRef ] [ Medline ]
• World Health Organization. A clinical case definition for post COVID-19 condition in children and adolescents by expert consensus. World Health Organization. Feb 16, 2023. URL: https:/​/www.​who.int/​publications/​i/​item/​WHO-2019-nCoV-Post -COVID-19-condition-CA-Clinical-case-definition-2023-1 [accessed 2024-03-26]
• Stephenson T, Allin B, Nugawela MD, Rojas N, Dalrymple E, Pinto Pereira S, CLoCk Consortium, et al. Long COVID (post-COVID-19 condition) in children: a modified Delphi process. Arch Dis Child. Jun 17, 2022;107(7):674-680. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Esposito S, Principi N, Azzari C, Cardinale F, Di Mauro G, Galli L, et al. Italian intersociety consensus on management of long Covid in children. Ital J Pediatr. Mar 09, 2022;48(1):1-9. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stephenson T, Pinto Pereira SM, Shafran R, de Stavola BL, Rojas N, McOwat K, CLoCk Consortium, et al. Physical and mental health 3 months after SARS-CoV-2 infection (long COVID) among adolescents in England (CLoCk): a national matched cohort study. Lancet Child Adolesc Health. Feb 08, 2022;6(4):230-239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stephenson T, Pinto Pereira SM, Nugawela MD, McOwat K, Simmons R, Chalder T, et al. CLoCk Consortium. Long COVID-six months of prospective follow-up of changes in symptom profiles of non-hospitalised children and young people after SARS-CoV-2 testing: a national matched cohort study (The CLoCk) study. PLoS One. Mar 6, 2023;18(3):e0277704. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Twitter users, stats, data and trends. Data Reportal. May 11, 2023. URL: https://datareportal.com/essential-twitter-stats#:~ :text=Based [accessed 2023-06-09]
• Rufai SR, Bunce C. World leaders' usage of Twitter in response to the COVID-19 pandemic: a content analysis. J Public Health (Oxf). Aug 18, 2020;42(3):510-516. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haman M. The use of Twitter by state leaders and its impact on the public during the COVID-19 pandemic. Heliyon. Nov 2020;6(11):e05540. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huang X, Li Z, Jiang Y, Li X, Porter D. Twitter reveals human mobility dynamics during the COVID-19 pandemic. PLoS One. Nov 10, 2020;15(11):e0241957. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wicke P, Bolognesi MM. Framing COVID-19: how we conceptualize and discuss the pandemic on Twitter. PLoS One. Sep 30, 2020;15(9):e0240010. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chang CH, Monselise M, Yang CC. What are people concerned about during the pandemic? detecting evolving topics about COVID-19 from Twitter. J Healthc Inform Res. Jan 17, 2021;5(1):70-97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xue J, Chen J, Hu R, Chen C, Zheng C, Su Y, et al. Twitter discussions and emotions about the COVID-19 pandemic: machine learning approach. J Med Internet Res. Nov 25, 2020;22(11):e20550. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boon-Itt S, Skunkan Y. Public perception of the COVID-19 pandemic on Twitter: sentiment analysis and topic modeling study. JMIR Public Health Surveill. Nov 11, 2020;6(4):e21978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Valdez D, Ten Thij M, Bathina K, Rutter LA, Bollen J. Social media insights into US mental health during the COVID-19 pandemic: longitudinal analysis of Twitter data. J Med Internet Res. Dec 14, 2020;22(12):e21418. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rashid MA, Yip SW, Gill D, Arfeen Z. Sharing is caring: an analysis of #FOAMed Twitter posts during the COVID-19 pandemic. Postgrad Med J. Mar 14, 2022;98(1157):199-204. [ CrossRef ] [ Medline ]
• Ghosh P, Schwartz G, Narouze S. Twitter as a powerful tool for communication between pain physicians during COVID-19 pandemic. Reg Anesth Pain Med. Feb 21, 2021;46(2):187-188. [ CrossRef ] [ Medline ]
• Burgos LM, Gil Ramirez A, Utengen A, Thamman R. Use of Twitter during COVID-19 pandemic: an opportunity for continuing medical education in cardiology. Medicina (B Aires). 2020;80 Suppl 6:122-123. [ FREE Full text ] [ Medline ]
• Margus C, Brown N, Hertelendy AJ, Safferman MR, Hart A, Ciottone GR. Emergency physician Twitter use in the COVID-19 pandemic as a potential predictor of impending surge: retrospective observational study. J Med Internet Res. Jul 14, 2021;23(7):e28615. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pulsar TRAC- Audience intelligence platform and social listening tool. Pulsar. URL: https://www.pulsarplatform.com/solutions/pulsar-trac/ [accessed 2022-03-01]
• Vera San Juan N, Aceituno D, Djellouli N, Sumray K, Regenold N, Syversen A, et al. Mental health and well-being of healthcare workers during the COVID-19 pandemic in the UK: contrasting guidelines with experiences in practice. BJPsych Open. Dec 10, 2020;7(1):e15. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Paranyushkin D. InfraNodus: generating insight using text network analysis. In: Proceedings of the World Wide Web Conference. 2019. Presented at: WWW '19; May 13-17, 2019; San Francisco, CA. [ CrossRef ]
• Creswell JW, Poth CN. Qualitative Inquiry and Research Design: Choosing Among Five Approaches 4th Edition. Thousand Oaks, CA. SAGE Publications, Inc; 2017.
• Eysenbach G, Till JE. Ethical issues in qualitative research on internet communities. BMJ. Nov 10, 2001;323(7321):1103-1105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• UCL research ethics. University College London. URL: https://www.ucl.ac.uk/research-ethics/do-i-need-ethical-approval# When [accessed 2023-06-12]
• Martin S, Kilich E, Dada S, Kummervold PE, Denny C, Paterson P, et al. "Vaccines for pregnant women…?! Absurd" - Mapping maternal vaccination discourse and stance on social media over six months. Vaccine. Sep 29, 2020;38(42):6627-6637. [ CrossRef ] [ Medline ]
• Kummervold PE, Martin S, Dada S, Kilich E, Denny C, Paterson P, et al. Categorizing vaccine confidence with a transformer-based machine learning model: analysis of nuances of vaccine sentiment in Twitter discourse. JMIR Med Inform. Oct 08, 2021;9(10):e29584. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ahmed W, Bath PA, Demartini G. Using Twitter as a data source: an overview of ethical, legal, and methodological challenges. In: Woodfield K, editor. The Ethics of Online Research. Bingley, UK. Emerald Publishing Limited; Dec 12, 2017.
• Ayers JW, Caputi TL, Nebeker C, Dredze M. Don't quote me: reverse identification of research participants in social media studies. NPJ Digit Med. Aug 2, 2018;1(1):30. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Varkey B. Principles of clinical ethics and their application to practice. Med Princ Pract. Jun 4, 2021;30(1):17-28. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Long Covid Kids homepage. Long Covid Kids. URL: https://es.longcovidkids.org/ [accessed 2023-06-12]
• The Guardian homepage. The Guardian. URL: https://www.theguardian.com/uk [accessed 2023-06-12]
• Welcome to the BBC. BBC. URL: https://www.bbc.com/ [accessed 2023-06-12]
• People with health. PeopleWith. URL: https://peoplewith.com/ [accessed 2023-06-12]
• Welcome to NCBI. National Institutes of Health National Library of Medicine. URL: https://www.ncbi.nlm.nih.gov/ [accessed 2023-06-12]
• Edobor M. Trust and public discourse during the Covid‐19 pandemic. IPPR Progress Rev. Feb 15, 2021;27(4):354-360. [ CrossRef ]
• Nielsen RK, Fletcher R, Kalogeropoulos A, Simon F. Communications in the coronavirus crisis: lessons for the second wave. Reuters Institute for the Study of Journalism. Oct 27, 2020. URL: https://reutersinstitute.politics.ox.ac.uk/communications-coronavirus-crisis-lessons-second-wave [accessed 2024-03-26]
• Zohar T, Negev M, Sirkin M, Levine H. Trust in COVID-19 policy among public health professionals in Israel during the first wave of the pandemic: a cross-sectional study. Isr J Health Policy Res. Apr 11, 2022;11(1):20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Enria L, Waterlow N, Rogers NT, Brindle H, Lal S, Eggo RM, et al. Trust and transparency in times of crisis: results from an online survey during the first wave (April 2020) of the COVID-19 epidemic in the UK. PLoS One. Feb 16, 2021;16(2):e0239247. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fotheringham P, Harriott T, Healy G, Arenge G, Wilson E. Pressures and influences on school leaders navigating policy development during the COVID‐19 pandemic. Brit Educ Res J. Jul 30, 2021;48(2):201-227. [ CrossRef ]
• Thomson P, Greany T, Martindale N. The trust deficit in England: emerging research evidence about school leaders and the pandemic. J Educ Admin History. Sep 13, 2021;53(3-4):296-300. [ CrossRef ]
• Callard F, Perego E. How and why patients made long COVID. Soc Sci Med. Jan 2021;268:113426. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davis HE, Assaf GS, McCorkell L, Wei H, Low RJ, Re'em Y, et al. Characterizing long COVID in an international cohort: 7 months of symptoms and their impact. EClinicalMedicine. Aug 2021;38:101019. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Miyake E, Martin S. Long Covid: online patient narratives, public health communication and vaccine hesitancy. Digit Health. Nov 29, 2021;7:20552076211059649. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Matharaarachchi S, Domaratzki M, Katz A, Muthukumarana S. Discovering long COVID symptom patterns: association rule mining and sentiment analysis in social media tweets. JMIR Form Res. Sep 07, 2022;6(9):e37984. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bhattacharyya A, Seth A, Rai S. The effects of long COVID-19, its severity, and the need for immediate attention: analysis of clinical trials and Twitter data. Front Big Data. Dec 15, 2022;5:1051386. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Samannodi M, Alwafi H, Naser AY, Al Qurashi AA, Qedair JT, Salawati E, et al. Determinants of post-COVID-19 conditions among SARS-CoV-2-infected patients in Saudi Arabia: a web-based cross-sectional study. Diseases. Aug 23, 2022;10(3):55. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Santarossa S, Rapp A, Sardinas S, Hussein J, Ramirez A, Cassidy-Bushrow AE, et al. Understanding the #longCOVID and #longhaulers conversation on Twitter: multimethod study. JMIR Infodemiology. 2022;2(1):e31259. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Déguilhem A, Malaab J, Talmatkadi M, Renner S, Foulquié P, Fagherazzi G, et al. Identifying profiles and symptoms of patients with long COVID in France: data mining infodemiology study based on social media. JMIR Infodemiology. Nov 22, 2022;2(2):e39849. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Awoyemi T, Ebili U, Olusanya A, Ogunniyi KE, Adejumo AV. Twitter sentiment analysis of long COVID syndrome. Cureus. Jun 2022;14(6):e25901. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Malone LA, Morrow A, Chen Y, Curtis D, de Ferranti SD, Desai M, et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of postacute sequelae of SARS-CoV-2 infection (PASC) in children and adolescents. PM R. Oct 2022;14(10):1241-1269. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shachar-Lavie I, Shorer M, Segal H, Fennig S, Ashkenazi-Hoffnung L. Mental health among children with long COVID during the COVID-19 pandemic. Eur J Pediatr. Apr 14, 2023;182(4):1793-1801. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brackel CL, Noij LC, Vijverberg SJ, Legghe CL, Maitland-van der Zee AH, van Goudoever JB, et al. International care programs for pediatric post-COVID condition (long COVID) and the way forward. Pediatr Res. Jan 29, 2024. (forthcoming). [ CrossRef ] [ Medline ]
• Ripp J, Peccoralo L, Charney D. Attending to the emotional well-being of the health care workforce in a New York City health system during the COVID-19 pandemic. Acad Med. Aug 2020;95(8):1136-1139. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buonsenso D, Camporesi A, Morello R, De Rose C, Fracasso M, Chieffo DP, et al. Social stigma in children with long COVID. Children (Basel). Sep 07, 2023;10(9):1518. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mitchell L, Frank MR, Harris KD, Dodds PS, Danforth CM. The geography of happiness: connecting twitter sentiment and expression, demographics, and objective characteristics of place. PLoS One. May 29, 2013;8(5):e64417. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Padilla JJ, Kavak H, Lynch CJ, Gore RJ, Diallo SY. Temporal and spatiotemporal investigation of tourist attraction visit sentiment on Twitter. PLoS One. Jun 14, 2018;13(6):e0198857. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 20.06.23; peer-reviewed by R Gore, A Wahbeh; comments to author 02.11.23; revised version received 14.02.24; accepted 08.03.24; published 17.04.24.

©Macarena Chepo, Sam Martin, Noémie Déom, Ahmad Firas Khalid, Cecilia Vindrola-Padros. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Healthc Inform Res
• v.28(1); 2022 Jan

Review of Smart Hospital Services in Real Healthcare Environments

Hyuktae kwon.

1 Department of Family Medicine, Seoul National University Hospital, Seoul, Korea

2 Transdisciplinary Department of Medicine and Advanced Technology, Seoul National University Hospital, Seoul, Korea

Ho-Young Lee

3 Department of Nuclear Medicine, Seoul National University Bundang Hospital, Seongnam, Korea

Won Chul Cha

4 Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea

5 Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, Korea

Sungwan Kim

6 Institute of Medical & Biological Engineering, Medical Research Center, Seoul National University College of Medicine, Seoul, Korea

7 Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Korea

8 Biomedical Research Institute, Seoul National University Hospital, Seoul, Korea

Hyoun-Joong Kong

9 Department of Medicine, Seoul National University College of Medicine, Seoul, Korea

Smart hospitals involve the application of recent information and communications technology (ICT) innovations to medical services; however, the concept of a smart hospital has not been rigorously defined. In this study, we aimed to derive the definition and service types of smart hospitals and investigate cases of each type.

A literature review was conducted regarding the background and technical characteristics of smart hospitals. On this basis, we conducted a focus group interview with experts in hospital information systems, and ultimately derived eight smart hospital service types.

Smart hospital services can be classified into the following types: services based on location recognition and tracking technology that measures and monitors the location information of an object based on short-range communication technology; high-speed communication network-based services based on new wireless communication technology; Internet of Things-based services that connect objects embedded with sensors and communication functions to the internet; mobile health services such as mobile phones, tablets, and wearables; artificial intelligence-based services for the diagnosis and prediction of diseases; robot services provided on behalf of humans in various medical fields; extended reality services that apply hyper-realistic immersive technology to medical practice; and telehealth using ICT.

Conclusions

Smart hospitals can influence health and medical policies and create new medical value by defining and quantitatively measuring detailed indicators based on data collected from existing hospitals. Simultaneously, appropriate government incentives, consolidated interdisciplinary research, and active participation by industry are required to foster and facilitate smart hospitals.

I. Introduction

In recent years, the focus in the medical field has shifted from disease treatment to prevention-oriented medicine, centering on medical consumers. In line with these changes, hospitals have introduced artificial intelligence (AI), robotics, and new technologies related to the Fourth Industrial Revolution, thereby raising expectations for improvements in the quality of disease diagnosis and treatment. There is an emerging need to reduce national medical costs by creating interconnections with other related medical facilities in the community, such as clinics, pharmacies, and rehabilitation centers, integrating the latest information and communication technologies (ICT); and providing “patient-centered care” services where patients are given access to various health information so that they can choose their own treatment methods [ 1 , 2 ]. For these reasons, various support measures to foster smart hospital-related industrial developments are being promoted in major countries, and the smart hospital-related market is expanding worldwide [ 1 ].

Efforts to apply ICT in the medical field have been made in the past, and this perspective was initially applied through the concept of the “digital hospital.” The digital hospital is a concept that emerged in the early 2000s in Korea, with the goal of fostering a complete transformation from analog hospital workflows where charts, films, slips, papers, and pens were mostly used for all aspects of recording patients’ medical histories to a digital hospital workflow involving the “four ‘lesses’” (filmless, chartless, slipless, paperless) method by establishing computerized physician order-entry, picture archiving and communication systems, Electronic Medical Record (EMR) systems, and hospital business systems [ 3 , 4 ]. Nonetheless, although the word “smart” in “smart hospitals” has been analogized to the SMART (Specific, Measurable, Action-oriented, Realistic, and Time-bound) framework used rhetorically in the field of management, the concept has not been rigorously defined from an academic standpoint [ 5 , 6 ].

Given the recent interest in smart hospitals, which are characterized by the application of recent innovative technology in the global information and communication field to medical services, this study sought to derive implications by exploring the definitions and service types of smart hospitals and reviewing cases of each type.

II. Exploring the Concept of the Smart Hospital

As a concept similar to smart hospitals, the term “intelligent hospitals” started to be used in the context of radiofrequency identification (RFID), in the Healthcare Consortium sponsored by the RFID Educational Foundation in the United States in 2009. With this concept, the researchers proposed a service that applies real-time location tracking technology, communication technology, and interoperability technology to various spaces in hospitals such as operating rooms, hospital wards, and outpatient clinics [ 7 ]. An organization called the Intelligent Health Association, with a focus on intelligent hospital product and service companies was established in the United States, and has been producing high-quality technology education, thereby promoting the adoption and implementation of new technology in the industry and expansion of the market. Medical institutions seek to improve patient management and safety, reduce medical costs, and revolutionize patient experiences through intelligent hospital services [ 8 ].

We conducted a literature review to define the concept of a smart hospital and derive its service types. The review was based on keywords such as “smart hospital,” and we searched literature databases such as the Research Information Sharing Service ( http://www.riss.kr ), National Science & Technology Information Service ( http://www.ntis.go.kr ), Google ( http://scholar.google.co.kr , http://www.google.com ), and others. Among the search results, papers, books, and trend reports that could provide an academic interpretation of the medical field were selected as the final research target. The literature introduced the concepts of smart hospitals in various ways, as shown in Table 1 [ 2 , 9 – 13 ].

Various concepts of the “smart hospital”

ICT: information and communication technology, IoT: Internet of Things.

McKinsey Health presented the main characteristics of smart hospitals [ 12 ] as the establishment of interoperability of systems, mobile integrated solutions, digitalization of all information, establishment of unified communication systems, provision of stable core infrastructure, and system automation. The Korea Health Industry Development Institute defined smart hospitals as a holistic system that requires the participation of all parties who provide medical consumer-oriented services to improve their productivity and the accuracy of medical services by using ICT [ 1 ] ( Table 2 ).

Key features of smart hospitals introduced by McKinsey Health [ 12 ]

Following the literature review on smart hospitals, we conducted a focus group interview (FGI) to derive the concept of smart hospitals and the relevant service types. The FGI proceeded in the following order: selecting the interview subjects, sharing the literature review with the subjects, and an expert interview.

For the FGI, researchers who were faculty members in the department of hospital information system management and development at tertiary hospitals, as well as researchers with academic expertise in medical informatics or health information technology, were interviewed.

Upon providing the interviewees with the literature review results, we asked them to propose a smart hospital concept and the types of service it would provide. The discussion among the interviewees lasted until a consensus was reached.

Smart hospitals can be defined as medical institutions that create new value and insights on patient safety, quality of care, cost-effectiveness, and patient-centeredness using ICT, and provide them in a quantitative manner to patients and medical staff. The services of smart hospitals can be categorized into eight types of technology: location recognition and tracking technology, high-speed communication networks, the Internet of Things (IoT), mobile health, AI, robotics, extended reality, and telehealth.

III. Examples of the Representative Service Types of Smart Hospitals

1. location recognition and tracking services.

Various services can be provided by measuring and monitoring the location information of objects in a specific space using location recognition and tracking technology based on short-range communication technology [ 14 ]. The associated technologies include beacons, Bluetooth, Wi-Fi, Zigbee, RFID, GPS, A-GPS, barcodes, QR codes, and ultra-wideband communication (e.g., 5G technology standard). In particular, by introducing a tracking system for real-time assets (e.g., medical devices, medical supplies, and pharmaceutical products) based on beacon sensors and tags, medical institutions can improve the efficiency of logistics management related to hospital work and the workflow of medical staff [ 15 , 16 ] ( Figure 1 ).

Example of an asset management service in hospitals. Adapted from Yoo et al. [ 15 ].

The University Health System, a medical institution in the United States, introduced a smart infusion pump with an attached RFID, which has been reported to increase productivity and efficiency by reducing the time required for the medical staff to use the device in the hospital from the previous maximum of 2 hours to 8–12 minutes through real-time location monitoring [ 17 ].

2. High-Speed Communication Network-based Services

High-speed communication network-based services, such as 5G high-speed communication and Wi-Fi 6, provide medical services that overcome the previous limitations of data collection, utilization, and communication based on new wireless communication technology. Wi-Fi 6, as a high-speed communication network, is particularly suitable for use in hospital systems where there is a large volume of traffic and frequent environmental changes and updates. By using orthogonal frequency-division multiple access (OFDMA; Wi-Fi 6 technology that allows multiple clients with different requirements to access a single access point at the same time), the transmission waiting time can be reduced, and the target wake time function allows power-saving for devices and improved battery life. Using Wi-Fi 6, it is possible to accurately analyze patient records and real-time data and improve treatment outcomes by accurately administering medication following an objective decision-making process based on accurate, up-to-date patient data [ 18 ].

Wi-Fi 6-enabled medical devices, such as infusion pumps with adjustable data transfer times, can reduce usage overlap and improve the efficiency of device operation and maintenance through OFDMA by allowing up to 30 different devices to share the same infusion pump and channel without changing orders [ 19 ].

Researchers from Huazhong College of Science and Technology have proposed an integrated system termed “5G-Smart Diabetes” that manages patients with diabetes using machine learning, medical big data, social networks, and smart clothing based on 5G wireless technology. Using 5G technology, this system continuously monitors physiological indicators in patients with diabetes, shares the relevant data, and applies a personalized data analysis model to provide tailored treatment services without any restrictions on the patient’s daily life [ 20 ].

In addition, the reduction in data transmission time has upgraded the use of smart hospitals in terms of health monitoring. Through fast communication network services, health monitoring has become reliable anytime, anywhere. Even in emerging countries and hard-to-reach areas, a lack of data access is no longer a serious problem. To improve the situation further, research is being conducted on the use of a low peak-to-average power ratio, an efficient spectrum, low latency, and faster data compression and delivery [ 21 ].

3. IoT-based Services

IoT-based services are technologies that connect various objects embedded with sensors and communication functions to the Internet, including object identification, network construction, sensor attachment (sensation), and action instruction (control) [ 22 , 23 ]. With IoT, smart hospitals can be developed by leveraging sensors, connection methods, internet protocols, databases, cloud computing, and analytics as infrastructure, and using multiple systems together in a composite [ 24 ]. IoT and intelligent building technologies can be used for a wide range of purposes, such as decreasing operating costs, increasing treatment effects, reducing diagnostic delays, detecting deterioration early, maximizing equipment utilization, improving patient safety, increasing energy efficiency in hospital buildings, increasing profit-ability, improving user experience, increasing operational efficiency, and protecting resources [ 25 ].

In the nursing field, IoT is used to automate the measurement of patients’ vital signs and various indicators. IoT-based vital sign measurement sensors have been gradually miniaturized, as shown in Figure 2 , to be attached or inserted into the body, and new indicators are being added, such as to measure patient posture and the number of steps [ 26 ]. Barcodes, RFID, fingerprint/iris/face recognition, and ultrasound-based recognition systems are used in smart hospital buildings [ 27 ].

Examples of applications of Internet of Things technology in hospitals. (A) A mobile Electronic Medical Record (EMR) communicates with a near-field communication (NFC) system, and the mobile EMR displays progress after NFC tagging. (B) Combination of a wearable patient sensor and a dashboard.

The most actively used IoT-based technology is RFID systems, which are used for people, buildings, and medical devices, and research on the use of mobile near-field communication (NFC) systems has also been carried out [ 28 – 32 ].

A study at an academic medical center in the United States investigated the use of a system in which a patch is attached to patients in the intensive care unit that sends an alarm to nurses through the dashboard. It reported that this system reduced the incidence of bedsores in patients to approximately one-third of the original rate, thereby improving patient safety and cost efficiency [ 33 ]. An analysis of the effectiveness of a mobile EMR system based on short-range communication technology in hospital emergency rooms, as depicted in Figure 2 , showed that it increased patient consultation time by optimizing the travel distance of the medical staff and reducing the time required for operating IT equipment [ 31 , 33 ].

Recent research has also explored the use of machine learning and IoT to diagnose coronavirus disease 2019 (COVID-19) patients. The proposed model, which is based on machine learning and IoT, can serve as a clinical decision support system. In addition, the results are expected to contribute to reducing the excessive workload of doctors and solving the problem of patient overcrowding caused by COVID-19 [ 34 ].

4. Mobile Health Services

Mobile health services are services provided through mobile devices such as cell phones, tablets, and wearable devices. Personal health records (PHRs), as a mobile health record system, are built to allow self-monitoring and management by collecting treatment and examination information from various medical institutions, as well as activity level, weight, and blood glucose data collected by smartphones [ 35 , 36 ]. Previous PHR-related research addressed the usability of these systems and their ability to increase patient participation [ 37 – 39 ].

Researchers at the Icahn School of Medicine at Mount Sinai in the United States analyzed heart rate variability (HRV) data from medical staff asked to wear a smartwatch equipped with an app that measured HRV. The analysis revealed that COVID-19 infection could be detected through changes in HRV, even among those without symptoms. In an experiment performed at Stanford University, 81% of test participants wearing smart watches or fitness trackers were reported to have HRV up to 9 days before the onset of COVID-19 symptoms [ 40 ].

Given the fragmentation in the locations and subjects of existing medical services, the demand for services such as mobile EMRs that medical staff use to contact patients in various places is also increasing. An NHS-based app in the United Kingdom provides access to hospital EMRs, prescription refills, appointments, and treatment plan reviews [ 41 ].

The use of mobile EMRs was confirmed to have partially increased actual work efficiency, but the results were often limited to Korea, and the research mostly focused on the targets and content [ 42 ] ( Figure 3 ).

Examples of hospital mobile terminal-based services. (A) Seoul National University Hospital’s PHR service. Using the application, the patient can check information such as the treatment schedule, treatment details, results of test, and prescription drugs. (B) Samsung Seoul Hospital’s mobile EMR service (DARWIN). PHR: personal health record, EMR: Electronic Medical Record.

5. AI-based Services

AI-based services use intelligence (learning, reasoning, perception, and understanding abilities) as the main technological element for the diagnosis and prediction of diseases [ 43 , 44 ]. AI serves as the engine of clinical decision support systems, which help physicians make clinical decisions for diagnosis and prescription, thereby enhancing the effectiveness, efficiency, and safety of treatment. In particular, image data-based AI services are being developed rapidly.

Johns Hopkins Hospital in the United States maximized the efficiency of the hospital system by introducing GE’s capacity command center (CCC) [ 45 ]. Furthermore, systems for making decisions using clinical data, not just by reading medical image data, are in progress [ 46 ].

Several hospitals have made efforts to introduce smart speakers. Many research results are expected to emerge owing to improvements in smart speakers’ performance and increasing awareness among the public over the last 2 to 3 years. Patients can easily use the system, mainly for routine functions such as TV control, but the protection of personal information is a major issue [ 47 , 48 ].

Several studies have reported on dashboard designs such as CCC, but reports on system improvements are rare. CCC was first applied in 2016, but only usability case reports have been published [ 49 ]. Although few studies have investigated the effects of AI dashboards, the effects of using data on system improvements have been demonstrated from various perspectives, and this framework has been applied to optimize human resource utilization, workflow improvement, and facilities [ 50 ].

6. Robot Services

Robot services involve robots performing medical actions on behalf of humans. The scope of these services in various fields includes surgery, rehabilitation, nursing care, and logistics.

As a remote collaborative care robot, “Rudy” by INF Robotics has nursing and telemedicine capabilities [ 43 ]. Seoul National University Hospital developed a robotic system using telepresence technology, which promotes direct communication between patients and medical staff to enable remote collaborative care. An economic evaluation showed that people made 4.01 times fewer doctor visits per year since the robot was deployed [ 51 , 52 ].

The Baylor Scott and White Health Care System evaluated the effect of the pet robot PARO, an FDA-approved biofeedback device, on the treatment of symptoms related to dementia and found that PARO reduced the pulse rate and the use of analgesics and psychoactive drugs [ 53 ]. Boston Children’s Hospital reported that a conversational social robot provided to children aged 3–10 years had a positive emotional impact [ 54 ].

Other caring service robots [ 43 ] include the excretion support robot by Cyberdyne, a rehabilitation robot company; TUG, an autonomous transport robot developed by Aethon in the United States that transfers and transports blood, medicines, meals, and garbage [ 55 ]; Cuboid, developed by Wireless City Planning; HOSPI, developed by Panasonic; quarantine robots such as GermFalcon, an aircraft sterilization robot sold by Dimer UVC Innovations; and UVD, produced by UVD Robots, a Danish robot company [ 56 ]. Examples of the use of medical services provided by robots are shown in Figure 4 .

Examples of the use of medical services by robots. (A) Autonomous transport robot, TUG. Adapted from Siao et al. [ 55 ]. (B) Quarantine robot, UVD. Adapted from Holland et al. [ 56 ]. (C) Remote collaborative care robot at Seoul National University Hospital. Adapted from Lee and Kim [ 52 ].

7. Extended Reality Services

Extended reality refers to ultra-realistic technologies and services, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR enables a virtual experience by providing a virtual signal to the user’s senses, whereas AR augments reality by providing digital information. MR combines AR with VR in the real world [ 57 ].

VR in healthcare applications can potentially be used for clinical skills, training, and education of practitioners to address cognitive, psychological, motor, and functional impairments. Clinicians are showing more interest in clinical trials and applied research with VR simulations because of the encouraging comments published in the medical literature [ 58 ]. Furthermore, certain surgical skills acquired via a VR simulator can be transferred to the operating room, and actual surgical performance can be predicted from its results [ 59 ].

The Imperial College Healthcare NHS Trust developed AR technology for extremity reconstruction surgery using a three-dimensional (3D) vascular model with perforating vessels. This system enables the preoperative identification of anatomical structures without an incision by providing CT angiography information. It reduces anesthesia time and adverse outcomes related to surgery, shortens surgical training time, and provides remote support for surgery. In addition, through drug information provision technology using AR, patients can see the mechanism of action of drugs in a 3D graphic format [ 60 , 61 ].

Virtual Iraq is a program that is part of BraveMind and was developed by the US Department of Defense and the University of Southern California [ 62 ]. They created a VR program based on clinical data to treat soldiers with PTSD [ 63 ]. VIPAAR ( http://helplightning.com ) has been providing telemedicine technology using MR in North America. In addition, the Surgical Navigation Advanced Platform ( https://www.surgicaltheater.com/ ) is an AR/VR solution that helps doctors establish precise and stable surgical plans by identifying the three-dimensional structures of blood vessels and nerves in the brain during head and neck surgery.

Shionogi Pharmaceutical in Japan developed a digital therapeutic app, AKL-T01, which led to significant improvements in children with attention deficit hyperactivity disorder who regularly played games with individually optimized difficulty levels using the app [ 64 ]. A clinical trial of reSET, a drug addiction treatment app developed by Pear Therapeutics, showed that the proportion of patients who remained abstinent while using the app was 40.3%, which was higher than that of patients who did not use the app (17.6%) [ 65 ].

8. Telehealth

Telehealth refers to a form of medical services provided from a distant site using ICT and includes concepts such as a tele-intensive care unit (tele-ICU), tele-consulting, and tele-collaboration. Telehealth can be performed synchronously (telephone and video), asynchronously (patient portal messages, e-consults), and through virtual agents (chatbots) and wearable devices, without physical contact [ 66 ].

In the United States, American Well and Doctor on Demand provide primary care to people living in remote locations. Twine Health, acquired by Fitbit, allows patients to work with doctors to create action plans and use the device and apps to track progress and induce behavioral changes. Through tablets, patients have discussions with their health coaches and participate in the treatment process. TytoCare is a portable device that allows parents to consult with clinicians by sharing images and records through video chats to monitor their children’s diseases and analyze the results.

A tele-ICU [ 67 ] is connected through an external command center to patients and medical staff via real-time audio-visual and electronic means in the ICU to exchange medical information [ 68 – 72 ]. According to a study by Rosenfeld et al. [ 68 ] at Johns Hopkins Hospital, an analysis of the financial impacts of the tele-ICU program showed that the length of stay in the ICU was shortened and the clinical results were improved, resulting in a 24.6% reduction in cost per case ( Table 3 ).

Telehealth-related research trends

ICU: intensive care unit, LOS: length of stay.

IV. Discussion

From the viewpoint of patient safety, quality of care, cost-effectiveness, and patient-centered services, we have shown through practical examples that smart hospital services can create new value and insights that are not provided by existing medical services and provide them to patients and medical staff.

The eight services provided by smart hospitals are expected to contribute to solving the following issues in medical institutions related to the COVID-19 response. By applying location recognition and tracking, IoT, and AI technologies to asset management and patient monitoring in the ward, it will be possible to improve the efficiency of integrated nursing care services. Using robots, high-speed communication, XR, and telehealth technologies for patient-to-doctor tele-medicine and professional-to-professional telementoring, treatment, education, and training in medical institutions will be operated more safely and efficiently.

The following policy efforts are needed to motivate useful smart-hospital services within medical institutions. First, the government should encourage the spread of smart hospitals through various policies. The market size should be expanded through various policies to revitalize the smart-hospital-related industrial ecosystem. Appropriate incentive policies to motivate existing hospitals to transform into smart hospitals are needed. It is necessary to promote the spread of smart hospitals through direct and indirect incentives such as medical institution certification and related fee settings.

Second, it is necessary for medical institutions to revise and supplement the existing hospital work process so that the technological units of smart hospital services can be actively reflected in the design of new building spaces, as well as in spaces and facilities in existing hospitals.

With the introduction of smart hospitals, preventive health management is provided in various living spaces of local communities, such as homes and workplaces, using the latest ICTs, such as mobile and wearable sensors, which is expected achieve customer-centered medical services that can be accessed from the comfort of people’s residences through the virtual expansion of hospitals without physical space restrictions. In addition, based on the data collected through smart hospitals, specific detailed indicators related to the core aspects of medical value can be defined, quantitatively measured, and fed back to inform healthcare policy.

Third, industry needs to maintain a close cooperative relationship with medical institutions, which are the consumers of the services, and pursue the commercialization of related technologies based on demonstrations in the practical medical field.

Finally, stakeholders from industry, academia, research, and the medical field should form a standardization-related consultative body to maintain close communication, lead standardization in domestic and foreign industries, and contribute to improving competitiveness in the relevant market [ 73 ].

Data collected through smart hospital services within medical institutions will contribute to the establishment of national healthcare policies by defining and quantitatively measuring detailed indicators related to core aspects of medical value. The active introduction of the smart hospital concept will enable preventive health management in the various living spaces of local communities, such as homes and work, beyond the physical space constraints of existing hospitals. This “virtual expansion of hospitals” will contribute to the realization of customer-oriented medical services that individuals encounter in their daily lives.

Acknowledgments

This research is partly based on the KOSMI Issue Report (2020), which was supported by the Ministry of Health and Welfare, Republic of Korea. And this research was supported by the Ministry of Science and ICT, Korea, under the Information Technology Research Center support program (No. IITP-2020-2021-0-01833) supervised by the Institute for Information & communications Technology Promotion (IITP).

Conflict of Interest

Hyoun-Joong Kong is an editorial member of Healthcare Informatics Research; however, he did not involve in the peer reviewer selection, evaluation, and decision process of this article. Otherwise, no potential conflict of interest relevant to this article was reported.

• Conference proceedings

Proceedings of Second International Conference on Intelligent System

• João Manuel R. S. Tavares 0 ,
• Souvik Pal 1 ,
• Vassilis C. Gerogiannis 2 ,
• Bui Thanh Hung 3

Faculdade de Engenharia da Universidade do Porto (FEUP), Porto, Portugal

You can also search for this editor in PubMed   Google Scholar

Department of Computer Science and Engineering, Sister Nivedita University (Techno India Group), Kolkata, India

Department of digital systems, university of thessaly, larissa, greece, data science laboratory, faculty of information technology, industrial university of ho chi minh city, ho chi minh city, vietnam.

Highlights on intelligent system concepts and techniques

Presents research work on various computing models and their relevance to the intelligent system applications

Serves as a reference for researchers and practitioners in academia and industry

Part of the book series: Algorithms for Intelligent Systems (AIS)

Conference series link(s): ICIS: International Conference on Intelligence Science

616 Accesses

Conference proceedings info: ICIS 2023.

• Table of contents
• Other volumes

About this book

Editors and affiliations, about the editors, bibliographic information.

• Publish with us

Buying options

• Durable hardcover edition
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

This is a preview of subscription content, log in via an institution to check for access.

Table of contents (47 papers)

Front matter, deep convolutional neural networks for brain tumor image segmentation.

• Masood Ahamed Shaik, J. Jabez

A Comparison and Evaluation of Handwritten Digit Recognition (HDR) Algorithms

• Shashwat Singh, Riya Agrawal, Swapnil Singh, Ayush Ranjan, Sujoy Datta, Monideepa Roy

An Analysis of Machine Learning Algorithms for AQI Prediction

• Naresh Sharma, Rohit Sharma

Application of the Harmonic Runge–Kutta with Forward–Backward Technique by Parallelism

• Mahmood D. Jasim, Rabeea M. H. Darghoth

A Preconditioned Conjugate Gradient Multigrid Method for Multi-material Topology Optimization

• Nam G. Luu, Thanh T. Banh

Generated Graph for Text Encryption Algorithm Based on BRHC Curve

• Batool Hatem Akar AlKfari, Ruma Kareem K. Ajeena

Issues and Challenges of Digital Banking System

• Sarita Kumari Singh, Prasant Kumar Pattnaik, Sasmita Samanta

Impact of COVID-19 Pandemic on Assessment Process of Technical Education System-Analysis and Critical Review

• Srabanti Chakraborty, Srinka Basu, Indrajit Saha

Ensuring Energy Efficiency Using AI and Nudge Theory to Reach Sustainable Development Goals: A Research Framework

• Anastasios Fountis, Anastasia Kiritsi

Configuration and Evaluation of Models for Ecological Systems the Case of Distribution of Koala

• Yuting Zhao, Masoud Mohammadian, Hamed Sarbazhosseini

Design and Experiments of a Manta-Ray Robot for STEM Education

• Radzi Ambar, Muhammad Firdaus Roslan, Chew Chang Choon, Mohd Helmy Abd Wahab, Muhammad Mahadi Abdul Jamil, Suhazlan Suhaimi

House Price Forecast for Imbalanced Label Data by Metaheuristic Model

• V. Anjana Devi, Ahmed J. Obaid, Vithya Ganesan, V. Sri Anima Padmini, Subrata Chowdhury, Saurabh Adhikari

Unmanned Aerial Vehicles (UAVs): Performance Analysis of Routing Protocols for Optimized Operations

• Vinti Gupta, Dambarudhar Seth

Processing the Missing Value Based on the Linear Regression Approach

• Pham Le Son, Nguyen Ngoc Dung, Doan Van Thang

Improving Object Detection Versatility with 6G in VANETs

• Vithya Ganesan, R. Dhanalashmi, Ahmed J. Obaid, Viswanathan Ramasamy, Sri Anima Padmini, Subrata Chowdhury

Analysis of Human Behaviour on Movie Review

• Saksham Sharma, V. Vipin Deval, Vidushi, Mohd Anas Wajid, Bharat Bhushan, Mustafa Al-Asadi

Artificial Intelligence (ΑΙ) in Education—Current Trends

• Anastasios Fountis, Christos Lemonakis

The Study of Randomness Properties Exhibited by LAO-3D Lightweight Block Cipher Algorithm

• Nik Abdullah Nik Azura, A. H. Azni, Zakaria Nur Hafiza, Zakaria Abdul Alif

Association Rule Mining-Based Food Preferences Analysis Using FP-Growth Method

• Hamidah Jantan, Nurhidayah Supardi, Hayati Adilin Mohd Abd Majid, Ummu Fatihah Mohd Bahrin

Other Volumes

This book gathers selected high-quality research papers presented at the Second International Conference on Intelligent System (ICIS 2023), organized by Industrial University of Ho Chi Minh City, Vietnam during September 29–30, 2023. It discusses high-quality and cutting-edge research in the areas of informatics, intelligent systems, and smart technologies and applications. The book is a collection of the latest research articles in intelligent control, artificial intelligence, neural networks, knowledge discovery, decision support systems, soft computing, data mining, and ontologies, machine learning, intelligent measurement, and other related fields.

• Big Data Analysis
• Smart Computing and Communication
• Data Security and Data Privacy
• Proceedings of ICIS 2023
• Internet of Things
• Intelligent Data Analytics

João Manuel R. S. Tavares

Vassilis C. Gerogiannis

Bui Thanh Hung

Dr. João Manuel R. S. Tavares graduated in Mechanical Engineering at the Universidade do Porto, Portugal, in 1992. He also earned his M.Sc. degree and Ph.D. degree in Electrical and Computer Engineering from the Universidade do Porto in 1995 and 2001 and attained his Habilitation in Mechanical Engineering in 2015. He is a senior researcher at the Instituto de Ciência e Inovação em Engenharia Mecânica e Engenharia Industrial (INEGI) and an associate professor at the Department of Mechanical Engineering (DEMec) of the Faculdade de Engenharia da Universidade do Porto (FEUP). João Tavares is a co-editor of more than 60 books and co-author of more than 50 book chapters, 650 articles in international and national journals and conferences, and 3 international and 3 national patents.

Dr. Souvik Pal   is an associate professor in the Department of Computer Science and Engineering, Sister Nivedita University (Techno India Group), Kolkata, India. Prior to that, he was associated with Brainware University, Kolkata, India; JIS College of Engineering, Nadia; Elite College of Engineering, Kolkata; and Nalanda Institute of Technology, Bhubaneswar, India. Dr. Pal received his B.Tech., M.Tech., and Ph.D. degrees in the field of Computer Science and Engineering. He has more than a decade of academic experience. He is an author or co-editor of 14 books from reputed publishers, including Elsevier, Springer, CRC Press, and Wiley. He is serving as a series editor for  “ Advances in Learning Analytics for Intelligent Cloud-IoT Systems, ” published by Scrivener Publishing (Scopus-indexed) and  “ Internet of Things: Data-Centric Intelligent Computing, Informatics, and Communication, ” published CRC Press, Taylor & Francis Group, USA. Dr. Pal has published a number of research papers in Scopus / SCI-indexed international journals and conferences.

Dr. Vassilis C. Gerogiannis holds a Diploma in Computer Engineering and Informatics, as well as a Ph.D. in Software Engineering from the University of Patras, Greece. He is a full-time professor and head of the Department of Digital Systems at the University of Thessaly in Greece. His academic subject is “Analysis and Design of Systems and Projects with emphasis on Decision Making.” In the recent past, he was a full-time professor in the Department of Business Administration at the Technological Educational Institute of Thessaly, Greece He is also a visiting professor at foreign Higher Education Institutes/Universities. He acts as a guest editor, member of the editorial board and reviewer in international journals. He serves as program chair, member of the organization/technical committee and invited speaker in international conferences. He has received the “best paper award” in two international conferences. His research interests include Software Engineering, Project Management and Decision Making.

Dr. Bui Thanh Hung received his M.S. degree and Ph.D. degree from Japan Advanced Institute of Science and Technology (JAIST) in 2010 and in 2013. He is currently a researcher at Data Science Laboratory, Faculty of Information Technology, Industrial University of Ho Chi Minh City, Vietnam. He has completed 2 projects and published 16 journals, 19 book chapters, 36 International conference papers and 16 domestic conference papers.  He had four best paper awards of FICTA 2018, RICE 2020, ICAIAA 2021, and CIEMA 2022. He was selected as Excellent Scientific Researcher of Thu Dau Mot University in 2019, 2020, and 2021 and got CIEMA Research Excellence Awards 2022. He is a reviewer of many reputed journals and conferences. He is also a keynote speaker of many domestic and international conferences. His main research interests are natural language processing, machine learning, machine translation, text processing, data analytics, computer vision, and artificial intelligence.

Book Title : Proceedings of Second International Conference on Intelligent System

Book Subtitle : ICIS 2023

Editors : João Manuel R. S. Tavares, Souvik Pal, Vassilis C. Gerogiannis, Bui Thanh Hung

Series Title : Algorithms for Intelligent Systems

DOI : https://doi.org/10.1007/978-981-99-8976-8

Publisher : Springer Singapore

eBook Packages : Intelligent Technologies and Robotics , Intelligent Technologies and Robotics (R0)

Copyright Information : The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024

Hardcover ISBN : 978-981-99-8975-1 Due: 27 April 2024

Softcover ISBN : 978-981-99-9012-2 Due: 27 April 2025

eBook ISBN : 978-981-99-8976-8 Due: 27 April 2024

Series ISSN : 2524-7565

Series E-ISSN : 2524-7573

Edition Number : 1

Number of Pages : XII, 518

Number of Illustrations : 53 b/w illustrations, 218 illustrations in colour

Topics : Computational Intelligence , Communications Engineering, Networks , Statistics, general , Systems and Data Security

Policies and ethics

• Find a journal
• Track your research

More From Forbes

5 tips to enhance your research paper’s visibility and altmetric score.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

US evangelist Billy Graham addresses a crowd of football supporters at Stamford Bridge, London, ... [+] during half-time at the match between Chelsea and Newcastle United. (Photo by Edward Miller/Getty Images)

I previously wrote about the importance of attracting public attention to scientific research . In today’s world, where billions of people are attached to their digital devices watching the very addictive but often useless TikTok content or receiving instant gratification by engaging in meaningless debates about celebrities, scientists need to find creative ways to have their research noticed. Popularizing scientific research helps inspire the younger generations to go into science and provide the general public with a sense of optimism enabling the government to channel more resources into science. People do need inspiration. But very often, even very important scientific breakthroughs requiring many years, hard work, skill, funding, and genuine serendipity go largely unnoticed by the general public.

One of the best ways to measure expert and public attention is the cumulative Altmetric Attention Score , originally developed by Digital Science and adopted by many prestigious publishers, including Nature Publishing Group. Every Nature paper and the papers published by pretty much every credible publisher are tracked by Digital Science by the Document Object Identification (DOI) or the Unique Resource Locator (URL) . While Altmetric has many limitations, for example, it does not track LinkedIn posts and may not adequately cover the impact of top-tier media coverage, at the moment it is the blueprint for tracking attention.

Altmetric Score in The Age of Generative AI

Media attention is likely to be very important in the age of generative AI. Many modern generative systems, such as ChatGPT, Claude, Mistral, and Gemini, as well as hundreds of Large Language Models (LLMs) in China, use the data from the same sources referenced in Altmetric to learn. The more times generative systems see the same concept presented in different contexts, the better they learn. So if you want to contribute to the training of AI systems that may thank you for it in the future - Altmetric is the way to go.

So what can a research group do to ensure they are communicating their findings effectively and increasing the visibility of their research to ensure it gets reflected in the Altmetric Attention Score?

New iPhone 16 Pro Exclusive Reveals Apple’s Bonus Button

The fallout show appears to have given fallout 4 a canon ending, goldman sachs issues stark bitcoin halving price warning.

Altmetric openly discloses the weights of the various sources and the scoring algorithm is relatively straightforward. It is easy to learn, and there are multiple online resources providing advice on how to share your research in ways that will be captured by Altmetric. Cambridge University Press published a guideline to Altmetric for the authors on how to popularize their research with Altmetric in mind. Wolters Kluwer put out a guide and the editor of Toxicology and Pathology wrote a comprehensive overview of Altmetric and how to use it. Surprisingly, this overview got an Altmetric Attention Score of only 4 at the time of the writing, but was cited 137 times according to Google Scholar .

Altmetric monitors social networks, including X (formerly Twitter), Facebook, and Reddit; all major top-tier mainstream media, mainstream science blogs, policy documents, patents, Wikipedia articles, peer review websites, F1000, Syllabi, X (formerly Twitter), tracked Facebook pages, Reddit, one of the Stack Exchange sites, and Youtube. Unfortunately, several powerful platforms, including LinkedIn, are not currently tracked.

The popularity of the paper depends on many factors. Firstly, it has to be novel, trendy, and newsworthy. You are unlikely to get high Altmetric Score with a boring topic. Secondly, papers coming out of popular labs in top-tier academic institutions and in top journals are likely to attract more attention. Often, the communications officers in these academic institutions work closely with the media to amplify notable research. Celebrity companies, for example, Google DeepMind, consistently get higher coverage.

Screenshot of the Altmetric Attention Score "Flower" showing several tracked sources

Here are the five tips for increasing the visibility of your work and ensuring that reach is tracked and reflected by Altmetric:

1. Understand How Altmetric System Works

Congratulations, if you read this article and looked at what sources are tracked by Altmetric. Most likely, you got the basics and will be able to get a “balanced flower” by making a press release, tweeting the DOI of the paper on X, posting a video overview of your paper on Youtube, announcing on Reddit (I still need to learn how to do this).

To understand how Altmetric works, I emailed a few questions to Miguel Garcia, Director of Product and Data Analytics Hub at Digital Science and my first question was wether the Altmetric algorithm is open source. “The Altmetric Attention Score's calculation is not open source but we try to provide as much information as possible around how we calculate it here, and are currently considering what steps we might take to make our algorithms more transparent.” He also provided a link to how the Altmetric Attention Score is calculated.

Many professionals use LinkedIn as the primary social media resource and I was wondering why Altmetric stopped tracking it. Bad news - technical reasons prevent tracking DOIs on LinkedIn. Good news - they are actively seeking ways to appropriately track mentions on LinkedIn and we may see some news toward the end of the year.

My other big question was how does Altmetric count tweets and retweets on X. What if there are many posts from the same account? Miguel’s response was: “Re-tweets count less than original tweets. In addition to that, modifiers are applied to the type of account that is tweeting in order to reduce the weight of the tweet in situations where we find signals of bias or promiscuity (for example a journal publisher only tweeting their own articles). Besides that, we have conditions around the maximum number of retweets in order to limit the maximum impact they would have.”

So tweeting the article many times will not help you. But if other scientists tweet you paper with a DOI - these tweets will get counted. So tweet others as you would like to be tweeted.

2. Make a Press Release and Distribute to Science-focused Media

If your paper is significant, for example, you elucidated novel disease biology, discovered a new drug, developed a new fancy algorithm, designed a new material, or developed a new application for a quantum computer, it is worthwhile investing some time and resources in writing a press release. If you are working for an academic institution, most likely they have a communications office that will help you. If you do not have this luxury, you will need to learn how to write a press release. Plenty of free online guides cover the basics of press release writing. And press releases are one area where ChatGPT and other generative tools do surprisingly well. Upload your paper and ask it to write a press release, check for errors or exaggerations, edit, and you are ready to go. Just make sure to include the DOI and the URL of your paper. A proper business press release on BusinessWire or PRNewswire may cost several thousand dollars. In my opinion, these resources are dramatically overcharging while providing little service. I don't remember a case where a journalist picked up our news based on a commercial press release. But these releases are often reposted by other online press release distributors and the boost to Altmetric may be considerable. The default news release distribution service for research news is EurekAlert. This resource may sometimes result in journalistic coverage as many reporters are using it for science news. There are many free resources you can use if you do not have any budget.

Once the press release is issued, share it with the media. Share the resulting news coverage via your social networks and contacts. Many journalists track the popularity of their news articles and giving them several thousand extra views from professional audience and increasing their social following increases the chances that they will cover the next important research paper.

3. Make a Blog Post

Writing a blog post can be longer and more comprehensive than the press release. Make sure to add fancy diagrams and graphical explainers. You can share the blog post with the journalists at the same time as the press release. Your blog may serve as a source of inspiration for third party news coverage. Make sure to reference the DOI and URL of your paper.

If your paper is in one of the Nature journals, consider writing a “Behind the Paper" blog post on Nature Bioengineering Community. Surprisingly, these blogs are rarely picked up by Altmetric but may serve as a source of inspiration for the journalists and social media influencers. Plus, it is a resource by the Nature Publishing Group.

4. Tweet and Ask Your Team Members to Tweet

Each post on X gives you a quarter of an Altmetric point. If your paper goes viral on X, your Altmetric score can be considerable. Plus, once journalists notice that it went viral, they will be more likely to cover the story, further increasing the score.

Try to choose the time of the post, the hashtags, and the images wisely. Since Elon Musk took over X and opened the algorithm it became very transparent and easy to optimize for. Here are the top 10 tips for boosting attention for a post on X. Make sure to include the DOI or the URL of the paper for Altmetric to find the post.

5. Experiment, Learn, Repeat

My highest Altmetric Attention Score core to date was around 1,500 for a paper in Nature Biotechnology published in 2019, where we used a novel method for designing small molecules called Generative Tensorial Reinforcement Learning (GENTRL) to generate new molecules with druglike properties that got synthesized and tested all the way into mice. In 2024, we went further and showed that an AI-generated molecule for an AI-discovered target was tested all the way up to Phase II human trials, but the paper published in Nature Biotechnology, let’s call it the TNIK paper , has achieved a score ofjust over 600 to date. So what has changed and what can we learn from these two papers?

The popularity of the paper depends on many factors. Ones which capture the public imagination or have widespread appeal are of course, much more likely to gain traction online. When we published the GENTRL paper in 2019, Generative AI was in its infancy, and there are pretty much no other companies that I heard of at the intersection of generative AI and drug discovery. We also published multiple articles in this field in the years leading to that paper and many key opinion leaders (KOLs) followed us. That following included a small army of generative AI skeptics who not only contributed to multiple rejections in peer-reviewed journals but also openly criticized this approach in social networks. This criticism also helped boost the Altmetric Score and bring more attention to the study. So first learning from this exercise - negative publicity helps overall publicity. As long as you are certain that your research results are honest - leave room for criticism and even help expose your paper’s weaknesses. Critics are your greatest Altmetric boosters. Since readers and, by extension journalists, react to negative news and drama stronger than to positive news, critical reviews will boost your Altmetric as long as the DOI or URL of the paper is properly referenced.

Secondly, papers coming out of popular labs in top-tier academic institutions and in top journals are likely to attract more attention. Often, the communications officers in these academic institutions work closely with the media to amplify notable research. Celebrity companies, for example, Google DeepMind, always get a higher level of coverage. For example, the AlphaFold paper published in July 2021 in Nature got an Altmetric Attention Score of over 3,500 . Even though I have not seen any drugs out of AlphaFold reaching preclinical candidate status, I predict the popularity of this tool will result in the first Nobel Prize in this area. Therefore, in order to become famous and popularize your research more effectively, it is a good idea to build up the public profile of yourself and your work. For example, Kardashians are famous for being famous .

Be careful with Wikipedia. I made a mistake explaining the importance of Wikipedia to students when lecturing on the future of generative AI, and one or two of them got banned for expanding the articles with paper references. Wikipedia requires that these are added by independent editors rather than the authors of papers themselves, but if some editors do not like it, they will not go deep or investigate - they will assume wrongdoing. So it is better to avoid even talking about Wikipedia. References there should happen naturally and often some of the more popular papers get picked up and referenced by veteran editors.

Experimenting with Altmetric will also help you explore new strategies for popularizing scientific research and develop new strategies for inspiring people to learn or even get into the new exciting field. UNESCO estimates that there was just over 8 million full-time equivalent (FTE) researchers in 2018 globally. Only a fraction of these are in biotechnology - less than 0.01% of the global population. If you motivate a million students to go into biotechnology by popularizing your research, you double this number.

• Editorial Standards
• Reprints & Permissions

Ohio State navigation bar

• BuckeyeLink
• Find People
• Search Ohio State
• Search  

Ohio State nav bar

The Ohio State University

Public Health Buckeyes: Angela Falconi

BSPH student combines passions for health care, policy

Falconi has been involved in CPH research and is an active member of Ohio State's Pilipino Student Association.

Meet Angela Falconi, a fourth-year student specializing in  environmental public health who aspires to advocate for others through public health policy.

What inspired you to pursue a public health education?

Growing up, I was surrounded by both medicine and public policy because of my parents. Since I was six, my father, a politician and elected official, had me act as his unofficial campaign staff—knocking on doors with him to speak to voters, sitting in on city council meetings and accompanying him to various events. My mother, a pediatric physician, inspired me to pursue a career in medicine by showing me the impact that she’s made on her patients and always encouraging me to learn more about the health care field. When choosing my major, it felt natural to me to combine policy and health into public health.

What public health topics are you passionate about?

“Your zip code determines your health.”

This is one of the most important phrases I have learned in my public health courses, and as a volunteer at Helping Hands Health and Wellness Center, a free clinic which provides health care services for the uninsured and underinsured. I see the realities of this phrase in the patients who I work with. 

As an aspiring elected official, I want to create health care reform which helps individuals the health care system has failed to provide with affordable service.

You spent last summer in Washington, D.C. interning in the U.S. Senate. What was that experience like?

I worked (there) through the IMPACT program, created by the US-Asia Institute in coordination with the Embassy for the Philippines for Filipino students interested in public policy. Working and living in D.C. was one of the best experiences I have had in my undergraduate career because I was able to learn about and research health care policy on the national stage, which is exactly what I hope to do in my future career.

What have you enjoyed most about being involved in research as a student?

I am a research assistant for the Consumer Access Project which utilizes a secret shopper survey of Affordable Care Act (ACA) insurance marketplace plan networks to study these barriers and inequities, including disparities related to race. I have loved getting to work with  Wendy Xu as she has helped me learn more about the research process as well as how everyday Americans experience the health care system.

What kind of extracurricular activities are you involved in?

The Pilipino Student Association (PSA) has been my home away from home since the start of my time at Ohio State. It has not only allowed me to learn more about my Filipino culture, but I met my best friend through this organization. I have been involved in PSA in numerous roles: culture night coordinator, vice-president internal, president and now dance leader. 

As dance leader, I lead PSA’s tinikling team. Tinikling is a dance which involves two people beating, sliding, and tapping two bamboo poles on the ground while two people dance above the sticks, trying not to get caught in between them. Our latest performance from PSA’s culture show “Barrio” was in October. I choreographed, taught and performed the modern part of this dance!

What are your goals for the future?

I hope to not only assist individual patients as a physician, but I also hope to help others on the national scale by being an advocate as an elected official. I hope to apply the experiences and lessons that I have learned from my time at Ohio State into my future career in the field of health policy.

More news stories

Public Health Buckeyes: Kaitlyn Jones

Public Health Buckeyes: Molly Mills

*****      

About The Ohio State University College of Public Health

The Ohio State University College of Public Health is a leader in educating students, creating new knowledge through research, and improving the livelihoods and well-being of people in Ohio and beyond. The College's divisions include biostatistics, environmental health sciences, epidemiology, health behavior and health promotion, and health services management and policy. It is ranked 29 th  among all colleges and programs of public health in the nation, and first in Ohio, by  U.S. News and World Report. Its specialty programs are also considered among the best in the country. The MHA program is ranked 8 th , the biostatistics specialty is ranked 22 nd , the epidemiology specialty is ranked 25 th and the health policy and management specialty is ranked 17 th .

• CEPH Report for Accreditation 2017-2024
• CPH Competencies Surveys

Faculty and Staff

• Faculty and Staff Resources
• Emergency and Safety Information

News and Communications

• News and Events
• Website Feedback

© 2024 College of Public Health            250 Cunz Hall, 1841 Neil Ave.            Columbus, OH 43210            Phone: 614-292-8350            Contact Admissions             Request an alternate format of this page             Privacy Policy

• Dean's Message
• Mission, Vision and Values
• Diversity, Equity and Inclusive Excellence
• Strategic Plan
• Undergraduate Programs
• Graduate Programs
• Office of Academic Programs and Student Services
• Biostatistics
• Environmental Health Sciences
• Epidemiology
• Health Behavior and Health Promotion
• Health Services Management and Policy
• Health Outcomes and Policy Evaluation Studies
• Center for Public Health Practice
• Information Sessions
• BSPH + MPH in 5 years
• Dual/Combined Degrees
• Minors/ Specializations/ Certificates
• Bachelor of Science in Public Health
• Master of Public Health
• Master of Health Administration
• Master of Science
• Doctor of Philosophy
• Student Forms and Resources
• Minors / Specializations / Certificates
• Competencies
• Course Descriptions
• Scholarships
• Student Organizations
• Advising and Student Services
• CPH Graduate Student Handbook
• Curriculum Guides
• MPH Applied Practice Experience
• MPH Integrative Learning Experience
• MHA Administrative Residency
• CPH Undergraduate Student Handbook
• Internships and Research
• Education Abroad
• Career Development
• Public Health Career Paths
• Career Events
• Career Resources
• Employer Resources
• Dean’s Thought Leader Series
• Public Health Buckeyes
• Media Requests