good research paper features

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

70k Accesses

22 Citations

72 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

What is Qualitative in Qualitative Research

Patrik Aspers & Ugo Corte

Environmental-, social-, and governance-related factors for business investment and sustainability: a scientometric review of global trends

Hadiqa Ahmad, Muhammad Yaqub & Seung Hwan Lee

Literature reviews as independent studies: guidelines for academic practice

Sascha Kraus, Matthias Breier, … João J. Ferreira

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

What Should Be the Characteristics of a Good Research Paper?

by team@blp

In miscellaneous.

When people want to get answers to various issues, they search for information on the problems. From their findings, they expand them, aiming to agree or refute them. Research papers are common assignments in colleges. 

They follow specific research and writing guidelines to answer particular questions or assigned topics. They look into the critical topic of credible research sources and argue their findings in an orderly manner. To be termed as good, the research paper must bear the following characteristics.

In this Article

Gives credit to previous research work on the topic

• It’s hooked on a relevant research question.

It must be based on appropriate, systematic research methods

• The information must be accurate and controlled.
• It must be verifiable and rigorous.

Be careful with the topic you choose

Decide the sources you want to use, create your thesis statement , plan your points, write your paper, characteristics of a good research paper.

Writing a research paper aims to discover new knowledge, but the knowledge must have a base. Its base is the research done previously by other scholars. The student must acknowledge the previous research and avoid duplicating it in their writing process.

A college student must engage in deep research work to create a credible research paper. This makes the process lengthy and complex when choosing your topic, selecting sources, and developing its design. In addition, it requires a great deal of knowledge to piece everything together. Fortunately, Studyclerk will give you professional help anytime you need it. If you do not have enough knowledge and time to write a paper on your own, you can ask for  research paper help  by StudyClerk, where experienced paper writers will write your paper in no time. You can trust their expert writers to handle your assignment well and get a well-written paper in a short time.

It’s hooked on a relevant research question .

All the time a student spends researching multiple sources is to answer a specific research question. The question must be relevant to the current needs. This question guides them into the information they use or the line of argument they take.

The methodology of research a student chooses will determine the value of the information they get or give. The methods must be valid and credible to provide reliable outcomes. Whether the student chooses a qualitative, quantitative, or mixed approach, they must all be valuable and relevant. 

The information must be accurate and controlled .

A good research paper cannot be generalized information but specific, scientific information. That is why they must include references and record tests or information accurately. Moreover, they must keep the information controlled by staying within the topic from the first step of research to the last. 

It must be verifiable and rigorous .

The student must use information or write arguments that can be verified. If it’s a test, it must be replicable by another researcher. The sources must be verifiable and accurate. Without rigorous deep  research strategies , the paper cannot be good. They must put a lot of labor into both the writing and research processes to ensure the information is credible, clear, concise, original, and precise. 

How to write a good research paper

To write a good research paper, you must first understand what kind of question you have been assigned. Then, you will choose the best topic that you will love to write about. The following points will help you write a good research paper.

You must select a topic you love. Go for a topic that will be easier to research, which will give you a broader area of study. 

Your instructor doesn’t restrict you on the sources you must use. Broaden your mind so that you don’t limit yourself to specific sources of information. Sometimes you will get helpful information from sources you slightest thought as good.

Write your central statement to base your position on the research. Make it coherent contentious, and let it be a summary of your arguments.

Create an outline that will guide you when arguing your points

• Start  with the most vital points and smooth the flow.
• Pay attention to  paragraph structure  and let your arguments be clear.
• Finish with a compelling conclusion, and don’t forget to cite your sources.

A research paper requires extensive research methods to get solid points for supporting your stand. First, the sources you use must be verifiable by any other researcher. You must ensure your research work is original for your paper to be credible. Third, each point should be coherent with each paragraph. Finally, your research findings must be tagged on the research question and provide answers that apply to the current society. 

Author’s Bio

Helen Birk is an online freelance writer who holds an outstanding record of helping numerous students do their academic assignments. She is an expert in essays and thesis writing, and students simply love her for her high-quality work. In addition, she enjoys cycling, doing pencil sketching, and listening to spiritual podcasts in her free time.

Guide on Where Can I Buy a Custom Essay Online?

Explore Where Can I Buy a Custom Essay Online? How to write an essay? Such a question is a main problem for schoolchildren and students because more and more often teachers offer to write a text in the essay genre. This type has become very popular, so we have...

Summer Office Wear For Men: A Style Guide For Teachers

As we dive into the scorching summer months, the struggle to maintain a professional appearance while staying cool and comfortable in the classroom becomes all too real. With global warming, expect harsh temperature changes with every seasonal transition.   But don’t...

From Application to Offer: Navigating the Job Search Journey

Embarking on the job search journey can be an exhilarating yet daunting process. From crafting your application to the moment you receive that much-anticipated job offer, each step requires careful planning, preparation, and execution. This comprehensive guide aims to...

Mastering the Test: Strategies for Exam Success

The path to academic and professional certification is often paved with the challenge of passing rigorous exams. For many, the mere thought of an upcoming test can evoke a mix of anticipation and apprehension. However, success is not just a product of what you know...

How the Discovery Stage Contributes to the Success of a Business Project

A positive return on your investment in a business project is only possible when you enter the market with an offer demanded by consumers. They determine the success of the company through their purchases and give it a chance for further growth. That is why special...

Exploring Innovative Educational Leadership: A Comparative Analysis of Global Teaching Trends

If you pay attention to the educational sector, you'll notice a shift toward transformational leadership across all academic levels. It's an approach born out of the growing need to prepare students for a changing world where learners are no longer insulated from...

The Importance of STEM Education

In today's fast-paced world, STEM education, encompassing Science, Technology, Engineering, and Mathematics, plays a pivotal role in shaping the future. It's not merely a set of subjects; it's a pathway to innovation, problem-solving, and economic growth. Fostering...

Building Blocks for a Project-Powered Future

Strolling through the campus nowadays, it’s not just about cramming for exams or pulling an all-nighter to finish an essay. It’s more – we’re gearing up students for those real-world challenges, for a career that’s more puzzle-solving than paint by numbers. Life after...

Artistic Expression: CapCut Online’s Text Additions to Images

Introduction In the realm of digital creativity, CapCut's online creative suite stands as a beacon for artistic expression. While renowned for its video editing prowess, the web-based version of CapCut, also known as the YouTube editor, conceals a remarkable feature:...

Enhancing Math Education for Diverse Learners with Customized Worksheets: Tips & Ideas

Mathematics is a fundamental subject that holds immense importance in a student's academic journey and life beyond the classroom. However, teaching mathematics to diverse learners, who have varying learning styles and abilities, can be quite challenging. One effective...

WANT MORE AMAZING CONTENT?

• Free GRE Practice Questions
• 100+ Personal Statement Templates
• 100+ Quotes to Kick Start Your Personal Statement
• 390 Adjectives to Use in a LOR

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

13.1 Formatting a Research Paper

Learning objectives.

• Identify the major components of a research paper written using American Psychological Association (APA) style.
• Apply general APA style and formatting conventions in a research paper.

In this chapter, you will learn how to use APA style , the documentation and formatting style followed by the American Psychological Association, as well as MLA style , from the Modern Language Association. There are a few major formatting styles used in academic texts, including AMA, Chicago, and Turabian:

• AMA (American Medical Association) for medicine, health, and biological sciences
• APA (American Psychological Association) for education, psychology, and the social sciences
• Chicago—a common style used in everyday publications like magazines, newspapers, and books
• MLA (Modern Language Association) for English, literature, arts, and humanities
• Turabian—another common style designed for its universal application across all subjects and disciplines

While all the formatting and citation styles have their own use and applications, in this chapter we focus our attention on the two styles you are most likely to use in your academic studies: APA and MLA.

If you find that the rules of proper source documentation are difficult to keep straight, you are not alone. Writing a good research paper is, in and of itself, a major intellectual challenge. Having to follow detailed citation and formatting guidelines as well may seem like just one more task to add to an already-too-long list of requirements.

Following these guidelines, however, serves several important purposes. First, it signals to your readers that your paper should be taken seriously as a student’s contribution to a given academic or professional field; it is the literary equivalent of wearing a tailored suit to a job interview. Second, it shows that you respect other people’s work enough to give them proper credit for it. Finally, it helps your reader find additional materials if he or she wishes to learn more about your topic.

Furthermore, producing a letter-perfect APA-style paper need not be burdensome. Yes, it requires careful attention to detail. However, you can simplify the process if you keep these broad guidelines in mind:

• Work ahead whenever you can. Chapter 11 “Writing from Research: What Will I Learn?” includes tips for keeping track of your sources early in the research process, which will save time later on.
• Get it right the first time. Apply APA guidelines as you write, so you will not have much to correct during the editing stage. Again, putting in a little extra time early on can save time later.
• Use the resources available to you. In addition to the guidelines provided in this chapter, you may wish to consult the APA website at http://www.apa.org or the Purdue University Online Writing lab at http://owl.english.purdue.edu , which regularly updates its online style guidelines.

General Formatting Guidelines

This chapter provides detailed guidelines for using the citation and formatting conventions developed by the American Psychological Association, or APA. Writers in disciplines as diverse as astrophysics, biology, psychology, and education follow APA style. The major components of a paper written in APA style are listed in the following box.

These are the major components of an APA-style paper:

Body, which includes the following:

• Headings and, if necessary, subheadings to organize the content
• In-text citations of research sources
• References page

All these components must be saved in one document, not as separate documents.

The title page of your paper includes the following information:

• Title of the paper
• Author’s name
• Name of the institution with which the author is affiliated
• Header at the top of the page with the paper title (in capital letters) and the page number (If the title is lengthy, you may use a shortened form of it in the header.)

List the first three elements in the order given in the previous list, centered about one third of the way down from the top of the page. Use the headers and footers tool of your word-processing program to add the header, with the title text at the left and the page number in the upper-right corner. Your title page should look like the following example.

The next page of your paper provides an abstract , or brief summary of your findings. An abstract does not need to be provided in every paper, but an abstract should be used in papers that include a hypothesis. A good abstract is concise—about one hundred fifty to two hundred fifty words—and is written in an objective, impersonal style. Your writing voice will not be as apparent here as in the body of your paper. When writing the abstract, take a just-the-facts approach, and summarize your research question and your findings in a few sentences.

In Chapter 12 “Writing a Research Paper” , you read a paper written by a student named Jorge, who researched the effectiveness of low-carbohydrate diets. Read Jorge’s abstract. Note how it sums up the major ideas in his paper without going into excessive detail.

Write an abstract summarizing your paper. Briefly introduce the topic, state your findings, and sum up what conclusions you can draw from your research. Use the word count feature of your word-processing program to make sure your abstract does not exceed one hundred fifty words.

Depending on your field of study, you may sometimes write research papers that present extensive primary research, such as your own experiment or survey. In your abstract, summarize your research question and your findings, and briefly indicate how your study relates to prior research in the field.

Margins, Pagination, and Headings

APA style requirements also address specific formatting concerns, such as margins, pagination, and heading styles, within the body of the paper. Review the following APA guidelines.

Use these general guidelines to format the paper:

• Set the top, bottom, and side margins of your paper at 1 inch.
• Use double-spaced text throughout your paper.
• Use a standard font, such as Times New Roman or Arial, in a legible size (10- to 12-point).
• Use continuous pagination throughout the paper, including the title page and the references section. Page numbers appear flush right within your header.
• Section headings and subsection headings within the body of your paper use different types of formatting depending on the level of information you are presenting. Additional details from Jorge’s paper are provided.

Begin formatting the final draft of your paper according to APA guidelines. You may work with an existing document or set up a new document if you choose. Include the following:

• Your title page
• The abstract you created in Note 13.8 “Exercise 1”
• Correct headers and page numbers for your title page and abstract

APA style uses section headings to organize information, making it easy for the reader to follow the writer’s train of thought and to know immediately what major topics are covered. Depending on the length and complexity of the paper, its major sections may also be divided into subsections, sub-subsections, and so on. These smaller sections, in turn, use different heading styles to indicate different levels of information. In essence, you are using headings to create a hierarchy of information.

The following heading styles used in APA formatting are listed in order of greatest to least importance:

• Section headings use centered, boldface type. Headings use title case, with important words in the heading capitalized.
• Subsection headings use left-aligned, boldface type. Headings use title case.
• The third level uses left-aligned, indented, boldface type. Headings use a capital letter only for the first word, and they end in a period.
• The fourth level follows the same style used for the previous level, but the headings are boldfaced and italicized.
• The fifth level follows the same style used for the previous level, but the headings are italicized and not boldfaced.

Visually, the hierarchy of information is organized as indicated in Table 13.1 “Section Headings” .

Table 13.1 Section Headings

A college research paper may not use all the heading levels shown in Table 13.1 “Section Headings” , but you are likely to encounter them in academic journal articles that use APA style. For a brief paper, you may find that level 1 headings suffice. Longer or more complex papers may need level 2 headings or other lower-level headings to organize information clearly. Use your outline to craft your major section headings and determine whether any subtopics are substantial enough to require additional levels of headings.

Working with the document you developed in Note 13.11 “Exercise 2” , begin setting up the heading structure of the final draft of your research paper according to APA guidelines. Include your title and at least two to three major section headings, and follow the formatting guidelines provided above. If your major sections should be broken into subsections, add those headings as well. Use your outline to help you.

Because Jorge used only level 1 headings, his Exercise 3 would look like the following:

Citation Guidelines

In-text citations.

Throughout the body of your paper, include a citation whenever you quote or paraphrase material from your research sources. As you learned in Chapter 11 “Writing from Research: What Will I Learn?” , the purpose of citations is twofold: to give credit to others for their ideas and to allow your reader to follow up and learn more about the topic if desired. Your in-text citations provide basic information about your source; each source you cite will have a longer entry in the references section that provides more detailed information.

In-text citations must provide the name of the author or authors and the year the source was published. (When a given source does not list an individual author, you may provide the source title or the name of the organization that published the material instead.) When directly quoting a source, it is also required that you include the page number where the quote appears in your citation.

This information may be included within the sentence or in a parenthetical reference at the end of the sentence, as in these examples.

Epstein (2010) points out that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (p. 137).

Here, the writer names the source author when introducing the quote and provides the publication date in parentheses after the author’s name. The page number appears in parentheses after the closing quotation marks and before the period that ends the sentence.

Addiction researchers caution that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (Epstein, 2010, p. 137).

Here, the writer provides a parenthetical citation at the end of the sentence that includes the author’s name, the year of publication, and the page number separated by commas. Again, the parenthetical citation is placed after the closing quotation marks and before the period at the end of the sentence.

As noted in the book Junk Food, Junk Science (Epstein, 2010, p. 137), “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive.”

Here, the writer chose to mention the source title in the sentence (an optional piece of information to include) and followed the title with a parenthetical citation. Note that the parenthetical citation is placed before the comma that signals the end of the introductory phrase.

David Epstein’s book Junk Food, Junk Science (2010) pointed out that “junk food cannot be considered addictive in the same way that we think of psychoactive drugs as addictive” (p. 137).

Another variation is to introduce the author and the source title in your sentence and include the publication date and page number in parentheses within the sentence or at the end of the sentence. As long as you have included the essential information, you can choose the option that works best for that particular sentence and source.

Citing a book with a single author is usually a straightforward task. Of course, your research may require that you cite many other types of sources, such as books or articles with more than one author or sources with no individual author listed. You may also need to cite sources available in both print and online and nonprint sources, such as websites and personal interviews. Chapter 13 “APA and MLA Documentation and Formatting” , Section 13.2 “Citing and Referencing Techniques” and Section 13.3 “Creating a References Section” provide extensive guidelines for citing a variety of source types.

Writing at Work

APA is just one of several different styles with its own guidelines for documentation, formatting, and language usage. Depending on your field of interest, you may be exposed to additional styles, such as the following:

• MLA style. Determined by the Modern Languages Association and used for papers in literature, languages, and other disciplines in the humanities.
• Chicago style. Outlined in the Chicago Manual of Style and sometimes used for papers in the humanities and the sciences; many professional organizations use this style for publications as well.
• Associated Press (AP) style. Used by professional journalists.

References List

The brief citations included in the body of your paper correspond to the more detailed citations provided at the end of the paper in the references section. In-text citations provide basic information—the author’s name, the publication date, and the page number if necessary—while the references section provides more extensive bibliographical information. Again, this information allows your reader to follow up on the sources you cited and do additional reading about the topic if desired.

The specific format of entries in the list of references varies slightly for different source types, but the entries generally include the following information:

• The name(s) of the author(s) or institution that wrote the source
• The year of publication and, where applicable, the exact date of publication
• The full title of the source
• For books, the city of publication
• For articles or essays, the name of the periodical or book in which the article or essay appears
• For magazine and journal articles, the volume number, issue number, and pages where the article appears
• For sources on the web, the URL where the source is located

The references page is double spaced and lists entries in alphabetical order by the author’s last name. If an entry continues for more than one line, the second line and each subsequent line are indented five spaces. Review the following example. ( Chapter 13 “APA and MLA Documentation and Formatting” , Section 13.3 “Creating a References Section” provides extensive guidelines for formatting reference entries for different types of sources.)

In APA style, book and article titles are formatted in sentence case, not title case. Sentence case means that only the first word is capitalized, along with any proper nouns.

Key Takeaways

• Following proper citation and formatting guidelines helps writers ensure that their work will be taken seriously, give proper credit to other authors for their work, and provide valuable information to readers.
• Working ahead and taking care to cite sources correctly the first time are ways writers can save time during the editing stage of writing a research paper.
• APA papers usually include an abstract that concisely summarizes the paper.
• APA papers use a specific headings structure to provide a clear hierarchy of information.
• In APA papers, in-text citations usually include the name(s) of the author(s) and the year of publication.
• In-text citations correspond to entries in the references section, which provide detailed bibliographical information about a source.

Writing for Success Copyright © 2015 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Research Paper

29 December 2023

last updated

A research paper is a product of seeking information, analysis, human thinking, and time. Basically, when scholars want to get answers to questions, they start to search for information to expand, use, approve, or deny findings. In simple words, research papers are results of processes by considering writing works and following specific requirements. Besides, scientists research and expand many theories, developing social or technological aspects of human science. However, in order to write relevant papers, they need to know a definition of the research, structure, characteristics, and types.

Definition of What Is a Research Paper and Its Meaning

A research paper is a common assignment. It comes to a situation when students, scholars, and scientists need to answer specific questions by using sources. Basically, a research paper is one of the types of papers where scholars analyze questions or topics , look for secondary sources , and write papers on defined themes. For example, if an assignment is to write a research paper on some causes of global warming or any other topic, a person must write a research proposal on it, analyzing important points and credible sources . Although essays focus on personal knowledge, writing a research paper means analyzing sources by following academic standards. Moreover, scientists must meet the structure of research papers. Therefore, writers need to analyze their research paper topics , start to research, cover key aspects, process credible articles, and organize final studies properly.

The Structure of a Research Work

The structure of research papers depends on assignment requirements. In fact, when students get their assignments and instructions, they need to analyze specific research questions or topics, find reliable sources , and write final works. Basically, the structure of research papers consists of the abstract , outline , introduction , literature review , methodology, results , discussion, recommendations, limitations, conclusion , acknowledgments , and references. However, students may not include some of these sections because of assigned instructions that they have and specific types of research papers. For instance, if instructions of papers do not suppose to conduct real experiments, the methodology section can be skipped because of the data’s absence. In turn, the structure of the final work consists of:

Join our satisfied customers who have received perfect papers from Wr1ter Team.

🔸 The First Part of a Research Study

Abstract or an executive summary means the first section of a research paper that provides the study’s purpose, research questions or suggestions, main findings with conclusions. Moreover, this paragraph of about 150 words should be written when the whole work is finished already. Hence, abstract sections should describe key aspects of studies, including discussions about the relevance of findings.

Outline serves as a clear map of the structure of a research study.

Introduction provides the main information on problem statements, the indication of methodology, important findings, and principal conclusion. Basically, this section of a research paper covers rationales behind the work or background research, explanation of the importance, defending its relevance, a brief description of experimental designs, defined research questions, hypotheses, or key aspects.

🔸 Literature Review and Research or Experiment

Literature Review is needed for the analysis of past studies or scholarly articles to be familiar with research questions or topics. Hence, this section summarizes and synthesizes arguments and ideas from scholarly sources without adding new contributions. In turn, this part is organized around arguments or ideas, not sources.

Methodology or Materials and Methods covers explanations of research designs. Basically, techniques for gathering information and other aspects related to experiments must be described in a research paper. For instance, students and scholars document all specialized materials and general procedures. In this case, individuals may use some or all of the methods in further studies or judge the scientific merit of the work. Moreover, scientists should explain how they are going to conduct their experiments.

Results mean the gained information or data after the research or experiment. Basically, scholars should present and illustrate their findings. Moreover, this section may include tables or figures.

🔸 Analysis of Findings

Discussion is a section of a research paper where scientists review the information in the introduction part, evaluate gained results, or compare it with past studies. In particular, students and scholars interpret gained data or findings in appropriate depth. For example, if results differ from expectations at the beginning, scientists should explain why that may have happened. However, if results agree with rationales, scientists should describe theories that the evidence is supported.

Recommendations take its roots from a discussion section where scholars propose potential solutions or new ideas based on obtained results in a research paper. In this case, if scientists have any recommendations on how to improve this research so that other scholars can use evidence in further studies, they must write what they think in this section.

Limitations mean a consideration of research weaknesses and results to get new directions. For instance, if researchers found any limitations of studies that could affect experiments, scholars must not use such knowledge because of the same mistakes. Moreover, scientists should avoid contradicting results, and, even more, they must write it in this section.

🔸 The Final Part of a Conducted Research

Conclusion includes final claims of a research paper based on findings. Basically, this section covers final thoughts and the summary of the whole work. Moreover, this section may be used instead of limitations and recommendations that would be too small by themselves. In this case, scientists do not need to use headings for recommendations and limitations. Also, check out conclusion examples .

Acknowledgments or Appendix may take different forms, from paragraphs to charts. In this section, scholars include additional information on a research paper.

References mean a section where students, scholars, or scientists provide all used sources by following the format and academic rules.

Research Characteristics

Any type of work must meet some standards. By considering a research paper, this work must be written accordingly. In this case, the main characteristics of research papers are the length, style, format, and sources. Firstly, the length of research work defines the number of needed sources to analyze. Then, the style must be formal and covers impersonal and inclusive language. In turn, the format means academic standards of how to organize final works, including its structure and norms. Finally, sources and their number define works as research papers because of the volume of analyzed information. Hence, these characteristics must be considered while writing research papers.

Types of Research Papers

In general, the length of assignments can be different because of instructions. For example, there are two main types of research papers, such as typical and serious works. Firstly, a typical research paper may include definitive, argumentative, interpretive, and other works. In this case, typical papers are from 2 to 10 pages, where students analyze research questions or specific topics. Then, a serious research study is the expanded version of typical works. In turn, the length of such a paper is more than 10 pages. Basically, such works cover a serious analysis with many sources. Therefore, typical and serious works are two types of research papers.

Typical Research Papers

Basically, typical research works depend on assignments, the number of sources, and the paper’s length. So, a typical research paper is usually a long essay with the analyzed evidence. For example, students in high school and colleges get such assignments to learn how to research and analyze topics. In this case, they do not need to conduct serious experiments with the analysis and calculation of data. Moreover, students must use the Internet or libraries in searching for credible secondary sources to find potential answers to specific questions. As a result, students gather information on topics and learn how to take defined sides, present unique positions, or explain new directions. Hence, typical research papers require an analysis of primary and secondary sources without serious experiments or data.

Serious Research Studies

Although long papers require a lot of time for finding and analyzing credible sources, real experiments are an integral part of research work. Firstly, scholars at universities need to analyze the information from past studies to expand or disapprove of researched topics. Then, if scholars want to prove specific positions or ideas, they must get real evidence. In this case, experiments can be surveys, calculations, or other types of data that scholars do personally. Moreover, a dissertation is a typical serious research paper that young scientists write based on the research analysis of topics, data from conducted experiments, and conclusions at the end of work. Thus, serious research papers are studies that take a lot of time, analysis of sources with gained data, and interpretation of results.

What makes a high quality clinical research paper?

Affiliation.

• 1 BMJ, London, UK. [email protected]
• PMID: 20233312
• DOI: 10.1111/j.1601-0825.2010.01663.x

The quality of a research paper depends primarily on the quality of the research study it reports. However, there is also much that authors can do to maximise the clarity and usefulness of their papers. Journals' instructions for authors often focus on the format, style, and length of articles but do not always emphasise the need to clearly explain the work's science and ethics: so this review reminds researchers that transparency is important too. The research question should be stated clearly, along with an explanation of where it came from and why it is important. The study methods must be reported fully and, where appropriate, in line with an evidence based reporting guideline such as the CONSORT statement for randomised controlled trials. If the study was a trial the paper should state where and when the study was registered and state its registration identifier. Finally, any relevant conflicts of interest should be declared.

Publication types

• Clinical Trials as Topic*
• Ethics, Research*
• Guidelines as Topic
• Journalism, Medical / standards*
• Periodicals as Topic
• Publishing / standards*
• Writing / standards*

Choose Your Test

Sat / act prep online guides and tips, 113 great research paper topics.

General Education

One of the hardest parts of writing a research paper can be just finding a good topic to write about. Fortunately we've done the hard work for you and have compiled a list of 113 interesting research paper topics. They've been organized into ten categories and cover a wide range of subjects so you can easily find the best topic for you.

In addition to the list of good research topics, we've included advice on what makes a good research paper topic and how you can use your topic to start writing a great paper.

What Makes a Good Research Paper Topic?

Not all research paper topics are created equal, and you want to make sure you choose a great topic before you start writing. Below are the three most important factors to consider to make sure you choose the best research paper topics.

#1: It's Something You're Interested In

A paper is always easier to write if you're interested in the topic, and you'll be more motivated to do in-depth research and write a paper that really covers the entire subject. Even if a certain research paper topic is getting a lot of buzz right now or other people seem interested in writing about it, don't feel tempted to make it your topic unless you genuinely have some sort of interest in it as well.

#2: There's Enough Information to Write a Paper

Even if you come up with the absolute best research paper topic and you're so excited to write about it, you won't be able to produce a good paper if there isn't enough research about the topic. This can happen for very specific or specialized topics, as well as topics that are too new to have enough research done on them at the moment. Easy research paper topics will always be topics with enough information to write a full-length paper.

Trying to write a research paper on a topic that doesn't have much research on it is incredibly hard, so before you decide on a topic, do a bit of preliminary searching and make sure you'll have all the information you need to write your paper.

#3: It Fits Your Teacher's Guidelines

Don't get so carried away looking at lists of research paper topics that you forget any requirements or restrictions your teacher may have put on research topic ideas. If you're writing a research paper on a health-related topic, deciding to write about the impact of rap on the music scene probably won't be allowed, but there may be some sort of leeway. For example, if you're really interested in current events but your teacher wants you to write a research paper on a history topic, you may be able to choose a topic that fits both categories, like exploring the relationship between the US and North Korea. No matter what, always get your research paper topic approved by your teacher first before you begin writing.

113 Good Research Paper Topics

Below are 113 good research topics to help you get you started on your paper. We've organized them into ten categories to make it easier to find the type of research paper topics you're looking for.

Arts/Culture

• Discuss the main differences in art from the Italian Renaissance and the Northern Renaissance .
• Analyze the impact a famous artist had on the world.
• How is sexism portrayed in different types of media (music, film, video games, etc.)? Has the amount/type of sexism changed over the years?
• How has the music of slaves brought over from Africa shaped modern American music?
• How has rap music evolved in the past decade?
• How has the portrayal of minorities in the media changed?

Current Events

• What have been the impacts of China's one child policy?
• How have the goals of feminists changed over the decades?
• How has the Trump presidency changed international relations?
• Analyze the history of the relationship between the United States and North Korea.
• What factors contributed to the current decline in the rate of unemployment?
• What have been the impacts of states which have increased their minimum wage?
• How do US immigration laws compare to immigration laws of other countries?
• How have the US's immigration laws changed in the past few years/decades?
• How has the Black Lives Matter movement affected discussions and view about racism in the US?
• What impact has the Affordable Care Act had on healthcare in the US?
• What factors contributed to the UK deciding to leave the EU (Brexit)?
• What factors contributed to China becoming an economic power?
• Discuss the history of Bitcoin or other cryptocurrencies  (some of which tokenize the S&P 500 Index on the blockchain) .
• Do students in schools that eliminate grades do better in college and their careers?
• Do students from wealthier backgrounds score higher on standardized tests?
• Do students who receive free meals at school get higher grades compared to when they weren't receiving a free meal?
• Do students who attend charter schools score higher on standardized tests than students in public schools?
• Do students learn better in same-sex classrooms?
• How does giving each student access to an iPad or laptop affect their studies?
• What are the benefits and drawbacks of the Montessori Method ?
• Do children who attend preschool do better in school later on?
• What was the impact of the No Child Left Behind act?
• How does the US education system compare to education systems in other countries?
• What impact does mandatory physical education classes have on students' health?
• Which methods are most effective at reducing bullying in schools?
• Do homeschoolers who attend college do as well as students who attended traditional schools?
• Does offering tenure increase or decrease quality of teaching?
• How does college debt affect future life choices of students?
• Should graduate students be able to form unions?

• What are different ways to lower gun-related deaths in the US?
• How and why have divorce rates changed over time?
• Is affirmative action still necessary in education and/or the workplace?
• Should physician-assisted suicide be legal?
• How has stem cell research impacted the medical field?
• How can human trafficking be reduced in the United States/world?
• Should people be able to donate organs in exchange for money?
• Which types of juvenile punishment have proven most effective at preventing future crimes?
• Has the increase in US airport security made passengers safer?
• Analyze the immigration policies of certain countries and how they are similar and different from one another.
• Several states have legalized recreational marijuana. What positive and negative impacts have they experienced as a result?
• Do tariffs increase the number of domestic jobs?
• Which prison reforms have proven most effective?
• Should governments be able to censor certain information on the internet?
• Which methods/programs have been most effective at reducing teen pregnancy?
• What are the benefits and drawbacks of the Keto diet?
• How effective are different exercise regimes for losing weight and maintaining weight loss?
• How do the healthcare plans of various countries differ from each other?
• What are the most effective ways to treat depression ?
• What are the pros and cons of genetically modified foods?
• Which methods are most effective for improving memory?
• What can be done to lower healthcare costs in the US?
• What factors contributed to the current opioid crisis?
• Analyze the history and impact of the HIV/AIDS epidemic .
• Are low-carbohydrate or low-fat diets more effective for weight loss?
• How much exercise should the average adult be getting each week?
• Which methods are most effective to get parents to vaccinate their children?
• What are the pros and cons of clean needle programs?
• How does stress affect the body?
• Discuss the history of the conflict between Israel and the Palestinians.
• What were the causes and effects of the Salem Witch Trials?
• Who was responsible for the Iran-Contra situation?
• How has New Orleans and the government's response to natural disasters changed since Hurricane Katrina?
• What events led to the fall of the Roman Empire?
• What were the impacts of British rule in India ?
• Was the atomic bombing of Hiroshima and Nagasaki necessary?
• What were the successes and failures of the women's suffrage movement in the United States?
• What were the causes of the Civil War?
• How did Abraham Lincoln's assassination impact the country and reconstruction after the Civil War?
• Which factors contributed to the colonies winning the American Revolution?
• What caused Hitler's rise to power?
• Discuss how a specific invention impacted history.
• What led to Cleopatra's fall as ruler of Egypt?
• How has Japan changed and evolved over the centuries?
• What were the causes of the Rwandan genocide ?

• Why did Martin Luther decide to split with the Catholic Church?
• Analyze the history and impact of a well-known cult (Jonestown, Manson family, etc.)
• How did the sexual abuse scandal impact how people view the Catholic Church?
• How has the Catholic church's power changed over the past decades/centuries?
• What are the causes behind the rise in atheism/ agnosticism in the United States?
• What were the influences in Siddhartha's life resulted in him becoming the Buddha?
• How has media portrayal of Islam/Muslims changed since September 11th?

Science/Environment

• How has the earth's climate changed in the past few decades?
• How has the use and elimination of DDT affected bird populations in the US?
• Analyze how the number and severity of natural disasters have increased in the past few decades.
• Analyze deforestation rates in a certain area or globally over a period of time.
• How have past oil spills changed regulations and cleanup methods?
• How has the Flint water crisis changed water regulation safety?
• What are the pros and cons of fracking?
• What impact has the Paris Climate Agreement had so far?
• What have NASA's biggest successes and failures been?
• How can we improve access to clean water around the world?
• Does ecotourism actually have a positive impact on the environment?
• Should the US rely on nuclear energy more?
• What can be done to save amphibian species currently at risk of extinction?
• What impact has climate change had on coral reefs?
• How are black holes created?
• Are teens who spend more time on social media more likely to suffer anxiety and/or depression?
• How will the loss of net neutrality affect internet users?
• Analyze the history and progress of self-driving vehicles.
• How has the use of drones changed surveillance and warfare methods?
• Has social media made people more or less connected?
• What progress has currently been made with artificial intelligence ?
• Do smartphones increase or decrease workplace productivity?
• What are the most effective ways to use technology in the classroom?
• How is Google search affecting our intelligence?
• When is the best age for a child to begin owning a smartphone?
• Has frequent texting reduced teen literacy rates?

How to Write a Great Research Paper

Even great research paper topics won't give you a great research paper if you don't hone your topic before and during the writing process. Follow these three tips to turn good research paper topics into great papers.

#1: Figure Out Your Thesis Early

Before you start writing a single word of your paper, you first need to know what your thesis will be. Your thesis is a statement that explains what you intend to prove/show in your paper. Every sentence in your research paper will relate back to your thesis, so you don't want to start writing without it!

As some examples, if you're writing a research paper on if students learn better in same-sex classrooms, your thesis might be "Research has shown that elementary-age students in same-sex classrooms score higher on standardized tests and report feeling more comfortable in the classroom."

If you're writing a paper on the causes of the Civil War, your thesis might be "While the dispute between the North and South over slavery is the most well-known cause of the Civil War, other key causes include differences in the economies of the North and South, states' rights, and territorial expansion."

#2: Back Every Statement Up With Research

Remember, this is a research paper you're writing, so you'll need to use lots of research to make your points. Every statement you give must be backed up with research, properly cited the way your teacher requested. You're allowed to include opinions of your own, but they must also be supported by the research you give.

#3: Do Your Research Before You Begin Writing

You don't want to start writing your research paper and then learn that there isn't enough research to back up the points you're making, or, even worse, that the research contradicts the points you're trying to make!

Get most of your research on your good research topics done before you begin writing. Then use the research you've collected to create a rough outline of what your paper will cover and the key points you're going to make. This will help keep your paper clear and organized, and it'll ensure you have enough research to produce a strong paper.

What's Next?

Are you also learning about dynamic equilibrium in your science class? We break this sometimes tricky concept down so it's easy to understand in our complete guide to dynamic equilibrium .

Thinking about becoming a nurse practitioner? Nurse practitioners have one of the fastest growing careers in the country, and we have all the information you need to know about what to expect from nurse practitioner school .

Want to know the fastest and easiest ways to convert between Fahrenheit and Celsius? We've got you covered! Check out our guide to the best ways to convert Celsius to Fahrenheit (or vice versa).

Need more help with this topic? Check out Tutorbase!

Our vetted tutor database includes a range of experienced educators who can help you polish an essay for English or explain how derivatives work for Calculus. You can use dozens of filters and search criteria to find the perfect person for your needs.

These recommendations are based solely on our knowledge and experience. If you purchase an item through one of our links, PrepScholar may receive a commission.

Christine graduated from Michigan State University with degrees in Environmental Biology and Geography and received her Master's from Duke University. In high school she scored in the 99th percentile on the SAT and was named a National Merit Finalist. She has taught English and biology in several countries.

Student and Parent Forum

Our new student and parent forum, at ExpertHub.PrepScholar.com , allow you to interact with your peers and the PrepScholar staff. See how other students and parents are navigating high school, college, and the college admissions process. Ask questions; get answers.

Ask a Question Below

Have any questions about this article or other topics? Ask below and we'll reply!

Improve With Our Famous Guides

• For All Students

The 5 Strategies You Must Be Using to Improve 160+ SAT Points

How to Get a Perfect 1600, by a Perfect Scorer

Series: How to Get 800 on Each SAT Section:

Score 800 on SAT Math

Score 800 on SAT Reading

Score 800 on SAT Writing

Series: How to Get to 600 on Each SAT Section:

Score 600 on SAT Math

Score 600 on SAT Reading

Score 600 on SAT Writing

Free Complete Official SAT Practice Tests

What SAT Target Score Should You Be Aiming For?

15 Strategies to Improve Your SAT Essay

The 5 Strategies You Must Be Using to Improve 4+ ACT Points

How to Get a Perfect 36 ACT, by a Perfect Scorer

Series: How to Get 36 on Each ACT Section:

36 on ACT English

36 on ACT Math

36 on ACT Reading

36 on ACT Science

Series: How to Get to 24 on Each ACT Section:

24 on ACT English

24 on ACT Math

24 on ACT Reading

24 on ACT Science

What ACT target score should you be aiming for?

ACT Vocabulary You Must Know

ACT Writing: 15 Tips to Raise Your Essay Score

How to Get Into Harvard and the Ivy League

How to Get a Perfect 4.0 GPA

How to Write an Amazing College Essay

What Exactly Are Colleges Looking For?

Is the ACT easier than the SAT? A Comprehensive Guide

Should you retake your SAT or ACT?

When should you take the SAT or ACT?

Stay Informed

Get the latest articles and test prep tips!

Looking for Graduate School Test Prep?

Check out our top-rated graduate blogs here:

GRE Online Prep Blog

GMAT Online Prep Blog

TOEFL Online Prep Blog

Holly R. "I am absolutely overjoyed and cannot thank you enough for helping me!”

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Academic writing

Forging good titles in academic writing

Published on March 20, 2015 by Shane Bryson . Revised on July 23, 2023.

The title is the first thing your reader will see, and most readers will make their first judgements of your work based on it. For this reason, it’s important to think about your titles carefully.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

Informative, striking, appropriate, title templates, writing effective headings, other interesting articles, informative title.

Your title should, above all else, convey the topic of your paper. In other words, no matter how witty, clever, original, or otherwise appealing your title may be, it fails if it is not informative.

Decide whether you’ve given a sense of the paper’s topic and claims by comparing your title’s content to the most important aspect(s) of your dissertation statement or hypothesis and conclusions.

Striking title

A striking title is one that entices your audience to read, so know your audience’s tastes.

The analogy of cultivating sexual attraction in a prospective mate is useful here: some audiences will be enticed by a title’s edginess (as with, for example, V. Alneng’s “‘What the Fuck is a Vietnam?’ Touristic Phantasms and the Popcolonization of [the] Vietnam [War],” published in Critique on Anthropology ); others will almost always prefer a more straightforward title (as with J.C. Henderson’s “War as a tourist attraction: The case of Vietnam,” published in the International Journal of Tourism Research ).

You should be able to gauge how edgy your title can be by the tone of your discipline or the publication you’re submitting to, and your main concern should be forming a title that appeals to your readers’ specific tastes.

Consider also that a title that highlights the paper’s fresh insights will often be striking.

An endocrinologist, for example, might become very excited upon seeing the collaboratively authored article “Comparison of the effects on glycaemic control and β-cell function in newly diagnosed type 2 diabetes patients of treatment with exenatide, insulin or pioglitazone: A multicentre randomized parallel-group trial,” published in 2015 in the Journal of Internal Medicine .

This rather long title is more acceptable in the sciences, where what readers tend to find provocative in a title is the degree to which it reveals the paper’s specifics.

Appropriate title

Ensuring that your title is appropriate in a way of making sure not only that your audience understands it, but also that its appeal contributes to its meaning. To make sure the title will be understood, you need to consider how familiar your research topic will be to your audience.

In an academic essay, you can use highly technical terms in your title, but generally avoid terms that the average well-read person in your discipline might not know.

In any writing that has a broad audience, titles need to avoid language that is too sophisticated; a news article, for example, should be easily understood by all.

As a second consideration of appropriateness, make sure that your title does not entice without substance.

The title of Alneng’s paper, for example, does not use “fuck” merely to shock and therefore entice the reader; the uncommon use of a swearword here helps convey the topic of the article: more or less vulgar representations of Vietnam.

The same is true for other striking titles, such as Nancy Tuana’s “Coming to Understand: Orgasm and the Epistemology of Ignorance,” published in Hypatia .

The title’s sexually charged play on words (“coming to understand”) hooks the audience, but is not merely a hook. The pun is directly relevant to the essay’s argument, which is that sexual pleasure offers an important form of knowledge.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

• Use key terms. Find words that your audience can easily identify as markers of the topic matter. These will include, for example, terms that convey the field of research, central concepts, or subjects of study.
• Identify the context (sometimes called “the location”). By context, I mean the source or the setting of the discussion, depending on discipline. In a history paper this might be a certain century or era; in literary studies a certain book or author; and in the sciences an organism or compound.

The following is a list of title formats, with examples of each. I’ve given the names of the publications in brackets to give a sense of how different disciplines treat titles.

Note that these are not mutually exclusive patterns (i.e. it’s possible to have various combinations; e.g. General & interesting: Informative & specific). Note also that this is not meant to be an exhaustive list.

• Striking: Informative – The Specter of Wall Street: “Bartleby, the Scrivener” and the Language of Commodities ( American Literature )
• Informative: Striking – Carbon capture and storage: How green can black be? ( Science )
• General: Specific – The issues of the sixties: An exploratory study in the dynamics of public opinion ( Public Opinion Quarterly )
• “Quotation”: Discussion (social studies) – “I’d rather not talk about it”: Adolescents’ and young adults’ use of topic avoidance in stepfamilies ( Journal of Applied Communication Research )
• “Quotation”: Discussion (literary studies) – “I Would Prefer Not To”: Giorgio Agamben, Bartleby and the Potentiality of the Law ( Law and Critique )
• Simple and precise – Methodological issues in the use of Tsimshian oral Traditions (Adawx) in Archaeology ( Canadian Journal of Archaeology )
• Topic: Method – Mortality in sleep apnea patients: A multivariate analysis of risk factors ( Sleep )
• Topic: Significance – LC3 binds externalized cardiolipin on injured mitochondria to signal mitophagy in neurons: Implications for Parkinson disease ( Autophagy )
• Technical and very specific – Single-shot quantum nondemolition measurement of a quantum-dot electron spin using cavity exciton-polaritons ( Physical Review )

Although similar, headings are not the same as titles. Headings head paragraphs and help structure a document. Effective headings make your paper easily scannable.

Common high level headings in dissertations and research papers are “Methods”, “Research results”, and “Discussion”. Lower level headings are often more descriptive.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

If you want to know more about AI for academic writing, AI tools, or fallacies make sure to check out some of our other articles with explanations and examples or go directly to our tools!

• Begging the question fallacy
• Hasty generalization fallacy
• Equivocation fallacy
• False cause fallacy
• Sunk cost fallacy
• Deep learning
• Generative AI
• Machine learning
• Reinforcement learning
• Supervised vs. unsupervised learning

 (AI) Tools

• Grammar Checker
• Paraphrasing Tool
• Text Summarizer
• AI Detector
• Plagiarism Checker
• Citation Generator

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bryson, S. (2023, July 23). Forging good titles in academic writing. Scribbr. Retrieved March 19, 2024, from https://www.scribbr.com/academic-writing/forging-good-titles-in-academic-writing/

Is this article helpful?

Shane Bryson

Shane finished his master's degree in English literature in 2013 and has been working as a writing tutor and editor since 2009. He began proofreading and editing essays with Scribbr in early summer, 2014.

Unlimited Academic AI-Proofreading

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Saudi J Anaesth
• v.13(Suppl 1); 2019 Apr

Writing the title and abstract for a research paper: Being concise, precise, and meticulous is the key

Milind s. tullu.

Department of Pediatrics, Seth G.S. Medical College and KEM Hospital, Parel, Mumbai, Maharashtra, India

This article deals with formulating a suitable title and an appropriate abstract for an original research paper. The “title” and the “abstract” are the “initial impressions” of a research article, and hence they need to be drafted correctly, accurately, carefully, and meticulously. Often both of these are drafted after the full manuscript is ready. Most readers read only the title and the abstract of a research paper and very few will go on to read the full paper. The title and the abstract are the most important parts of a research paper and should be pleasant to read. The “title” should be descriptive, direct, accurate, appropriate, interesting, concise, precise, unique, and should not be misleading. The “abstract” needs to be simple, specific, clear, unbiased, honest, concise, precise, stand-alone, complete, scholarly, (preferably) structured, and should not be misrepresentative. The abstract should be consistent with the main text of the paper, especially after a revision is made to the paper and should include the key message prominently. It is very important to include the most important words and terms (the “keywords”) in the title and the abstract for appropriate indexing purpose and for retrieval from the search engines and scientific databases. Such keywords should be listed after the abstract. One must adhere to the instructions laid down by the target journal with regard to the style and number of words permitted for the title and the abstract.

Introduction

This article deals with drafting a suitable “title” and an appropriate “abstract” for an original research paper. Because the “title” and the “abstract” are the “initial impressions” or the “face” of a research article, they need to be drafted correctly, accurately, carefully, meticulously, and consume time and energy.[ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] Often, these are drafted after the complete manuscript draft is ready.[ 2 , 3 , 4 , 5 , 9 , 10 , 11 ] Most readers will read only the title and the abstract of a published research paper, and very few “interested ones” (especially, if the paper is of use to them) will go on to read the full paper.[ 1 , 2 ] One must remember to adhere to the instructions laid down by the “target journal” (the journal for which the author is writing) regarding the style and number of words permitted for the title and the abstract.[ 2 , 4 , 5 , 7 , 8 , 9 , 12 ] Both the title and the abstract are the most important parts of a research paper – for editors (to decide whether to process the paper for further review), for reviewers (to get an initial impression of the paper), and for the readers (as these may be the only parts of the paper available freely and hence, read widely).[ 4 , 8 , 12 ] It may be worth for the novice author to browse through titles and abstracts of several prominent journals (and their target journal as well) to learn more about the wording and styles of the titles and abstracts, as well as the aims and scope of the particular journal.[ 5 , 7 , 9 , 13 ]

The details of the title are discussed under the subheadings of importance, types, drafting, and checklist.

Importance of the title

When a reader browses through the table of contents of a journal issue (hard copy or on website), the title is the “ first detail” or “face” of the paper that is read.[ 2 , 3 , 4 , 5 , 6 , 13 ] Hence, it needs to be simple, direct, accurate, appropriate, specific, functional, interesting, attractive/appealing, concise/brief, precise/focused, unambiguous, memorable, captivating, informative (enough to encourage the reader to read further), unique, catchy, and it should not be misleading.[ 1 , 2 , 3 , 4 , 5 , 6 , 9 , 12 ] It should have “just enough details” to arouse the interest and curiosity of the reader so that the reader then goes ahead with studying the abstract and then (if still interested) the full paper.[ 1 , 2 , 4 , 13 ] Journal websites, electronic databases, and search engines use the words in the title and abstract (the “keywords”) to retrieve a particular paper during a search; hence, the importance of these words in accessing the paper by the readers has been emphasized.[ 3 , 4 , 5 , 6 , 12 , 14 ] Such important words (or keywords) should be arranged in appropriate order of importance as per the context of the paper and should be placed at the beginning of the title (rather than the later part of the title, as some search engines like Google may just display only the first six to seven words of the title).[ 3 , 5 , 12 ] Whimsical, amusing, or clever titles, though initially appealing, may be missed or misread by the busy reader and very short titles may miss the essential scientific words (the “keywords”) used by the indexing agencies to catch and categorize the paper.[ 1 , 3 , 4 , 9 ] Also, amusing or hilarious titles may be taken less seriously by the readers and may be cited less often.[ 4 , 15 ] An excessively long or complicated title may put off the readers.[ 3 , 9 ] It may be a good idea to draft the title after the main body of the text and the abstract are drafted.[ 2 , 3 , 4 , 5 ]

Types of titles

Titles can be descriptive, declarative, or interrogative. They can also be classified as nominal, compound, or full-sentence titles.

Descriptive or neutral title

This has the essential elements of the research theme, that is, the patients/subjects, design, interventions, comparisons/control, and outcome, but does not reveal the main result or the conclusion.[ 3 , 4 , 12 , 16 ] Such a title allows the reader to interpret the findings of the research paper in an impartial manner and with an open mind.[ 3 ] These titles also give complete information about the contents of the article, have several keywords (thus increasing the visibility of the article in search engines), and have increased chances of being read and (then) being cited as well.[ 4 ] Hence, such descriptive titles giving a glimpse of the paper are generally preferred.[ 4 , 16 ]

Declarative title

This title states the main finding of the study in the title itself; it reduces the curiosity of the reader, may point toward a bias on the part of the author, and hence is best avoided.[ 3 , 4 , 12 , 16 ]

Interrogative title

This is the one which has a query or the research question in the title.[ 3 , 4 , 16 ] Though a query in the title has the ability to sensationalize the topic, and has more downloads (but less citations), it can be distracting to the reader and is again best avoided for a research article (but can, at times, be used for a review article).[ 3 , 6 , 16 , 17 ]

From a sentence construct point of view, titles may be nominal (capturing only the main theme of the study), compound (with subtitles to provide additional relevant information such as context, design, location/country, temporal aspect, sample size, importance, and a provocative or a literary; for example, see the title of this review), or full-sentence titles (which are longer and indicate an added degree of certainty of the results).[ 4 , 6 , 9 , 16 ] Any of these constructs may be used depending on the type of article, the key message, and the author's preference or judgement.[ 4 ]

Drafting a suitable title

A stepwise process can be followed to draft the appropriate title. The author should describe the paper in about three sentences, avoiding the results and ensuring that these sentences contain important scientific words/keywords that describe the main contents and subject of the paper.[ 1 , 4 , 6 , 12 ] Then the author should join the sentences to form a single sentence, shorten the length (by removing redundant words or adjectives or phrases), and finally edit the title (thus drafted) to make it more accurate, concise (about 10–15 words), and precise.[ 1 , 3 , 4 , 5 , 9 ] Some journals require that the study design be included in the title, and this may be placed (using a colon) after the primary title.[ 2 , 3 , 4 , 14 ] The title should try to incorporate the Patients, Interventions, Comparisons and Outcome (PICO).[ 3 ] The place of the study may be included in the title (if absolutely necessary), that is, if the patient characteristics (such as study population, socioeconomic conditions, or cultural practices) are expected to vary as per the country (or the place of the study) and have a bearing on the possible outcomes.[ 3 , 6 ] Lengthy titles can be boring and appear unfocused, whereas very short titles may not be representative of the contents of the article; hence, optimum length is required to ensure that the title explains the main theme and content of the manuscript.[ 4 , 5 , 9 ] Abbreviations (except the standard or commonly interpreted ones such as HIV, AIDS, DNA, RNA, CDC, FDA, ECG, and EEG) or acronyms should be avoided in the title, as a reader not familiar with them may skip such an article and nonstandard abbreviations may create problems in indexing the article.[ 3 , 4 , 5 , 6 , 9 , 12 ] Also, too much of technical jargon or chemical formulas in the title may confuse the readers and the article may be skipped by them.[ 4 , 9 ] Numerical values of various parameters (stating study period or sample size) should also be avoided in the titles (unless deemed extremely essential).[ 4 ] It may be worthwhile to take an opinion from a impartial colleague before finalizing the title.[ 4 , 5 , 6 ] Thus, multiple factors (which are, at times, a bit conflicting or contrasting) need to be considered while formulating a title, and hence this should not be done in a hurry.[ 4 , 6 ] Many journals ask the authors to draft a “short title” or “running head” or “running title” for printing in the header or footer of the printed paper.[ 3 , 12 ] This is an abridged version of the main title of up to 40–50 characters, may have standard abbreviations, and helps the reader to navigate through the paper.[ 3 , 12 , 14 ]

Checklist for a good title

Table 1 gives a checklist/useful tips for drafting a good title for a research paper.[ 1 , 2 , 3 , 4 , 5 , 6 , 12 ] Table 2 presents some of the titles used by the author of this article in his earlier research papers, and the appropriateness of the titles has been commented upon. As an individual exercise, the reader may try to improvise upon the titles (further) after reading the corresponding abstract and full paper.

Checklist/useful tips for drafting a good title for a research paper

Some titles used by author of this article in his earlier publications and remark/comment on their appropriateness

The Abstract

The details of the abstract are discussed under the subheadings of importance, types, drafting, and checklist.

Importance of the abstract

The abstract is a summary or synopsis of the full research paper and also needs to have similar characteristics like the title. It needs to be simple, direct, specific, functional, clear, unbiased, honest, concise, precise, self-sufficient, complete, comprehensive, scholarly, balanced, and should not be misleading.[ 1 , 2 , 3 , 7 , 8 , 9 , 10 , 11 , 13 , 17 ] Writing an abstract is to extract and summarize (AB – absolutely, STR – straightforward, ACT – actual data presentation and interpretation).[ 17 ] The title and abstracts are the only sections of the research paper that are often freely available to the readers on the journal websites, search engines, and in many abstracting agencies/databases, whereas the full paper may attract a payment per view or a fee for downloading the pdf copy.[ 1 , 2 , 3 , 7 , 8 , 10 , 11 , 13 , 14 ] The abstract is an independent and stand-alone (that is, well understood without reading the full paper) section of the manuscript and is used by the editor to decide the fate of the article and to choose appropriate reviewers.[ 2 , 7 , 10 , 12 , 13 ] Even the reviewers are initially supplied only with the title and the abstract before they agree to review the full manuscript.[ 7 , 13 ] This is the second most commonly read part of the manuscript, and therefore it should reflect the contents of the main text of the paper accurately and thus act as a “real trailer” of the full article.[ 2 , 7 , 11 ] The readers will go through the full paper only if they find the abstract interesting and relevant to their practice; else they may skip the paper if the abstract is unimpressive.[ 7 , 8 , 9 , 10 , 13 ] The abstract needs to highlight the selling point of the manuscript and succeed in luring the reader to read the complete paper.[ 3 , 7 ] The title and the abstract should be constructed using keywords (key terms/important words) from all the sections of the main text.[ 12 ] Abstracts are also used for submitting research papers to a conference for consideration for presentation (as oral paper or poster).[ 9 , 13 , 17 ] Grammatical and typographic errors reflect poorly on the quality of the abstract, may indicate carelessness/casual attitude on part of the author, and hence should be avoided at all times.[ 9 ]

Types of abstracts

The abstracts can be structured or unstructured. They can also be classified as descriptive or informative abstracts.

Structured and unstructured abstracts

Structured abstracts are followed by most journals, are more informative, and include specific subheadings/subsections under which the abstract needs to be composed.[ 1 , 7 , 8 , 9 , 10 , 11 , 13 , 17 , 18 ] These subheadings usually include context/background, objectives, design, setting, participants, interventions, main outcome measures, results, and conclusions.[ 1 ] Some journals stick to the standard IMRAD format for the structure of the abstracts, and the subheadings would include Introduction/Background, Methods, Results, And (instead of Discussion) the Conclusion/s.[ 1 , 2 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 17 , 18 ] Structured abstracts are more elaborate, informative, easy to read, recall, and peer-review, and hence are preferred; however, they consume more space and can have same limitations as an unstructured abstract.[ 7 , 9 , 18 ] The structured abstracts are (possibly) better understood by the reviewers and readers. Anyway, the choice of the type of the abstract and the subheadings of a structured abstract depend on the particular journal style and is not left to the author's wish.[ 7 , 10 , 12 ] Separate subheadings may be necessary for reporting meta-analysis, educational research, quality improvement work, review, or case study.[ 1 ] Clinical trial abstracts need to include the essential items mentioned in the CONSORT (Consolidated Standards Of Reporting Trials) guidelines.[ 7 , 9 , 14 , 19 ] Similar guidelines exist for various other types of studies, including observational studies and for studies of diagnostic accuracy.[ 20 , 21 ] A useful resource for the above guidelines is available at www.equator-network.org (Enhancing the QUAlity and Transparency Of health Research). Unstructured (or non-structured) abstracts are free-flowing, do not have predefined subheadings, and are commonly used for papers that (usually) do not describe original research.[ 1 , 7 , 9 , 10 ]

The four-point structured abstract: This has the following elements which need to be properly balanced with regard to the content/matter under each subheading:[ 9 ]

Background and/or Objectives: This states why the work was undertaken and is usually written in just a couple of sentences.[ 3 , 7 , 8 , 9 , 10 , 12 , 13 ] The hypothesis/study question and the major objectives are also stated under this subheading.[ 3 , 7 , 8 , 9 , 10 , 12 , 13 ]

Methods: This subsection is the longest, states what was done, and gives essential details of the study design, setting, participants, blinding, sample size, sampling method, intervention/s, duration and follow-up, research instruments, main outcome measures, parameters evaluated, and how the outcomes were assessed or analyzed.[ 3 , 7 , 8 , 9 , 10 , 12 , 13 , 14 , 17 ]

Results/Observations/Findings: This subheading states what was found, is longer, is difficult to draft, and needs to mention important details including the number of study participants, results of analysis (of primary and secondary objectives), and include actual data (numbers, mean, median, standard deviation, “P” values, 95% confidence intervals, effect sizes, relative risks, odds ratio, etc.).[ 3 , 7 , 8 , 9 , 10 , 12 , 13 , 14 , 17 ]

Conclusions: The take-home message (the “so what” of the paper) and other significant/important findings should be stated here, considering the interpretation of the research question/hypothesis and results put together (without overinterpreting the findings) and may also include the author's views on the implications of the study.[ 3 , 7 , 8 , 9 , 10 , 12 , 13 , 14 , 17 ]

The eight-point structured abstract: This has the following eight subheadings – Objectives, Study Design, Study Setting, Participants/Patients, Methods/Intervention, Outcome Measures, Results, and Conclusions.[ 3 , 9 , 18 ] The instructions to authors given by the particular journal state whether they use the four- or eight-point abstract or variants thereof.[ 3 , 14 ]

Descriptive and Informative abstracts

Descriptive abstracts are short (75–150 words), only portray what the paper contains without providing any more details; the reader has to read the full paper to know about its contents and are rarely used for original research papers.[ 7 , 10 ] These are used for case reports, reviews, opinions, and so on.[ 7 , 10 ] Informative abstracts (which may be structured or unstructured as described above) give a complete detailed summary of the article contents and truly reflect the actual research done.[ 7 , 10 ]

Drafting a suitable abstract

It is important to religiously stick to the instructions to authors (format, word limit, font size/style, and subheadings) provided by the journal for which the abstract and the paper are being written.[ 7 , 8 , 9 , 10 , 13 ] Most journals allow 200–300 words for formulating the abstract and it is wise to restrict oneself to this word limit.[ 1 , 2 , 3 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 22 ] Though some authors prefer to draft the abstract initially, followed by the main text of the paper, it is recommended to draft the abstract in the end to maintain accuracy and conformity with the main text of the paper (thus maintaining an easy linkage/alignment with title, on one hand, and the introduction section of the main text, on the other hand).[ 2 , 7 , 9 , 10 , 11 ] The authors should check the subheadings (of the structured abstract) permitted by the target journal, use phrases rather than sentences to draft the content of the abstract, and avoid passive voice.[ 1 , 7 , 9 , 12 ] Next, the authors need to get rid of redundant words and edit the abstract (extensively) to the correct word count permitted (every word in the abstract “counts”!).[ 7 , 8 , 9 , 10 , 13 ] It is important to ensure that the key message, focus, and novelty of the paper are not compromised; the rationale of the study and the basis of the conclusions are clear; and that the abstract is consistent with the main text of the paper.[ 1 , 2 , 3 , 7 , 9 , 11 , 12 , 13 , 14 , 17 , 22 ] This is especially important while submitting a revision of the paper (modified after addressing the reviewer's comments), as the changes made in the main (revised) text of the paper need to be reflected in the (revised) abstract as well.[ 2 , 10 , 12 , 14 , 22 ] Abbreviations should be avoided in an abstract, unless they are conventionally accepted or standard; references, tables, or figures should not be cited in the abstract.[ 7 , 9 , 10 , 11 , 13 ] It may be worthwhile not to rush with the abstract and to get an opinion by an impartial colleague on the content of the abstract; and if possible, the full paper (an “informal” peer-review).[ 1 , 7 , 8 , 9 , 11 , 17 ] Appropriate “Keywords” (three to ten words or phrases) should follow the abstract and should be preferably chosen from the Medical Subject Headings (MeSH) list of the U.S. National Library of Medicine ( https://meshb.nlm.nih.gov/search ) and are used for indexing purposes.[ 2 , 3 , 11 , 12 ] These keywords need to be different from the words in the main title (the title words are automatically used for indexing the article) and can be variants of the terms/phrases used in the title, or words from the abstract and the main text.[ 3 , 12 ] The ICMJE (International Committee of Medical Journal Editors; http://www.icmje.org/ ) also recommends publishing the clinical trial registration number at the end of the abstract.[ 7 , 14 ]

Checklist for a good abstract

Table 3 gives a checklist/useful tips for formulating a good abstract for a research paper.[ 1 , 2 , 3 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 17 , 22 ]

Checklist/useful tips for formulating a good abstract for a research paper

Concluding Remarks

This review article has given a detailed account of the importance and types of titles and abstracts. It has also attempted to give useful hints for drafting an appropriate title and a complete abstract for a research paper. It is hoped that this review will help the authors in their career in medical writing.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Acknowledgement

The author thanks Dr. Hemant Deshmukh - Dean, Seth G.S. Medical College & KEM Hospital, for granting permission to publish this manuscript.

• WordPress.org
• Documentation
• Learn WordPress

Product Development Use Cases

Risks with product development, what product development engineers do, what are product development strategies, is product development part of r&d, why product development is necessary, is product development agile, product development key skills, stages to product development.

What Are the 5 Characteristics of a Good Research Paper?

What are the 5 characteristics of a good research paper? Tune in to find out.

Top 5 Characteristics of a Good Research Paper

Good research is anchored on a sound research question.

A good research paper is one that gets you the best grade; it is one that makes a statement about the real world around you. 

In essence, what makes it a good research paper is the way in which it answers the question that you wanted to answer. 

The question is usually posed as a hypothesis. So, it needs to be answered in a way that shows proof of your observation about the world around you. A good research paper also needs to be interesting to read and support your point of view.

Good research follows a systematic, appropriate research methodology

A good research methodology is one that is  systematic and scientific . First, it is one that follows a logical and coherent pattern of development. 

It starts with an idea then moves on to design, data collection, analysis, and finally interpretation and presentation of results.

It also needs to be credible. Also, it does not make use of weak sources of information; it does not depend on hearsay for the core data. 

In essence, a good research paper uses credible sources that have been vetted for accuracy and authenticity. They are also relevant to your topic and will support your argument.

Good research makes use of appropriate methods of data collection

The whole point of a research paper is to collect information from credible sources and analyze them in order to draw a conclusion. 

This conclusion then becomes the thesis statement that you need to write about in the paper itself. 

Good research papers use appropriate methods for collecting data. Whether it was through surveys, questionnaires, or interviews, or through analyzing existing data in the form of statistics, graphs, or charts. 

Good research is guided by logic

A good research paper is one that you can logically support. The data that you collect needs to be analyzed and interpreted in a logical way. 

The conclusion should be the one that logically follows from the research questions, the methodology, and the data. 

Good research acknowledges its limitations and provides suggestions for future research

Finally, a good research paper is one that acknowledges any shortcomings or limitations in the research. It is one that recognizes the possibility of alternative conclusions and alternative interpretations of the data. 

It is also one that makes suggestions for further research in the same area.

The Bottom Line

A good research paper follows a systematic, logical, and coherent pattern of development. It uses credible sources that have been verified for accuracy and authenticity. 

It is guided by logic, it is systematic and it acknowledges its limitations.

What do you think are the top qualities of a good academic research paper? As always, we want to hear from you. 

If you have any comments, questions, or concerns about this article, please leave them in the comments section below

https://www.youtube.com/watch?v=m3ijYqyWnog.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

More Stories

Is product development agile? If not, how can you make it agile? What are the steps you need to do? To find...

Product Development Factors

What are some of the most important product development factors that you need to know? How can these help you...

Why New Product Development Is Important

There are many reasons why new product development is important. These are the reasons why you need to do this process meticulously....

Product Development Qualifications

What are some of the product development qualifications that you need to have? How do you ensure you have these...

Product Development Lead

A product development lead is a person that has many responsibilities within the company he is working for. What are...

Product Development vs Market Development

Product development vs market development. What are the differences and similarities between these two? What is their purpose? To know...

Recent Post

Top category, vp research and development.

This paper is in the following e-collection/theme issue:

Published on 18.3.2024 in Vol 26 (2024)

Development and Validation of an Interpretable Conformal Predictor to Predict Sepsis Mortality Risk: Retrospective Cohort Study

Authors of this article:

Original Paper

• Meicheng Yang 1 , PhD   ; 
• Hui Chen 2 , MD   ; 
• Wenhan Hu 2 , PhD   ; 
• Massimo Mischi 3 , PhD   ; 
• Caifeng Shan 4, 5 , PhD   ; 
• Jianqing Li 1, 6 * , PhD   ; 
• Xi Long 3 * , PhD   ; 
• Chengyu Liu 1 * , PhD  

1 State Key Laboratory of Digital Medical Engineering, School of Instrument Science and Engineering, Southeast University, Nanjing, China

2 Department of Critical Care Medicine, Jiangsu Provincial Key Laboratory of Critical Care Medicine, Zhongda Hospital, Southeast University, Nanjing, China

3 Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands

4 College of Electrical Engineering and Automation, Shandong University of Science and Technology, Qingdao, China

5 School of Intelligence Science and Technology, Nanjing University, Nanjing, China

6 School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China

*these authors contributed equally

Corresponding Author:

Chengyu Liu, PhD

State Key Laboratory of Digital Medical Engineering, School of Instrument Science and Engineering

Southeast University

35 Jinxianghe Road

Nanjing, 210096

Phone: 86 25 83793993

Email: [email protected]

Background: Early and reliable identification of patients with sepsis who are at high risk of mortality is important to improve clinical outcomes. However, 3 major barriers to artificial intelligence (AI) models, including the lack of interpretability, the difficulty in generalizability, and the risk of automation bias, hinder the widespread adoption of AI models for use in clinical practice.

Objective: This study aimed to develop and validate (internally and externally) a conformal predictor of sepsis mortality risk in patients who are critically ill, leveraging AI-assisted prediction modeling. The proposed approach enables explaining the model output and assessing its confidence level.

Methods: We retrospectively extracted data on adult patients with sepsis from a database collected in a teaching hospital at Beth Israel Deaconess Medical Center for model training and internal validation. A large multicenter critical care database from the Philips eICU Research Institute was used for external validation. A total of 103 clinical features were extracted from the first day after admission. We developed an AI model using gradient-boosting machines to predict the mortality risk of sepsis and used Mondrian conformal prediction to estimate the prediction uncertainty. The Shapley additive explanation method was used to explain the model.

Results: A total of 16,746 (80%) patients from Beth Israel Deaconess Medical Center were used to train the model. When tested on the internal validation population of 4187 (20%) patients, the model achieved an area under the receiver operating characteristic curve of 0.858 (95% CI 0.845-0.871), which was reduced to 0.800 (95% CI 0.789-0.811) when externally validated on 10,362 patients from the Philips eICU database. At a specified confidence level of 90% for the internal validation cohort the percentage of error predictions (n=438) out of all predictions (n=4187) was 10.5%, with 1229 (29.4%) predictions requiring clinician review. In contrast, the AI model without conformal prediction made 1449 (34.6%) errors. When externally validated, more predictions (n=4004, 38.6%) were flagged for clinician review due to interdatabase heterogeneity. Nevertheless, the model still produced significantly lower error rates compared to the point predictions by AI (n=1221, 11.8% vs n=4540, 43.8%). The most important predictors identified in this predictive model were Acute Physiology Score III, age, urine output, vasopressors, and pulmonary infection. Clinically relevant risk factors contributing to a single patient were also examined to show how the risk arose.

Conclusions: By combining model explanation and conformal prediction, AI-based systems can be better translated into medical practice for clinical decision-making.

Introduction

Sepsis is a life-threatening systemic illness resulting from a dysregulated host response to microbial invasion (infection) and is associated with high morbidity and mortality [ 1 , 2 ]. According to the most recent Global Burden of Diseases study, nearly 49 million people experience sepsis each year and approximately 11 million die from sepsis and its complications, accounting for 19.7% of all deaths worldwide [ 3 ]. In addition, a Chinese epidemiological study in 2020 showed that the incidence of sepsis in intensive care units (ICUs) was 20.6%, with a mortality rate of 35.5% [ 4 ]. Recent evidence suggests that early identification of patients who are critically ill with the potential for acute deterioration is effective in improving clinical outcomes [ 5 ]. Therefore, a pragmatic model that could help identify high-risk patients and further improve the prognosis of patients with sepsis is critical.

Recent significant increases in electronic health record data and advancements in artificial intelligence (AI) have led to rapid growth in the development of machine learning algorithms to identify patients with sepsis at high risk of in-hospital mortality [ 6 - 8 ]. However, there are 3 major barriers to the widespread adoption of AI models for their deployment in clinical practice: first, the lack of interpretability of AI models; second, the lack of external validation so that the model generalizability across institutions cannot be guaranteed; and third, the risk of automation bias, where users tend to rely too much on the system output rather than actively seeking information and assessing the model uncertainty.

Although numerous AI methods, especially deep learning, have demonstrated remarkable performance in medicine, surprisingly, few constructed models have been used in clinical practice, with poor interpretability being a major reason [ 9 ]. Clinicians need to understand how AI-based algorithms generate their predictions and gain insight into the precise changes in risk induced by certain factors of an individual patient [ 10 , 11 ]. Moreover, the majority of current sepsis mortality prediction models published are built on data from a single hospital or a uniform health care system, where the care processes are standardized or similar [ 6 - 8 ]. It is challenging for AI models to ensure accuracy in different hospital settings [ 7 , 12 ]. Therefore, another challenge in predictive modeling is how to quantify the reliability of the model predictions for new patients, especially when such data are outside the “domain” on which the model was trained [ 13 ]. Furthermore, most AI models only provide binary predictions, that is, yes or no, without assessing how reliable a prediction is [ 14 ]. However, when using AI models in high-risk environments (such as ICUs), uncertainty quantification is required to avoid unexpected model failures by gaining insight into the confidence of the predictions made by an AI algorithm [ 15 ].

To the best of our knowledge, no models have been explicitly developed to estimate the uncertainty of AI-assisted critical illness risk predictions in patients admitted to the ICU using electronic health record data. In this study, we aimed to develop and validate an AI model, called CPMORS (Conformal Predictor for Mortality Risk in Sepsis), to assess the risk of in-hospital sepsis mortality in ICU admissions. To mitigate the impact of insufficient model generalization performance and automation bias on its clinical application, we expected the model to provide confidence measures to monitor predictions and flag uncertain predictions at a customized confidence level for human intervention, as well as to provide interpretable risk factors, in this case, to improve the translation of AI-assisted sepsis prediction systems into medical practice and enable intensivists to use them in clinical decision-making.

Data Sources

The data used in this retrospective study were obtained from 2 different databases with different clinical information systems: Medical Information Mart for Intensive Care database-IV (MIMIC-IV; version 2.2; MetaVision system) [ 16 ] and eICU Collaborative Research Database (eICU-CRD; Philips eICU system) [ 17 ]. MIMIC-IV provided critical care data of 73,181 patients admitted to the ICUs at Beth Israel Deaconess Medical Center between 2008 and 2019. eICU-CRD is a large multicenter ICU database including more than 200,000 ICU admissions from 335 units in 208 hospitals across the United States between 2014 and 2015.

Ethical Considerations

The MIMIC-IV and eICU-CRD were publicly available databases and were previously ethically approved by the institutional review boards at Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology in accordance with the tenets of the Declaration of Helsinki. The waiver of the requirement for informed consent was included in the institutional review board approval as all protected health information was deidentified [ 16 , 17 ]. The authors were granted access to the database after completing training in human research and signing a data use agreement in PhysioNet (certification number: 27252652).

Participant Selection

In MIMIC-IV, participants were enrolled based on the sepsis-3 criteria [ 2 ], that is, known or suspected infection and a Sequential Organ Failure Assessment (SOFA) score ≥2 points. In eICU-CRD, patients with sepsis were identified according to the admission diagnosis recorded in the Acute Physiology and Chronic Health Evaluation IV data set [ 18 ]. For those with multiple ICU admissions, only the first ICU admission was included. Patients who were discharged from the ICU within 24 hours and were aged <18 years were excluded. Patients with >30% missing individual data were also excluded.

Data Extraction and Preprocessing

Patient data from the first 24 hours after ICU admission were retrieved from the MIMIC-IV and eICU-CRD databases to predict in-hospital mortality risk. The study retrospectively collected the following data: (1) demographic characteristics, including sex, age, BMI, and ethnicity; (2) site of infection, including pulmonary and gastrointestinal infections; (3) comorbidities, including chronic kidney disease, congestive heart failure, chronic pulmonary disease, diabetes, and liver disease; (4) worst reported Glasgow Coma Scale, Acute Physiology Score III (APS III), and SOFA score; (5) vital signs, including heart rate, respiratory rate, systolic blood pressure, mean blood pressure, diastolic blood pressure, temperature, and oxygen saturation; (6) laboratory data, including pH, lactate, bicarbonate, base excess, PaO 2 , PaCO 2 , FiO 2 , PaO 2 /FiO 2 ratio, hematocrit, hemoglobin, platelets, white blood cells count, albumin, anion gap, blood glucose, blood urea nitrogen, serum calcium, serum creatinine, serum sodium, serum potassium, international normalized ratio, prothrombin time, partial thromboplastin time, alanine transaminase, alkaline phosphatase, aspartate aminotransferase, and total bilirubin; (7) therapeutic management, including the use of vasopressors; and (8) total urine output.

We used one-hot encoding for the representation of categorical variables. For vital signs with multiple measurements during the first day, we included the maximum, minimum, mean, and SD values for analysis. For laboratory values with multiple measurements, we included the maximum and minimum values for analysis. This resulted in a total of 103 features used to train and validate AI models for in-hospital mortality risk prediction ( Multimedia Appendix 1 ). For missing data, the BMI was imputed with the k-nearest neighbors algorithm using the demographic characteristics. For the remaining missing values, if the predictive models could not support the missing data, the mean value from the training data was used to fill the remaining missing values; if the predictive models could support the missing data, no imputation was performed. All numerical features were standardized by removing the mean and scaling to unit variance. To avoid information leakage, the preprocessing operations were derived from the training data and applied to other validation data sets.

Model Development and Validation

The CPMORS model was developed to predict in-hospital mortality risk from sepsis and provide uncertain predictions and risk factors for further possible active management ( Figure 1 ). Patients with sepsis from the MIMIC-IV database were randomly divided into a development set (n=16,746, 80%) for model training and an internal validation set (n=4187, 20%) for testing ( Figure 2 ). Septic ICU admissions derived from the eICU-CRD database were used for external validation. The CPMORS model was constructed using the gradient boosting machines [ 19 ] prediction algorithm. Three common machine learning algorithms were also constructed for comparative purposes, including neural decision forest [ 20 ], random forest [ 21 ], and logistic regression [ 7 ]. Conventional scoring systems that have been widely used in clinical practice without machine learning, including APS III and SOFA, were also tested for comparison. Missing data were handled with a mean imputation method for neural decision forest, random forest, and logistic regression, while the gradient boosting machines did not require imputation.

To estimate the uncertainty of the model outputs at a customized or user-specified confidence level that can be set by clinicians, we used the conformal prediction (CP) framework built on top of the prediction algorithm. CP is a user-friendly paradigm for generating statistically rigorous uncertainty sets or intervals for the predictions of unknown samples that differ from the training data [ 15 ]. This approach could provide reliable predictions at a user-specified desired error rate (equal to the significance level and 1-confidence). The output of CP is a prediction region, that is, sets of labels, rather than a single value prediction (point prediction) for classification problems. In this study, the possible prediction sets were {survival} or {nonsurvival}, called a single prediction, {survival, nonsurvival}, called multiple predictions, or {null} called the empty set. Central to CP is the use of nonconformity measures to assess how dissimilar a new sample is from the data on which the model was built. In this study, we used a common nonconformity measure, that is, using the predicted probability of an example belonging to a given class to calculate the nonconformity score. The nonconformity was then used to calculate a P value for each possible class label when making a prediction using the conformal predictor. The P value represented the proportion of observations with more extreme nonconformity scores. Labels were included in the prediction set if the P value exceeded a user-specified desired significance level (1-confidence), such as .05 ( Table 1 ). A multiple prediction meant that the prediction was uncertain, and the model could not distinguish between survival and nonsurvival. Empty predictions were examples where the model could not assign any label, typically meaning that the example differed from the data the model was trained on. In this study, a Mondrian CP was specifically implemented to handle classification tasks with unbalanced data [ 22 ]. It could work on a class basis to ensure the desired error rate within each class. At a higher confidence level, we got fewer error predictions but more multiple predictions ( Table 2 ). To develop the Mondrian conformal predictor, we further split the development set into a training set (n=13,397, 80%) and a calibration set (n=3349, 20%; Figure 2 ). The training set was used to train the AI prediction algorithm, while the calibration set was used to construct the conformal predictor and also to tune the model hyperparameters using a Bayesian optimizer [ 23 ].

a Label 0=survival, label 1=nonsurvival.

b Correct and error predictions refer to single predictions.

To explain the model, the impact of the features on the risk output was quantified using Shapley additive explanation (SHAP) values [ 24 ] to obtain interpretability for the developed model. We provide both global feature importance from the whole population outputs and individual interpretability for a single patient output.

Statistical Analysis

Continuous variables were expressed as median with IQR, and 2 groups were compared using the Wilcoxon rank sum test. Categorical variables were expressed as numbers and percentages and compared using chi-square tests. The discriminative performance of the model for predicting sepsis mortality was assessed using the area under the curve (AUC), and calibration was performed using calibration curves and the Brier score (the lower the better). We calculated the 95% CIs for these metrics using bootstrapping (2000 stratified bootstrap replicates).

The conformal predictor could produce multiple predictions or empty predictions in cases where it could not assign reliable single predictions. Therefore, it was not possible to directly calculate the sensitivity and specificity of the CP. To evaluate the CP framework, we assessed efficiency, defined as the proportion of all predictions that resulted in a single correct prediction, and validity (the error rate), the proportion of all predictions that did not exceed the prespecified significance level [ 15 ]. All statistical analysis and calculations were performed using R (version 4.2.2; R Foundation for Statistical Computing) and Python (version 3.8.16; Python Software Foundation).

Patient Description

This study follows the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [ 25 ]. A total of 20,933 adult patients from the MIMIC-IV database meeting the sepsis-3 criteria were analyzed, of whom 3457 (16.5%) were nonsurvivors. The external validation cohort from the eICU-CRD included 10,362 patients with sepsis (n=1757, 17% for nonsurvivors). Table 3 describes the baseline characteristics between survivors and nonsurvivors of patients with sepsis admitted to the ICU. Multimedia Appendix 1 shows that 93 out of 103 features were statistically different ( P ≤.05) between MIMIC-IV and eICU-CRD. In both data sets, compared with patients whose outcome was in-hospital survival, nonsurvivors were older, had a higher BMI, had more pulmonary infections, were more likely to receive vasopressors, had higher APS III and SOFA scores, and had a longer ICU stay.

a ICU: intensive care unit.

b MIMIC-IV: Medical Information Mart for Intensive Care database-IV.

c eICU-CRD: eICU Collaborative Research Database.

d APS III: Acute Physiology Score III.

e SOFA: Sequential Organ Failure Assessment.

Prediction Performance and Explanation

Figure 3 shows the prediction performance of the developed 6 models in terms of receiver operating characteristic curves. When tested on the internal validation population for MIMIC-IV, the AUC value obtained using the proposed CPMORS model was 0.858 (95% CI 0.845-0.871), which is significantly higher than the other 3 machine learning models ranging from 0.829 (95% CI 0.811-0.844) to 0.853 (95% CI 0.838-0.867) and the 2 scoring systems with 0.708 (95% CI 0.685-0.728) of SOFA and 0.757 (95% CI 0.737-0.774) of APS III. When externally validated on the multicenter database eICU-CRD, the performance of all models deteriorated due to interdatabase heterogeneity. Nevertheless, CPMORS still showed the best performance with AUC 0.800 (95% CI 0.789-0.811) versus other models ranging from 0.693 (95% CI 0.680-0.706) to 0.786 (95% CI 0.775-0.798). In addition, the model showed good calibration ( Figure 3 C), where the curves were close to the diagonal dash line. The Brier score was 0.101 (95% CI 0.096-0.106) in MIMIC-IV and 0.116 (95% CI 0.113-0.119) in eICU-CRD, indicating its high calibration ability.

The top 15 clinical features that contributed to the prediction of sepsis mortality risk are summarized in Figure 3 D. The interpretable summary of the impact of the features across patients showed that a higher APS III score, older age, oliguria, the use of vasopressor, pulmonary infection, a higher lactate, a lower BMI, a higher white blood cell count, an unstable oxygen saturation, and a lower platelet count were associated with a higher mortality risk, which are consistent with the previous statistical analysis. Figure 3 E shows examples of the relationship between SHAP value and feature value. Explanations representing the effects of interpretable sets of extracted features for an individual nonsurvivor and survivor are shown in Figure 3 F and G. These effects explained why the model predicted a particular risk, allowing a clinician to plan appropriate interventions.

Results for CP

Calibration curves of the observed prediction error at a significance (1-confidence) level between 0% and 100% showed that CPMORS was well calibrated for the internal MIMIC-IV validation when the populations were from the same hospital system ( Figure 4 ). The calibration curve for external eICU-CRD populations deviated from the ideal diagonal line but was good when the significance level was below 20%. When the conformal predictor was set at a prespecified confidence level of 90% to provide valid predictions, for the MIMIC-IV validation cohort, there were 2520 (efficiency=60.2%) single correct predictions and 1229 (29.4%) multiple predictions ( Table 4 ). The overall validity, the percentage of error predictions (n=438) out of all predictions (n=4187), was 10.5%. In contrast, the AI model without CP made 1449 (34.6%) errors in predicting sepsis mortality risk. When the model was externally validated on a large multicenter database, that is, eICU-CRD, more multiple predictions (n=4004, 38.6%) were flagged for clinician review. In this case, the conformal predictor still produced significantly lower error rates compared to the point predictions by AI (n=1221, 11.8% vs n=4540, 43.8%). No empty predictions were made in either validation cohort.

a MIMIC-IV: Medical Information Mart for Intensive Care database-IV.

b eICU-CRD: eICU Collaborative Research Database.

c AI: artificial intelligence.

d For comparison with the AI model with conformal prediction at a user-specified 90% confidence level, the AI point prediction without conformal prediction was made when setting the error rate in nonsurvivors as 11.1% for MIMIC-IV validation and 12.7% for eICU-CRD validation.

The top 15 features summarized by the SHAP values could not be well differentiated between the multiple predictions in survivors and nonsurvivors ( Table 5 and Multimedia Appendix 2 ). In addition, for those patients who survived, the analysis showed that patients identified for clinician review (those with uncertain predictions) were more likely to develop new acute kidney injury after the first day of ICU admission (n=227, 22.4% vs n=428, 20.2% for MIMIC-IV; n=799, 24% vs n=875, 20.4% for eICU-CRD) and had longer ICU stays with a median of 89 (IQR 54-173) hours versus 55 (IQR 37-99) hours for MIMIC-IV, and 78 (IQR 48-141) hours versus 55 (IQR 40-91) hours for eICU-CRD, compared to the patients who were not flagged for review.

b Statistical analysis between survival and nonsurvival multiple predictions.

c APS III: Acute Physiology Score III.

d WBC: white blood cell.

e SpO 2 : oxygen saturation.

f PLT: platelet.

g BUN: blood urea nitrogen.

h aPTT: activated partial thromboplastin time.

Principal Findings

This study proposed CPMORS, an AI model to provide reliable predictions of mortality risk for patients with sepsis admitted to the ICU. CPMORS was developed using gradient boosting machines with 103 clinical features. Internal and external validation assessed its best predictive performance in terms of discrimination and calibration compared with several other commonly used models. Global feature importance showed that a higher APS III score, older age, oliguria, the use of vasopressor, and pulmonary infection were most associated with a higher mortality risk. A Mondrian CP was built on top of the prediction algorithm to detect uncertain predictions. Compared to traditional AI models that only provide point predictions, our method could provide reliable and uncertain predictions with user-specified confidence, especially when the performance of the external validation could not be guaranteed as good as the internal validation. Interpretable information and estimates of prediction uncertainty enabled CPMORS to provide informative support for clinical decision-making.

Recent guidelines on AI-based predictive models in health care emphasize the importance of AI systems conveying their prediction confidence to users while furnishing accurate predictions and explanations [ 26 ]. In previous studies, Zhang et al [ 27 ] developed a traditional sepsis mortality risk score system that is interpretable and transparent, but the accuracy of the model is limited, with AUC in the development and validation sets of 0.789 and 0.765. Park et al [ 6 ] and Kong et al [ 7 ] demonstrated that machine learning can improve the accuracy of sepsis mortality risk prediction, but there is a lack of strong interpretable analysis. Hu et al [ 8 ] performed a detailed and interpretable analysis. However, to date, no studies have estimated the prediction uncertainty of sepsis mortality risk in patients admitted to the ICU. In the case of physicians supported by AI systems, such feedback of prediction uncertainty was invaluable in enabling them to exercise caution and not rely solely on the output of the AI system. This could serve to safeguard patients from the potential hazards of automation bias in AI systems [ 28 ]. Various methods for quantifying uncertainty, such as the Gaussian process [ 29 ], Bayesian inference [ 30 ], deep ensembles [ 31 ], and dropout [ 32 ], have been implemented in computer vision and natural language processing applications. However, their use in clinical decision-making has been limited due to the accompanying computational cost and modeling complexity. In addition, they did not provide adjustable confidence levels to suit different clinical requirements for predicting critical illness.

In this study, we used the CP framework, which is characterized by its light mathematical nature and offers a potential solution by generating prediction regions. These prediction regions are similar to the confidence intervals used in statistics, except that they are based on individual predictions rather than overall statistics [ 14 ]. The use of CP-derived prediction regions could therefore provide reliable and uncertain predictions in clinical settings. In addition, clinicians would not rely entirely on the model’s output and make immediate decisions [ 28 ]. They still need to review patients’ critical information before making decisions. Throughout this process, CPMORS could not only provide explainable risk factors so that clinicians can double-check the reasoning of AI systems and identify problematic elements [ 9 ] but could also help to achieve stratified patient management ( Figure 1 ). The single predictions of nonsurvival made by CPMORS could be treated as red flags, meaning that clinicians should pay the most attention. The single survival predictions could be treated as green flags, meaning that the patients are relatively safe and would be monitored regularly. For the multiple predictions, we flag the yellow warnings as such unreliable cases require clinician review; it is clinically very important that AI-based decision support systems also flag uncertainty when they are not certain about the produced output [ 13 ]. In this case, CPMORS could produce much fewer false alarms to avoid exposing patients to unnecessary tests and treatments [ 33 ]. The process also enables a symbiotic interaction between AI and clinicians, which is the key to AI adoption [ 34 ]. The AI system expertly identifies cases where its predictions are highly reliable, allowing clinicians to focus their attention on challenging and less reliable predictions. Therefore, this process would not delay critical decisions. Instead, the proposed AI model could provide explainable information and also help take the pressure off clinicians by stratifying patients. Furthermore, survivors whose predictions were uncertain (multiple predictions) were more likely to develop acute kidney injury and had a longer ICU stay, which proves the validity of their clinician review. In this case, a real-time compatible software pipeline can be developed into clinical decision systems ( Figure 1 ) by fulfilling the requirements stated in de Hond et al [ 26 ]. This includes providing an explanation of the model output, enabling end users to visually comprehend the connection between the input data and the predicted output, fostering feedback, and improvement of the predictions.

A systematic review of AI in critical care concluded that it is important to ensure the validity and reliability of predictive models across clinical settings [ 35 ]. However, it has been reported that only 6% of studies of AI applications in health care have performed external validation [ 36 ], and the sepsis mortality risk prediction model is no exception. Furthermore, previous research on external validation shows a significant decrease when the AI system is applied in different hospital settings or data sets due to several potential threats, such as population heterogeneity, differences in clinical practice, and software diversity [ 11 , 37 ]. Consistent with these studies, the prediction results in this study reported that all AI models deteriorated when externally validated on eICU-CRD, with the AUC of CPMORS decreasing from 0.858 to 0.800. Although transfer learning (TL) methods have been proposed to improve the model generalization ability, additional requirements are needed. For example, additional labeled target data are required for supervised TL (fine-tuning) [ 38 ] or many unlabeled data for unsupervised TL (domain adaptation) [ 39 ]. Although model performance can be improved, uncertainty estimation is not guaranteed. In this study, the proposed CPMORS did not require target data, could provide confidence measures, and was mathematically proven to be valid, which were important for application across hospital settings. For a given confidence level, the conformal predictor provided a prediction interval within which the true value should lie with a probability of the given confidence, that is, the true class is in the prediction set. Therefore, CPMORS did not aim to directly address the problem of insufficient model generalization performance by simply improving the model’s accuracy, as it is impossible for the model to achieve 100% accuracy. CPMORS could help to mitigate the risk of the model generalizability issue from another perspective, by flagging those patients whose predictions are uncertain due to dissimilarity to the training samples and reminding clinicians to check. In addition, CP can also help to detect systematic differences, as shown in Olsson et al [ 15 ] for the diagnosis of prostate cancer.

However, we should carefully consider the trade-off between producing single predictions and multiple predictions, as the number of uncertain predictions should be kept limited to avoid overloading clinicians and creating an unmanageable situation. We can see that the proportions of single predictions, empty predictions, and multiple predictions can vary for different levels of significance (1-confidence; Figure 4 ). To increase confidence in the estimates, a low level of significance must be set, but this would generate an increase in the number of multiple predictions (uncertain predictions). The findings of this study indicate that, when the prespecified confidence level is set to 90%, the CPMORS provides a prediction region around the point prediction that contains the true label with a 90% probability. Our results show that, for the MIMIC-IV and eICU-CRD data sets, CPMORS generated a total of 217 (31.4%) and 681 (38.8%) multiple predictions for nonsurvivors, respectively, and 1012 (28.9%) and 3323 (38.6%) multiple predictions for survivors, respectively ( Table 4 ). The implication of the multiple predictions in this study is that there is insufficient information for the model to discriminate between survival and nonsurvival outcomes. This is demonstrated by the inability to effectively discriminate the top 15 features between the multiple predictions of survivors and nonsurvivors ( Table 5 and Multimedia Appendix 2 ), which reminds us that new information from patients should be provided to arrive at a single correct prediction.

Limitations

This study has several limitations. First, although we have demonstrated this model can help to mitigate the impact of insufficient model generalization performance and automation bias by providing the uncertainty estimation, we still need further studies to collect weekly or monthly statistics of uncertainty rates so that we can set a balanced confidence level for more efficient patient management. Second, more advanced nonconformal functions should be tested to achieve smaller prediction regions and higher proportions of reliable single predictions. Third, this work was not designed to directly address generalizability and bias issues, although it touched upon them. Future work is needed to solve these 2 issues more thoroughly. Nevertheless, we provided an experimental example to describe how a combination of model interpretability and CP could work to assist clinicians in predicting sepsis mortality in ICU admissions. Conformal predictors are built on top of the underlying prediction algorithm; therefore, the framework can be applied to all prediction algorithms and other predictive tasks in critical illness prediction.

Conclusions

In summary, this study presents the development and validation of an AI model, called CPMORS, for predicting sepsis mortality risk in patients who are critically ill. CPMORS emerges as the most effective model among all the predictive models tested in this study. Importantly, CPMORS offers interpretability, allowing for transparency in the prediction-making process. The internal and external validation procedures demonstrate the ability of CPMORS to make reliable predictions and flag uncertain predictions. These findings suggest that the integration of model explanation and CP can enhance the clinical applicability of AI-assisted systems, thereby facilitating clinical decision-making in the context of sepsis management.

Acknowledgments

This work was funded by the National Natural Science Foundation of China (62171123, 62211530112, and 62071241), the National Key Research and Development Program of China (2023YFC3603600 and 2022YFC2405600), and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX21_0088).

Data Availability

The data sets and code generated and analyzed during this study are available on the PhysioNet website [ 40 , 41 ] and GitHub repository [ 42 ], respectively.

Authors' Contributions

The study conception and design were developed by MY, XL, JL, and CL. Material preparation, data collection, and analysis were performed by MY, HC, and WH. The predictive model was designed by MY and CL. The first draft of the manuscript was written by MY. MM, CS, and all authors commented on previous versions of the manuscript. JL provided clinical resources. All authors read and approved the final manuscript.

Conflicts of Interest

None declared.

Statistical analysis of variables between Medical Information Mart for Intensive Care database-IV and eICU Collaborative Research Database.

Statistical analysis of the top 15 features between the single correct prediction patients and the multiple prediction patients in the eICU Collaborative Research Database validation populations.

• Fleischmann C, Scherag A, Adhikari NKJ, Hartog CS, Tsaganos T, Schlattmann P, et al. Assessment of global incidence and mortality of hospital-treated sepsis. Current estimates and limitations. Am J Respir Crit Care Med. 2016;193(3):259-272. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315(8):801-810. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the global burden of disease study. Lancet. 2020;395(10219):200-211. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xie J, Wang H, Kang Y, Zhou L, Liu Z, Qin B, et al. The epidemiology of sepsis in Chinese ICUs: a national cross-sectional survey. Crit Care Med. 2020;48(3):e209-e218. [ CrossRef ] [ Medline ]
• Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med. 2020;383(20):1951-1960. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Park JY, Hsu TC, Hu J, Chen CY, Hsu WT, Lee M, et al. Predicting sepsis mortality in a population-based national database: machine learning approach. J Med Internet Res. 2022;24(4):e29982. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kong G, Lin K, Hu Y. Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU. BMC Med Inform Decis Mak. 2020;20(1):251. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hu C, Li L, Huang W, Wu T, Xu Q, Liu J, et al. Interpretable machine learning for early prediction of prognosis in sepsis: a discovery and validation study. Infect Dis Ther. 2022;11(3):1117-1132. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28(1):31-38. [ CrossRef ] [ Medline ]
• Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. [ CrossRef ] [ Medline ]
• Yang M, Liu C, Wang X, Li Y, Gao H, Liu X, et al. An explainable artificial intelligence predictor for early detection of sepsis. Crit Care Med. 2020;48(11):e1091-e1096. [ CrossRef ] [ Medline ]
• Panch T, Mattie H, Celi LA. The "inconvenient truth" about AI in healthcare. NPJ Digit Med. 2019;2:77. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shashikumar SP, Wardi G, Malhotra A, Nemati S. Artificial intelligence sepsis prediction algorithm learns to say "i don't know". NPJ Digit Med. 2021;4(1):134. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alvarsson J, McShane SA, Norinder U, Spjuth O. Predicting with confidence: using conformal prediction in drug discovery. J Pharm Sci. 2021;110(1):42-49. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Olsson H, Kartasalo K, Mulliqi N, Capuccini M, Ruusuvuori P, Samaratunga H, et al. Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction. Nat Commun. 2022;13(1):7761. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Johnson AEW, Pollard TJ, Shen L, Lehman LWH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU collaborative research database, a freely available multi-center database for critical care research. Sci Data. 2018;5:180178. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297-1310. [ CrossRef ] [ Medline ]
• Chen T, Guestrin CE. XGBoost: A Scalable Tree Boosting System. In: ACM Digital Library. 2016. Association for Computing Machinery; Presented at: KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 13-17, 2016;785-794; San Francisco, CA. URL: https://dl.acm.org/doi/proceedings/10.1145/2939672 [ CrossRef ]
• Kontschieder P, Fiterau M, Criminisi A, Bulò SR. Deep Neural Decision Forests. In: Computer Vision Foundation. 2015. Presented at: IEEE International Conference on Computer Vision (ICCV); December 07-13, 2015; Santiago, Chile. URL: https://ieeexplore.ieee.org/document/7410529 [ CrossRef ]
• Lin K, Hu Y, Kong G. Predicting in-hospital mortality of patients with acute kidney injury in the ICU using random forest model. Int J Med Inform. 2019;125:55-61. [ CrossRef ] [ Medline ]
• Toccaceli P, Gammerman A. Combination of inductive mondrian conformal predictors. Mach Learn. 2018;108(3):489-510. [ CrossRef ]
• Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N. Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE. 2016;104(1):148-175. [ FREE Full text ] [ CrossRef ]
• Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, et al. From local explanations to global understanding with explainable AI for trees. Nat Mach Intell. 2020;2(1):56-67. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323. [ FREE Full text ] [ CrossRef ] [ Medline ]
• de Hond AAH, Leeuwenberg AM, Hooft L, Kant IMJ, Nijman SWJ, van Os HJA, et al. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. NPJ Digit Med. 2022;5(1):2. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang K, Zhang S, Cui W, Hong Y, Zhang G, Zhang Z. Development and validation of a sepsis mortality risk score for sepsis-3 patients in intensive care unit. Front Med (Lausanne). 2020;7:609769. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Volovici V, Syn NL, Ercole A, Zhao JJ, Liu N. Steps to avoid overuse and misuse of machine learning in clinical research. Nat Med. 2022;28(10):1996-1999. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lederer A, Umlauft J, Hirche S. Uniform error bounds for Gaussian process regression with application to safe control. ArXiv. Preprint posted online Dec 19, 2019. [ FREE Full text ] [ CrossRef ]
• Ghosh S, Yao J, Doshi-Velez F. Structured variational learning of Bayesian neural networks with horseshoe priors. ArXiv. Preprint posted online Jul 31, 2018. [ FREE Full text ] [ CrossRef ]
• Lakshminarayanan B, Pritzel A, Blundell C. Simple and scalable predictive uncertainty estimation using deep ensembles. ArXiv. Preprint posted online Nov 4, 2017. [ FREE Full text ] [ CrossRef ]
• Gal Y, Ghahramani Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. PMLR. 2016;48:1050-1059. [ FREE Full text ] [ CrossRef ]
• Mitka M. Joint commission warns of alarm fatigue: multitude of alarms from monitoring devices problematic. JAMA. 2013;309(22):2315-2316. [ CrossRef ] [ Medline ]
• Henry KE, Kornfield R, Sridharan A, Linton RC, Groh C, Wang T, et al. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system. NPJ Digit Med. 2022;5(1):97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yoon JH, Pinsky MR, Clermont G. Artificial intelligence in critical care medicine. Crit Care. 2022;26(1):75. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol. 2019;20(3):405-410. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Song X, Yu ASL, Kellum JA, Waitman LR, Matheny ME, Simpson SQ, et al. Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction. Nat Commun. 2020;11(1):5668. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Holder AL, Shashikumar SP, Wardi G, Buchman TG, Nemati S. A locally optimized data-driven tool to predict sepsis-associated vasopressor use in the ICU. Crit Care Med. 2021;49(12):e1196-e1205. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhu Y, Venugopalan J, Zhang Z, Chanani NK, Maher KO, Wang MD. Domain adaptation using convolutional autoencoder and gradient boosting for adverse events prediction in the intensive care unit. Front Artif Intell. 2022;5:640926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• MIMIC-IV data. PhysioNet. URL: https://physionet.org/content/mimiciv/2.2/ [accessed 2023-04-28]
• eICU-CRD Data. MIT Laboratory. URL: http://eicu-crd.mit.edu/ [accessed 2023-04-28]
• CPMORS (Conformal Predictor for Mortality Risk in Sepsis). GitHub. URL: https://github.com/Meicheng-SEU/CPMORS [accessed 2023-10-15]

Abbreviations

Edited by T de Azevedo Cardoso; submitted 29.06.23; peer-reviewed by Z Xu, E Kawamoto; comments to author 12.10.23; revised version received 16.10.23; accepted 24.01.24; published 18.03.24.

©Meicheng Yang, Hui Chen, Wenhan Hu, Massimo Mischi, Caifeng Shan, Jianqing Li, Xi Long, Chengyu Liu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

This paper is in the following e-collection/theme issue:

Published on 18.3.2024 in Vol 10 (2024)

Predicting COVID-19 Vaccination Uptake Using a Small and Interpretable Set of Judgment and Demographic Variables: Cross-Sectional Cognitive Science Study

Authors of this article:

Original Paper

• Nicole L Vike 1 , PhD   ; 
• Sumra Bari 1 , PhD   ; 
• Leandros Stefanopoulos 2, 3 * , MSc   ; 
• Shamal Lalvani 2 * , MSc   ; 
• Byoung Woo Kim 1 * , MSc   ; 
• Nicos Maglaveras 3 , PhD   ; 
• Martin Block 4 , PhD   ; 
• Hans C Breiter 1, 5 , MD   ; 
• Aggelos K Katsaggelos 2, 6, 7 , PhD  

1 Department of Computer Science, University of Cincinnati, Cincinnati, OH, United States

2 Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, United States

3 School of Medicine, Faculty of Health Sciences, Aristotle University of Thessaloniki, Thessaloniki, Greece

4 Integrated Marketing Communications, Medill School, Northwestern University, Evanston, IL, United States

5 Department of Psychiatry, Massachusetts General Hospital, Harvard School of Medicine, Boston, MA, United States

6 Department of Computer Science, Northwestern University, Evanston, IL, United States

7 Department of Radiology, Northwestern University, Evanston, IL, United States

*these authors contributed equally

Corresponding Author:

Hans C Breiter, MD

Department of Computer Science

University of Cincinnati

2901 Woodside Drive

Cincinnati, OH, 45219

United States

Phone: 1 617 413 0953

Email: [email protected]

Background: Despite COVID-19 vaccine mandates, many chose to forgo vaccination, raising questions about the psychology underlying how judgment affects these choices. Research shows that reward and aversion judgments are important for vaccination choice; however, no studies have integrated such cognitive science with machine learning to predict COVID-19 vaccine uptake .

Objective: This study aims to determine the predictive power of a small but interpretable set of judgment variables using 3 machine learning algorithms to predict COVID-19 vaccine uptake and interpret what profile of judgment variables was important for prediction.

Methods: We surveyed 3476 adults across the United States in December 2021. Participants answered demographic, COVID-19 vaccine uptake (ie, whether participants were fully vaccinated), and COVID-19 precaution questions. Participants also completed a picture-rating task using images from the International Affective Picture System. Images were rated on a Likert-type scale to calibrate the degree of liking and disliking. Ratings were computationally modeled using relative preference theory to produce a set of graphs for each participant (minimum R 2 >0.8). In total, 15 judgment features were extracted from these graphs, 2 being analogous to risk and loss aversion from behavioral economics. These judgment variables, along with demographics, were compared between those who were fully vaccinated and those who were not. In total, 3 machine learning approaches (random forest, balanced random forest [BRF], and logistic regression) were used to test how well judgment, demographic, and COVID-19 precaution variables predicted vaccine uptake . Mediation and moderation were implemented to assess statistical mechanisms underlying successful prediction.

Results: Age, income, marital status, employment status, ethnicity, educational level, and sex differed by vaccine uptake (Wilcoxon rank sum and chi-square P <.001). Most judgment variables also differed by vaccine uptake (Wilcoxon rank sum P <.05). A similar area under the receiver operating characteristic curve (AUROC) was achieved by the 3 machine learning frameworks, although random forest and logistic regression produced specificities between 30% and 38% (vs 74.2% for BRF), indicating a lower performance in predicting unvaccinated participants. BRF achieved high precision (87.8%) and AUROC (79%) with moderate to high accuracy (70.8%) and balanced recall (69.6%) and specificity (74.2%). It should be noted that, for BRF, the negative predictive value was <50% despite good specificity. For BRF and random forest, 63% to 75% of the feature importance came from the 15 judgment variables. Furthermore, age, income, and educational level mediated relationships between judgment variables and vaccine uptake .

Conclusions: The findings demonstrate the underlying importance of judgment variables for vaccine choice and uptake, suggesting that vaccine education and messaging might target varying judgment profiles to improve uptake. These methods could also be used to aid vaccine rollouts and health care preparedness by providing location-specific details (eg, identifying areas that may experience low vaccination and high hospitalization).

Introduction

In early 2020, the COVID-19 pandemic wreaked havoc worldwide, triggering rapid vaccine development efforts. Despite federal, state, and workplace vaccination mandates, many individuals made judgments against COVID-19 vaccination, leading researchers to study the psychology underlying individual vaccination preferences and what might differentiate the framework for judgment between individuals who were fully vaccinated against COVID-19 and those who were not (henceforth referred to as vaccine uptake ). A better understanding of these differences in judgment may highlight targets for public messaging and education to increase the incidence of choosing vaccination.

Multiple studies have sought to predict an individual’s intention to receive a COVID-19 vaccine or specific variables underlying vaccination choices or mitigation strategies [ 1 - 7 ], but few have predicted vaccine uptake . One such study used 83 sociodemographic variables (with education, ethnicity, internet access, income, longitude, and latitude being the most important predictors) to predict vaccine uptake with 62% accuracy [ 8 ], confirming both the importance and limitations of these variables in prediction models. Other studies have compared demographic groups between vaccinated and nonvaccinated persons; Bulusu et al [ 9 ] found that young adults (aged 18-35 years), women, and those with higher levels of education had higher odds of being vaccinated. In a study of >12 million persons, the largest percentage of those who initiated COVID-19 vaccination were White, non-Hispanic women between the ages of 50 and 64 years [ 10 ]. Demographic variables are known to affect how individuals judge what is rewarding or aversive [ 11 , 12 ] yet are not themselves variables quantifying how individuals make judgments that then frame decisions.

Judgment reflects an individual’s preferences, or the variable extent to which they approach or avoid events in the world based on the rewarding or aversive effects of these events [ 13 - 15 ]. The definition of preference in psychology differs from that in economics. In psychology, preferences are associated with “wanting” and “liking” and are framed by judgments that precede decisions, which can be quantified through reinforcement reward or incentive reward tasks [ 12 , 16 - 21 ]. In economics, preferences are relations derived from consumer choice data (refer to the axioms of revealed preference [ 22 ]) and reflect choices or decisions based on judgments that place value on behavioral options. Economist Paul Samuelson noted that decisions are “assumed to be correlative to desire or want” [ 23 ]. In this study, we focused on a set of variables that frame judgment, with the presumption that judgments precede choices [ 12 , 20 ]. Variables that frame judgment can be derived from tasks using operant key-pressing tasks that quantify “wanting” [ 24 - 33 ] or simple rating tasks that are analogous to “liking” [ 20 , 34 ]. Both operant keypress and rating tasks measure variables that quantify the average (mean) magnitude ( K ), variance ( σ ), and pattern (ie, Shannon entropy [ H ]) of reward and aversion judgments [ 35 ]. We refer to this methodology and the multiple relationships between these variables and features based on their graphical relationships as relative preference theory (RPT; Figure 1 ) [ 18 , 36 ]. RPT has been shown to produce discrete, recurrent, robust, and scalable relationships between judgment variables [ 37 ] that produce mechanistic models for prediction [ 33 ], and which have demonstrated relationships to brain circuitry [ 24 - 27 , 30 ] and psychiatric illness [ 28 ]. Of the graphs produced for RPT, 2 appear to resemble graphs derived with different variables in economics, namely, prospect theory [ 38 ] and the mean-variance function for portfolio theory described by Markowitz [ 39 ]. Given this graphical resemblance, it is important to note that RPT functions quantifying value are not the same as standard representations of preference in economics. Behavioral economic variables such as loss aversion and risk aversion [ 38 , 40 - 51 ] are not to be interpreted in the same context given that both reflect biases and bounds to human rationality. In psychology, they are grounded in judgments that precede decisions, whereas in economics, they are grounded in consumer decisions themselves. Going forward, we will focus on judgment-based loss aversion, representing the overweighting of negative judgments relative to positive ones, and judgment-based risk aversion, representing the preference for small but certain assessments over larger but less certain ones (ie, assessments that have more variance associated with them) [ 38 , 40 - 51 ]. Herein, loss aversion and risk aversion refer to ratings or judgments that precede decisions.

A number of studies have described how risk aversion and other judgment variables are important for individual vaccine choices and hesitancies [ 52 - 58 ]. Hudson and Montelpare [ 54 ] found that risk aversion may promote vaccine adherence when people perceive contracting a disease as more dangerous or likely. Trueblood et al [ 52 ] noticed that those who were more risk seeking (as measured via a gamble ladder task) were more likely to receive the vaccine even if the vaccine was described as expedited. Wagner et al [ 53 ] described how risk misperceptions (when the actual risk does not align with the perceived risk) may result from a combination of cognitive biases, including loss aversion. A complex theoretical model using historical vaccine attitudes grounded in decision-making has also been proposed to predict COVID-19 vaccination, but this model has not yet been tested [ 59 ]. To our knowledge, no study has assessed how well a model comprising variables that reflect reward and aversion judgments predicts vaccine uptake .

Goal of This Study

Given the many vaccine-related issues that occurred during the COVID-19 pandemic (eg, vaccine shortages, hospital overload, and vaccination resistance or hesitancy), it is critical to develop methods that might improve planning around such shortcomings. Because judgment variables are fundamental to vaccine choice, they provide a viable target for predicting vaccine uptake . In addition, the rating methodology used to quantify variables of judgment is independent of methods quantifying vaccine uptake or intent to vaccinate, limiting response biases within the study data.

In this study, we aimed to predict COVID-19 vaccine uptake using judgment, demographic, and COVID-19 precaution (ie, behaviors minimizing potential exposure to COVID-19) variables using multiple machine learning algorithms, including logistic regression, random forest, and balanced random forest (BRF). BRF was hypothesized to perform best given its potential benefits with handling class imbalances [ 60 ], balancing both recall and specificity, and producing Gini scores that provide relative variable importance to prediction. In this study, the need for data imbalance techniques was motivated by the importance of the specificity metric, which would reflect the proportion of participants who did not receive full vaccination; without balancing, the model might not achieve similar recall and specificity values. When there is a large difference between recall and specificity, specificity might instead reflect the size of the minority class (those who did not receive full vaccination). In general, random forest approaches have been reported to have benefits over other approaches such as principal component analysis and neural networks, in which the N-dimensional feature space or layers (in the case of neural networks) are complex nonlinear functions, making it difficult to interpret variable importance and relationships to the outcome variable. To provide greater certainty about these assumptions, we performed logistic regression in parallel with random forest and BRF. The 3 machine learning approaches used a small feature set (<20) with interpretable relationships to the predicted variable. Such interpretations may not be achievable in big data approaches that use hundreds to thousands of variables that seemingly add little significance to the prediction models. Interpretation was facilitated by (1) the Gini importance criterion associated with BRF and random forest, which provided a profile of the judgment variables most important for prediction; and (2) mediation and moderation analyses that offered insights into statistical mechanisms among judgment variables, demographic (contextual) variables, and vaccine uptake . Determining whether judgment variables are predictive of COVID-19 vaccine uptake and defining which demographic variables facilitate this prediction presents a number of behavioral targets for vaccine education and messaging—and potentially identifies actionable targets for increasing vaccine uptake .

More broadly, the prediction of vaccine uptake may aid (1) vaccine supply chain and administration logistics by indicating areas that may need more or fewer vaccines, (2) targeted governmental messaging to locations with low predicted uptake, and (3) preparation of areas that may experience high cases of infection that could ultimately impact health care preparedness and infrastructure. The proposed method could also be applied to other mandated or government-recommended vaccines (eg, influenza and human papillomavirus) to facilitate the aforementioned logistics. Locally, vaccine uptake prediction could facilitate local messaging and prepare health care institutions for vaccine rollout and potential hospital overload. Nationally, prediction might inform public health officials and government communication bodies that are responsible for messaging and vaccine rollout with the goal of improving vaccine uptake and limiting infection and hospital overload.

Recruitment

Similar recruitment procedures for a smaller population-based study have been described previously [ 61 - 63 ]. In this study, participants were randomly sampled from the general US population using an email survey database used by Gold Research, Inc. Gold Research administered questionnaires in December 2021 using recruitment formats such as (1) customer databases from large companies that participate in revenue-sharing agreements, (2) social media, and (3) direct mail. Recruited participants followed a double opt-in consent procedure that included primary participation in the study as well as secondary use of anonymized, deidentified data (ie, all identifying information was removed by Gold Research before retrieval by the research group) in secondary analyses (refer to the Ethical Considerations section for more detail). During consent procedures, participants provided demographic information (eg, age, race, and sex) to ensure that the sampled participants adequately represented the US census at the time of the survey (December 2021). Respondents were also presented with repeated test questions to screen out those providing random and illogical responses or showing flatline or speeder behavior. Participants who provided such data were flagged, and their data were removed.

Because other components of the survey required an adequate sample of participants with mental health conditions, Gold Research oversampled 15% (60,000/400,000) of the sample for mental health conditions, and >400,000 respondents were contacted to complete the questionnaire. Gold Research estimated that, of the 400,000 participants, >300,000 (>75%) either did not respond or declined to participate. Of the remaining 25% (100,000/400,000) who clicked on the survey link, >50% (52,000/100,000) did not fully complete the questionnaire. Of the ≥48,000 participants who completed the survey (ie, ≥48,000/400,000, ≥12% of the initial pool of queried persons), those who did not clear data integrity assessments were omitted. Participants who met quality assurance procedures (refer to the following section) were selected, with a limit of 4000 to 4050 total participants.

Eligible participants were required to be aged between 18 and 70 years at the time of the survey, comprehend the English language, and have access to an electronic device (eg, laptop or smartphone).

Ethical Considerations

All participants provided informed consent, which included their primary participation in the study as well as the secondary use of their anonymized, deidentified data (ie, all identifying information removed by Gold Research before retrieval by the research group) in secondary analyses. This study was approved by the Northwestern University institutional review board (approval STU00213665) for the initial project start and later by the University of Cincinnati institutional review board (approval 2023-0164) as some Northwestern University investigators moved to the University of Cincinnati. All study approvals were in accordance with the Declaration of Helsinki. All participants were compensated with US $10 for taking part. Detailed survey instructions have been published previously [ 61 - 63 ].

Quality Assurance and Data Exclusion

Three additional quality assurance measures were used to flag nonadhering participants: (1) participants who indicated that they had ≥10 clinician-diagnosed illnesses (refer to Figure S1 in Multimedia Appendix 1 [ 18 , 33 , 36 , 64 - 68 ] for a list), (2) participants who showed minimal variance in the picture-rating task (ie, all pictures were rated the same or the ratings varied only by 1 point; refer to the Picture-Rating Task section), and (3) inconsistencies between educational level and years of education and participants who completed the questionnaire in <800 seconds.

Data from 4019 participants who passed the initial data integrity assessments were anonymized and then sent to the research team. Data were further excluded if the quantitative feature set derived from the picture-rating task was incomplete or if there were extreme outliers (refer to the RPT Framework section). Using these exclusion criteria, of the 4019 participants, 3476 (86.49%) were cleared for statistical analysis, representing 0.87% (3476/400,000) of the initial recruitment pool. A flowchart of participant exclusion is shown in Figure 2 .

Questionnaire

Participants were asked to report their age, sex, ethnicity, annual household income, marital status, employment status, and educational level. Participants were asked to report whether they had received the full vaccination ( yes or no responses). At the time of the survey, participants were likely to have received either 2 doses of the Pfizer or Moderna vaccine or 1 dose of the Johnson & Johnson vaccine as per the Centers for Disease Control and Prevention guidelines. Participants were also asked to respond yes (they routinely followed the precaution) or no (they did not routinely follow the precaution) to 4 COVID-19 precaution behaviors: mask wearing, social distancing, washing or sanitizing hands, and not gathering in large groups (refer to Tables S1 and S2 in Multimedia Appendix 1 for the complete questions and sample sizes, respectively). In addition, participants completed a picture-rating task at 2 points during the survey (refer to the Picture-Rating Task section).

Picture-Rating Task

A picture-rating task was administered to quantify participants’ degree of liking and disliking a validated picture set using pictures calibrated over large samples for their emotional intensity and valence [ 69 , 70 ]. Ratings from this task have been mathematically modeled using RPT to define graphical features of reward and aversion judgments. Each feature quantifies a core aspect of judgment, including risk aversion and loss aversion. Judgment variables have been shown to meet the criteria for lawfulness [ 37 ] that produce mechanistic models for prediction [ 33 ], with published relationships to brain circuitry [ 24 - 27 , 30 ] and psychiatric illness [ 28 ]. A more complete description of these judgment variables and their computation can be found in the RPT Framework section and in Table 1 .

For this task, participants were shown 48 unique color images from the International Affective Picture System [ 69 , 70 ]. A total of 6 picture categories were used: sports, disasters, cute animals, aggressive animals, nature (beach vs mountains), and men and women dressed minimally, with 8 pictures per category (48 pictures in total; Figure 1 A). These images have been used and validated in research on human emotion, attention, and preferences [ 69 , 70 ]. The images were displayed on the participants’ digital devices with a maximum size of 1204 × 768 pixels. Below each picture was a rating scale from −3 ( dislike very much ) to +3 ( like very much ), where 0 indicated indifference ( Figure 1 A). While there was no time limit for selecting a picture rating, participants were asked to rate the images as quickly as possible and use their first impression. Once a rating was selected, the next image was displayed.

RPT Framework

Ratings from the picture-rating task were analyzed using an RPT framework. This framework fits approach and avoidance curves and derives mathematical features from graphical plots ( Figures 1 B-1D). These methods have been described at length in prior work and are briefly described in this section [ 11 , 18 , 33 , 36 ]. More complete descriptions and quality assurance procedures can be found in Multimedia Appendix 1 .

At least 15 judgment variables can be mathematically derived from this framework and are psychologically interpretable; they have been validated using both operant keypress [ 9 , 25 - 27 ] and picture-rating tasks [ 11 , 34 ]. The 15 judgment variables are loss aversion, risk aversion, loss resilience, ante, insurance, peak positive risk, peak negative risk, reward tipping point, aversion tipping point, total reward risk, total aversion risk, reward-aversion trade-off, trade-off range, reward-aversion consistency, and consistency range. Loss aversion, risk aversion, loss resilience, ante, and insurance are derived from the logarithmic or power-law fit of mean picture ratings ( K ) versus entropy of ratings ( H ); this is referred to as the value function ( Figure 1 B). Peak positive risk, peak negative risk, reward tipping point, aversion tipping point, total reward risk, and total aversion risk are derived from the quadratic fit of K versus the SD of picture ratings ( σ ); this is referred to as the limit function ( Figure 1 C). Risk aversion trade-off, trade-off range, risk aversion consistency, and consistency range are derived from the radial fit of the pattern of avoidance judgments ( H − ) versus the pattern of approach judgments ( H + ); this is referred to as the trade-off function ( Figure 1 D). Value (Figure S2A in Multimedia Appendix 1 ), limit (Figure S2B in Multimedia Appendix 1 ), and trade-off (Figure S2C in Multimedia Appendix 1 ) functions were plotted for 500 randomly sampled participants, and nonlinear curve fits were assessed for goodness of fit, yielding R 2 , adjusted R 2 , and the associated F statistic for all participants (Figure S2D in Multimedia Appendix 1 ). Only the logarithmic and quadratic fits are listed in Table S3 in Multimedia Appendix 1 . Each feature describes a quantitative component of a participant’s reward and aversion judgment (refer to Table 1 for abbreviated descriptions and Multimedia Appendix 1 for complete descriptions). Collectively, the 15 RPT features will be henceforth referred to as “judgment variables.” The summary statistics for these variables can be found in Table S3 in Multimedia Appendix 1 .

Statistical and Machine Learning Analyses

Wilcoxon rank sum tests, chi-square tests, and Gini importance plotting were performed in Stata (version 17; StataCorp) [ 72 ]. Machine learning algorithms were run in Python (version 3.9; Python Software Foundation) [ 73 ], where the scikit-learn (version 1.2.2) [ 74 ] and imbalanced-learn (version 0.10.1) [ 75 ] libraries were used. Post hoc mediation and moderation analyses were performed in R (version 4.2.0; R Foundation for Statistical Computing) [ 76 ].

Demographic and Judgment Variable Differences by Vaccination Uptake

Each of the 7 demographic variables (age, income, marital status, employment status, ethnicity, educational level, and sex) was assessed for differences using yes or no responses to receiving the full COVID-19 vaccination (2525/3476, 72.64% yes responses and 951/3476, 27.36% no responses), henceforth referred to as vaccine uptake . Ordinal (income and educational level) and continuous (age) demographic variables were analyzed using the Wilcoxon rank sum test ( α =.05). Expected and actual rank sums were reported using Wilcoxon rank sum tests. Nominal variables were analyzed using the chi-square test ( α =.05). For significant chi-square results, demographic response percentages were computed to compare the fully vaccinated and not fully vaccinated groups.

Each of the 15 judgment variables was assessed for differences across yes or no responses to vaccine uptake using the Wilcoxon rank sum test ( α =.05). The expected and actual rank sums were reported. Significant results ( α <.05) were corrected for multiple comparisons using the Benjamini-Hochberg correction, and Q values of <0.05 ( Q Hoch ) were reported.

Prediction Analyses

Logistic regression, random forest, and BRF were used to predict vaccine uptake using judgment, demographic, and COVID-19 precaution variables. Gini plots were produced for random forest and BRF to determine the importance of the judgment variables in predicting COVID-19 vaccination. The BRF algorithm balances the samples by randomly downsampling the majority class at each bootstrapped iteration to match the number of samples in the minority class. To provide greater certainty about the results, random forest and logistic regression were performed to compare with BRF results.

Two sets of BRF, random forest, and logistic regression analyses were run: (1) with the 7 demographic variables and 15 judgment variables included as predictors and (2) with the 7 demographic variables, 15 judgment variables, and 4 COVID-19 precaution behaviors included as predictors. COVID-19 precaution behaviors included yes or no responses to wearing a mask, social distancing, washing hands, and avoiding large gatherings (refer to Table S1 in Multimedia Appendix 1 for more details). The sample sizes for yes or no responses to the COVID-19 precaution behavior questions are provided in Table S2 in Multimedia Appendix 1 . For all 3 models, 10-fold cross-validation was repeated 100 times to obtain performance metrics, where data were split for training (90%) and testing (10%) for each of the 10 iterations in cross-validation. The averages of the performance metrics were reported across 100 repeats of 10-fold cross-validation for the test sets. The reported metrics included accuracy, recall, specificity, negative predictive value (NPV), precision, and area under the receiver operating characteristic curve (AUROC). For BRF, the Python toolbox imbalanced-learn was used to build the classifier, where the training set for each iteration of cross-validation was downsampled but the testing set was unchanged (ie, imbalanced). That is, downsampling only occurred with the bootstrapped samples for training the model, and balancing was not performed on the testing set. The default number of estimators was 100, and the default number of tree splits was 10; the splits were created using the Gini criterion. In separate analyses, estimators were increased to 300, and splits were increased to 15 to test model performance. Using the scikit-learn library, the same procedures used for BRF were followed for random forest without downsampling. Logistic regression without downsampling was implemented with a maximum of 100 iterations and optimization using a limited-memory Broyden-Fletcher-Goldfarb-Shanno solver. For logistic regression, model coefficients with respective SEs, z statistics, P values, and 95% CIs were reported.

Relative feature importance based on the Gini criterion (henceforth referred to as Gini importance ) was determined from BRF and random forest using the .feature_importances_ attribute from scikit-learn, and results were reported as the mean decrease in the Gini score and plotted in Stata. To test model performance using only the top predictors, two additional sets of BRF analyses were run: (1) with the top 3 features as predictors and (2) with the top 3 features and 15 judgment variables as predictors.

Post Hoc Mediation and Moderation

Given the importance of both judgment variables and demographic variables (refer to the Results section), we evaluated post hoc how age, income, and educational level (ie, the top 3 predictors) might statistically influence the relationship between the 15 judgment variables and COVID-19 vaccine uptake . To identify statistical mechanisms influencing our prediction results, we used mediation and moderation, which can (1) determine the directionality between variables and (2) assess variable influence in statistical relationships. Mediation is used to determine whether one variable, the mediator, statistically improves the relationship between 2 other variables (independent variables [IVs] and dependent variables [DVs]) [ 77 - 80 ]. When mediating variables improve a relationship, the mediator is said to sit in the statistical pathway between the IVs and DVs [ 77 , 80 , 81 ]. Moderation is used to test whether the interaction between an IV and a moderating variable predicts a DV [ 81 , 82 ].

For mediation, primary and secondary mediations were performed. Primary mediations included each of the 15 judgment behaviors as the IV, each of the 3 demographic variables (age, income, and educational level) as the mediator, and vaccine uptake as the DV. Secondary mediations held the 15 judgment behaviors as the mediator, the 3 demographic variables as the IV, and vaccine uptake as the DV. For moderation, the moderating variable was each of the 3 demographic variables (age, income, and educational level), the IV was each of the 15 judgment behaviors, and the DV was vaccine uptake . The mathematical procedures for mediation and moderation can be found in Multimedia Appendix 1 .

Demographic Assessment

Of the 400,000 persons queried by Gold Research, Inc, 48,000 (12%) completed the survey, and 3476 (0.87%) survived all quality assurance procedures. Participants were predominately female, married, and White individuals; employed full time with some college education; and middle-aged (mean age 51.40, SD 14.92 years; Table 2 ). Of the 3476 participants, 2525 (72.64%) reported receiving a full dose of a COVID-19 vaccine, and 951 (27.36%) reported not receiving a full dose. Participants who indicated full vaccination were predominately female, married, White individuals, and retired; had some college education; and were older on average (mean age 54.19, SD 14.13 years) when compared to the total cohort. Participants who indicated that they did not receive the full vaccine were also predominately female, married, and White individuals. In contrast to those who received the full vaccination, those not fully vaccinated were predominately employed full time, high school graduates, and of average age (mean age 43.98, SD 14.45 years; median age 45, IQR 32-56 years) when compared to the total cohort. Table 2 summarizes the demographic group sample size percentages for the total cohort, those fully vaccinated, and those not fully vaccinated.

When comparing percentages between vaccination groups, a higher percentage of male individuals were fully vaccinated, and a higher percentage of female individuals were not fully vaccinated ( Table 2 ). In addition, a higher percentage of married, White and Asian or Pacific Islander, and retired individuals indicated receiving the full vaccine when compared to the percentages of those who did not receive the vaccine ( Table 2 ). Conversely, a higher percentage of single, African American, and unemployed individuals indicated not receiving the full vaccine ( Table 2 ).

Analysis of Machine Learning Features

Demographic variable differences by vaccine uptake.

Age, income level, and educational level significantly differed between those who did and did not receive the vaccine (Wilcoxon rank sum test α <.05; Table 3 ). Those who indicated full vaccination were, on average, older (median age 59 y), had a higher annual household income (median reported income level US $50,000-$75,000), and had higher levels of education (the median reported educational level was a bachelor’s degree).

Chi-square tests revealed that marital status, employment status, sex, and ethnicity also varied by full vaccine uptake (chi-square α <.05; Table 3 ).

a N/A: not applicable.

Judgment Variable Differences by Vaccine Uptake

In total, 10 of the 15 judgment variables showed nominal rank differences ( α <.05), and 9 showed significant rank differences after correction for multiple comparisons ( Q Hoch <0.05) between those who indicated full vaccination and those who indicated that they did not receive the full vaccination ( Table 4 ). The 10 features included loss aversion, risk aversion, loss resilience, ante, insurance, peak positive risk, peak negative risk, total reward risk, total aversion risk, and trade-off range. Those who indicated full vaccination exhibited lower loss aversion, ante, peak positive risk, peak negative risk, total reward risk, and total aversion risk as well as higher risk aversion, loss resilience, insurance, and trade-off range when compared to the expected rank sum. Those who did not receive the full vaccination exhibited lower risk aversion, loss resilience, insurance, and trade-off range and higher loss aversion, ante, peak positive risk, peak negative risk, total reward risk, and total aversion risk when compared to the expected rank sum.

Machine Learning Results: Predicting Vaccination Uptake

Prediction results.

With the inclusion of demographic and judgment variables, the BRF classifier with the highest accuracy (68.9%) and precision (86.7%) in predicting vaccine uptake resulted when the number of estimators was set to 300 and the number of splits was set to 10 ( Table 5 ). With the addition of 4 COVID-19 precaution behaviors, the BRF classifier with the highest accuracy (70.8%) and precision (87.8%) to predict vaccine uptake occurred when the number of estimators was set to 300 and the number of splits was set to 10. It is notable that specificity was consistently >72%, precision was >86%, and the AUROC was >75% but the NPV was consistently <50%. For random forest and logistic regression, recall and accuracy values were higher than those for BRF, but specificity was always <39%, indicating a lower performance in predicting those who did not receive the vaccine. Precision was also lower, yet the AUROC was consistent with that of the BRF results.

a A total of 15 judgment variables ( Table 4 ), 7 demographic variables ( Table 3 ), and 4 COVID-19 precaution behavior (covid_beh) variables (Table S1 in Multimedia Appendix 1 ) were included in balanced random forest, random forest, and logistic regression models to predict COVID-19 vaccine uptake . We used 10-fold cross-validation, where the data were split 90-10 for each of the 10 iterations.

b NPV: negative predictive value.

c AUROC: area under the receiver operating characteristic curve.

d BRF: balanced random forest.

e N/A: not applicable.

Feature Importance for BRF and Random Forest

Regarding BRF, Gini importance was highest for age, educational level, and income in both BRF classifiers (both without [ Figures 3 A and 3B] and with [ Figures 3 C and 3D] inclusion of the COVID-19 precaution behaviors; refer to the clusters outlined in red in Figures 3 B and 3D). For both BRF classifiers, the top 3 predictors (age, income, and educational level) had a combined effect of 23.4% on the Gini importance for prediction. Following these predictors, the 15 judgment variables had similar importance scores for both BRF classifiers (range 0.037-0.049; refer to the clusters outlined in black in Figures 3 B and 3D). These 15 predictors had a combined effect of 62.9% to 68.7% on the Gini importance for prediction, indicating that judgment variables were collectively the most important for prediction outcomes. The least important features for predicting vaccination status were demographic variables regarding employment status, marital status, ethnicity, sex, and the 4 COVID-19 precaution behaviors. These predictors only contributed 7.3% to the Gini importance for prediction. As a follow-up analysis, BRF analyses were run using the top 3 features from both the Gini importance plots (age, educational level, and income; Table S4 in Multimedia Appendix 1 ) and the top 3 features plus 15 judgment variables (Table S5 in Multimedia Appendix 1 ). The results did not outperform those presented in Table 5 .

For random forest, the Gini importance was highest for age and educational level ( Figure 4 ). These top 2 predictors had a combined effect of 16.5% to 16.8% for the 2 models ( Figures 4 A and 4C). Following these predictors, the 15 judgment variables and the income variable had similar Gini importance, with a combined effect of 69.4% to 75.5% for Gini importance. The least important predictors mirrored those of the BRF results.

Logistic Regression Model Statistics

Both model 1 (demographic and judgment variables) and model 2 (demographic, judgment, and COVID-19 precaution behavior variables) were significant ( P <.001). The model statistics are provided in Tables 6 (model 1) and 7 (model 2). In model 1, age, income, marital status, employment status, sex, educational level, ante, aversion tipping point, reward-aversion consistency, and consistency range were significant ( α <.05). In model 2, age, income, marital status, employment status, sex, educational level, risk aversion, ante, peak negative risk, mask wearing, and not gathering in large groups were significant ( α <.05).

a Overall model: P <.001; pseudo- R 2 =0.149; log-likelihood=−1736.8; log-likelihood null=−2039.7.

a Overall model: P <.001; pseudo- R 2 =0.206; log-likelihood=−1620.0; log-likelihood null=−2039.7.

Because judgment variables and demographic variables (age, income, and educational level) were important predictors, we evaluated post hoc whether demographics statistically mediated or moderated the relationship between each of the 15 judgment variables and binary responses to COVID-19 vaccination.

For primary mediations, age significantly mediated the statistical relationship between 11 judgment variables and vaccine uptake ( α <.05; Table 8 ), income mediated 8 relationships α < <.05; Table 8 ), and educational level mediated 9 relationships ( α <.05; Table 8 ). In total, 7 judgment variables overlapped across the 3 models: loss resilience, ante, insurance, peak positive risk, peak negative risk, risk aversion trade-off, and consistency range. Of these, 5 significantly differed between vaccine uptake (those fully vaccinated and those not): loss resilience, ante, insurance, peak positive risk, and peak negative risk ( Table 3 ). Thus, 2 judgment features did not differ by vaccine uptake but were connected with uptake by significant mediation.

For the secondary mediation analyses, 5 judgment variables mediated the statistical relationship between age and vaccine uptake ; these variables overlapped with the 11 findings of the primary mediation analyses. Furthermore, 4 judgment variables mediated the statistical relationship between income and vaccine uptake ; these variables overlapped with the 8 findings of the primary mediation analyses. Finally, 4 judgment variables mediated the statistical relationship between educational level and vaccine uptake ; these variables overlapped with the 9 findings of the primary mediation analyses. In all secondary analyses, approximately half of the judgment variables were involved in mediation as compared to the doubling of judgment variable numbers observed in the primary mediation analyses. In the secondary mediation analyses, the same 4 judgment variables were found in both primary and secondary mediation results, indicating a mixed mediation framework.

From the moderation analyses, only 2 interactions out of a potential 45 were observed. Age interacted with risk aversion trade-off, and income interacted with loss resilience to statistically predict vaccine uptake ( α <.05; Table 8 ). The 2 moderation results overlapped with the mediation results, indicating mixed mediation-moderation relationships [ 78 , 80 , 81 ].

Principal Findings

Relatively few studies have sought to predict COVID-19 vaccine uptake using machine learning approaches [ 8 , 59 ]. Given that a small set of studies has assessed the psychological basis that may underlie vaccine uptake and choices [ 6 , 52 , 53 , 56 , 58 , 59 , 83 ], but none have used computational cognition variables based on reward and aversion judgment to predict vaccine uptake , we sought to assess whether variables quantifying human judgment predicted vaccine uptake . This study found that 7 demographic and 15 judgment variables predicted vaccine uptake with balanced and moderate recall and specificity, moderate accuracy, high AUROC, and high precision using a BRF framework. Other machine learning approaches (random forest and logistic regression) produced higher accuracies but lower specificities, indicating a lower prediction of those who did not receive the vaccine. The BRF also had challenges predicting the negative class, as demonstrated by the relatively low NPV despite having higher specificity than random forest and logistic regression. Feature importance analyses from both BRF and random forest showed that the judgment variables collectively dominated the Gini importance scores. Furthermore, demographic variables acted as statistical mediators in the relationship between judgment variables and vaccine uptake . These mediation findings support the interpretation of the machine learning results that demographic factors, together with judgment variables, predict COVID-19 vaccine uptake .

Interpretation of Judgment Differences Between Vaccinated and Nonvaccinated Individuals

Those who were fully vaccinated had lower values for loss aversion, ante, peak positive risk, peak negative risk, total reward risk, and total aversion risk, along with higher values for risk aversion, loss resilience, insurance, and trade-off range (refer to Table 1 for variable descriptions). Lower loss aversion corresponds to less overweighting of bad outcomes relative to good ones [ 84 ] and a potential willingness to obtain a vaccine with uncertain outcomes. A lower ante suggests that individuals are less willing to engage in risky behaviors surrounding potential infection, which is also consistent with the 4 other judgment variables that define relationships between risk and value (peak positive risk, peak negative risk, total reward risk, and total aversion risk). In participants who indicated full vaccination, lower peak positive risk and peak negative risk were related to individuals having a lower risk that they must overcome to make a choice to either approach or avoid, as per the decision utility equation by Markowitz [ 39 , 71 ]. The lower total reward risk and total aversion risk indicate that the interactions between reward, aversion, and the risks associated with them did not scale significantly; namely, higher reward was not associated with higher risk, and higher negative outcomes were not associated with the uncertainty of them. For these participants, the ability of the vaccine to increase the probability of health and reduce the probability of harm from illness did not have to overcome high obstacles in their vaccine choice. Higher risk aversion in vaccinated participants suggests that these participants viewed contracting COVID-19 as a larger risk and, therefore, were more likely to receive the full dose. These findings are consistent with those of a study by Lepinteur et al [ 58 ], who found that risk-averse individuals were more likely to accept the COVID-19 vaccination, indicating that the perceived risk of contracting COVID-19 was greater than any risk from the vaccine. Hudson and Montelpare [ 54 ] also found that risk aversion may promote vaccine adherence when people perceive contracting a disease as more dangerous or likely. Higher loss resilience in the vaccinated group was also consistent with the perspective that vaccination would improve their resilience and act as a form of insurance against negative consequences. The higher trade-off range suggests that vaccinated individuals have a broader portfolio of preferences and are more adaptive to bad things occurring, whereas a lower trade-off indicates a restriction in preferences and less adaptability in those who did not receive the vaccine.

Comparison of Prediction Algorithms

When testing these judgment variables (with demographic and COVID-19 precaution behavior variables) in a BRF framework to predict vaccine uptake , we observed a high AUROC of 0.79, where an AUROC of 0.8 is often the threshold for excellent model performance in machine learning [ 85 , 86 ]. The similarity of our reported recall and specificity values with the BRF suggests a balance between predicting true positives and true negatives. The high precision indicates a high certainty in predicting those who were fully vaccinated. The BRF model was successful in identifying those who received the full vaccine (positive cases; indicated by high precision and moderate recall) and those who did not (negative cases; indicated by the specificity). However, NPV was low, indicating a higher rate of false prediction of those who did not receive a full dose counterbalanced by a higher specificity that reflects a higher rate of predicting true negatives. These observations are reflected in the moderate accuracy, which measures the number of correct predictions. A comparison of random forest, logistic regression, and BRF revealed that random forest and logistic regression models produced less balance between recall (high) and specificity (low), which could be interpreted as a bias toward predicting the majority class (ie, those who received the vaccine). That being said, the NPV for BRF was lower than that for random forest and logistic regression, where a low NPV indicates a low probability that those predicted to have not received the vaccine truly did not receive the vaccine when taking both classes into account. Together, the results from all 3 machine learning approaches reveal challenges in predicting the negative class (ie, those who did not receive the vaccine). Overall, the 3 models achieved high accuracy, recall, precision, and AUROC. BRF produced a greater balance between recall and specificity, and the outcome of the worst-performing metric (ie, NPV) was still higher than the specificities for the random forest and logistic regression models.

Feature Importance

Of the 3 prediction algorithms, random forest and BRF had very similar Gini importance results, whereas logistic regression elevated most demographic variables and a minority of judgment variables. This observation could be due to the large variance in each of the judgment variables, which could present challenges for achieving a good fit with logistic regression. In contrast, the demographic and COVID-19 precaution variables had low variance and could be more easily fit in a linear model, hence their significance in the logistic regression results. In comparison to logistic regression, decision trees (eg, BRF and random forest) use variable variance as additional information to optimize classification, potentially leading to a higher importance of judgment variables over most demographic and all COVID-19 precaution variables.

Focusing on the model with balanced recall and specificity (ie, the BRF classifiers [with and without COVID-19 precaution behaviors]), the top predictors were 3 demographic variables (age, income, and educational level), with distributions that varied by vaccine uptake in manners consistent with those of other reports. Namely, older individuals, those identifying as male and White individuals, and those who indicated a higher income and educational level corresponded to those who were or intended to be vaccinated [ 2 , 5 , 87 ]. Despite their saliency, these 3 variables together only contributed 23% to the prediction, corresponding to approximately one-third of the contribution from the 15 judgment variables (63%-69%). The individual Gini importance scores for the 15 judgment variables only ranged from 0.039 to 0.049 but were the dominant set of features behind the moderate accuracy, high precision, and high AUROC. The 18% difference between the accuracy and precision measures suggests that variables other than those used in this study may improve prediction, including contextual variables that may influence vaccine choices. Variables may include political affiliation [ 7 ], longitude and latitude [ 8 ], access to the internet [ 8 ], health literacy [ 54 ], and presence of underlying conditions [ 9 ]. Future work should seek to include these types of variables.

In the second BRF classifier, the 4 COVID-19 precaution behaviors only contributed 6.6% to the prediction. This low contribution could be due to these variables being binary, unlike the other demographic variables, which included a range of categories. In addition, COVID-19 precaution behaviors are specific to the context of the COVID-19 pandemic and do not promote interpretation beyond their specific context. The 15 judgment variables represent a contrast to this as they are empirically computed from a set of functions across many picture categories. An individual with higher risk aversion will generally tolerate higher amounts of uncertainty regarding a potential upside or gain as opposed to settling for what they have. This does not depend on what stimulus category they observe or the stimulus-response condition. Instead, it is a general feature of the bounds to their judgment and is part of what behavioral economists such as Kahneman consider as bounds to human rationality [ 84 ].

Mechanistic Relationships Between Judgment and Demographic Variables

The Gini score plots were clear sigmoid-like graphs ( Figure 3 ), with only 3 of the 7 demographic variables ranking above the judgment variables. This observation was consistent in both BRF classifiers (with and without COVID-19 precaution behaviors), raising the possibility of a statistically mechanistic relationship among the top 3 demographic variables, the 15 judgment variables, and vaccine uptake . Indeed, we observed 28 primary mediation effects and 13 secondary mediation effects in contrast to 2 moderation relationships, which also happened to overlap with mediation findings, suggesting mixed mediation-moderation relationships [ 81 , 88 ]. The observation that most judgment variables were significant in mediation relationships but not in moderation relationships argues that prediction depended on the directional relationship between judgment and demographic variables to predict vaccine uptake . Furthermore, there were more significant primary mediations (when judgment variables were the IVs) compared to secondary mediations, suggesting the importance of judgment variables as IVs and demographic variables as mediators. Mathematically, judgment variables (IVs) influenced vaccine uptake (DV), and this relationship was stronger when demographic variables were added to the equation. The 13 secondary mediations all overlapped with the 28 primary mediations, where demographic variables were IVs and judgment variables were mediators, suggesting that demographic variables influenced vaccine uptake (DV) and that this relationship became stronger with the addition of judgment variables. This overlap of primary and secondary mediations for 4 of the judgment variables suggests that both judgment and demographic variables influenced the choice of being vaccinated within a mixed mediation framework because adding either one of them to the mediation model regressions made the relationships stronger [ 49 ]. The lack of moderation results and a considerable number of overlapping primary and secondary mediation results imply that the relationship between judgment variables and vaccine uptake did not depend purely on their interaction with age, income, or educational level (ie, moderation) but, instead, depended on the direct effects of these 3 demographic variables to strengthen the relationship between judgment variables and vaccine uptake . This type of analysis of statistical mechanisms is helpful for understanding contextual effects on our biases and might be important for considering how best to target or message those with higher loss aversion, ante, peak positive risk, peak negative risk, total reward risk, and total aversion risk (ie, in those who were not fully vaccinated).

Model Utility

The developed model is automatable and may have applications in public health. The picture-rating task can be deployed on any smart device or computer, making it accessible to much of the US population or regional populations. The ratings from this task can be automatically processed, and the results can be stored in local or national databases. This method of data collection is novel in that persons cannot bias their responses as the rating task has no perceivable relation to vaccination choices. Government and public health bodies can access these data to determine predicted vaccine uptake rates locally or nationally, which can be used to (1) prepare vaccine rollouts and supply chain demand, (2) prepare health care institutions in areas that may experience low vaccine adherence and potentially higher infection rates, and (3) determine which areas may need more targeted messaging to appeal to specific judgment profiles. For use case 3, messaging about infection risks or precaution behaviors could be framed to address those with lower risk aversion, who, in this study, tended to forgo vaccination. Given that such individualized data would not be available a priori, it would be more plausible to collect data from similarly sized cohorts in geographic regions of concern to obtain regional judgment behavior profiles and, thus, target use cases 1 to 3. Further development of this model with different population samples might also improve our understanding of how certain judgment variables may be targeted with different types of messaging, offering a means to potentially improve vaccine uptake . This model might also be applied to other mandated or recommended vaccines such as those for influenza or human papillomavirus, ultimately improving preparation and messaging efforts. However, future work would be needed to model these varying vaccine choices.

Given the use of demographic variables in the proposed model, specific demographic populations could be assessed or considered for messaging. If particular demographic groups are predicted to have a low vaccine uptake rate, messaging can be targeted to those specific groups. For example, we observed that a higher percentage of female individuals were not fully vaccinated when compared to male individuals. This could be related to concerns about the COVID-19 vaccine affecting fertility or pregnancy. To improve uptake in this population, scientifically backed messaging could be used to confirm the safety of the vaccine in this context. Lower rates of vaccination have been reported in Black communities, which was also observed in this study. Researchers have identified targetable issues related to this observation, which include engagement of Black faith leaders and accessibility of vaccination clinics in Black communities, to name a few [ 89 ].

In summary, this model could be used to predict vaccine uptake at the local and national levels and further assess the demographic and judgment features that may underlie these choices.

Limitations

This study has a number of limitations that should be considered. First, there are the inherent limitations of using an internet survey—namely, the uncontrolled environment in which participants provide responses. Gold Research, Inc, and the research team applied stringent exclusion criteria, including the evaluation of the judgment graphs given that random responses produce graphs with extremely low R 2 fits (eg, <0.1). This was not the case in our cohort of 3476 participants, but this cannot perfectly exclude random or erroneous responses to other questionnaire components. Second, participants with mental health conditions were oversampled to meet the criteria for other survey components not discussed in this paper. This oversampling could potentially bias the results, and future work should use a general population sample to verify these findings. Third, demographic variability and the resulting confounds are inherent in population surveys, and other demographic factors not collected in this study may be important for prediction (eg, religion and family size). Future work might consider collecting a broader array of demographic factors to investigate and include in predictive modeling. Fourth, we used a limited set of 7 demographic variables and 15 judgment variables; however, a larger set of judgment variables is potentially computable and could be considered for future studies. There is also little information on how post–COVID-19 effects, including socioeconomic effects, affect COVID-19 vaccination choices.

Conclusions

To our knowledge, there has been minimal research on how biases in human judgment might contribute to the psychology underlying individual vaccination preferences and what differentiates individuals who were fully vaccinated against COVID-19 from those who were not. This population study of several thousand participants demonstrated that a small set of demographic variables and 15 judgment variables predicted vaccine uptake with moderate to high accuracy and high precision and AUROC, although a large range of specificities was achieved depending on the classification method used. In an age of big data machine learning approaches, this study provides an option for using fewer but more interpretable variables. Age, income, and educational level were independently the most important predictors of vaccine uptake , but judgment variables collectively dominated the importance rankings and contributed almost two-thirds to the prediction of COVID-19 vaccination for the BRF and random forest models. Age, income, and educational level significantly mediated the statistical relationship between judgment variables and vaccine uptake , indicating a statistically mechanistic relationship grounding the prediction results. These findings support the hypothesis that small sets of judgment variables might provide a target for vaccine education and messaging to improve uptake. Such education and messaging might also need to consider contextual variables (ie, age, income, and educational level) that mediate the effect of judgment variables on vaccine uptake . Judgment and demographic variables can be readily collected using any digital device, including smartphones, which are accessible worldwide. Further development and use of this model could (1) improve vaccine uptake , (2) better prepare vaccine rollouts and health care institutions, (3) improve messaging efforts, and (4) have applications for other mandated or government-recommended vaccines.

Acknowledgments

The authors thank Carol Ross, Angela Braggs-Brown, Tom Talavage, Eric Nauman, and Marc Cahay at the University of Cincinnati (UC) College of Engineering and Applied Sciences, who significantly impacted the transfer of research funding to UC. Funding for this work was provided in part to HCB by the Office of Naval Research (awards N00014-21-1-2216 and N00014-23-1-2396) and to HCB from a Jim Goetz donation to the UC College of Engineering and Applied Sciences. Finally, the authors thank the anonymous reviewers for their constructive input, which substantially improved the manuscript. The opinions expressed in this paper are those of the authors and are not necessarily representative of those of their respective institutions.

Data Availability

The data set and corresponding key used in this study are available in Multimedia Appendix 2 .

Conflicts of Interest

A provisional patent has been submitted by the following authors (NLV, SB, HCB, SL, LS, and AKK): “Methods of predicting vaccine uptake,” provisional application # 63/449,460.

Supplementary material.

• Ezati Rad R, Kahnouji K, Mohseni S, Shahabi N, Noruziyan F, Farshidi H, et al. Predicting the COVID-19 vaccine receive intention based on the theory of reasoned action in the south of Iran. BMC Public Health. Feb 04, 2022;22(1):229. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mewhirter J, Sagir M, Sanders R. Towards a predictive model of COVID-19 vaccine hesitancy among American adults. Vaccine. Mar 15, 2022;40(12):1783-1789. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Romate J, Rajkumar E, Greeshma R. Using the integrative model of behavioural prediction to understand COVID-19 vaccine hesitancy behaviour. Sci Rep. Jun 04, 2022;12(1):9344. [ CrossRef ] [ Medline ]
• Kalam MA, Davis TPJ, Shano S, Uddin MN, Islam MA, Kanwagi R, et al. Exploring the behavioral determinants of COVID-19 vaccine acceptance among an urban population in Bangladesh: implications for behavior change interventions. PLoS One. 2021;16(8):e0256496. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Koesnoe S, Siddiq TH, Pelupessy DC, Yunihastuti E, Awanis GS, Widhani A, et al. Using integrative behavior model to predict COVID-19 vaccination intention among health care workers in Indonesia: a nationwide survey. Vaccines (Basel). May 04, 2022;10(5):719. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hudson A, Hall PA, Hitchman SC, Meng G, Fong GT. Cognitive predictors of COVID-19 mitigation behaviors in vaccinated and unvaccinated general population members. Vaccine. Jun 19, 2023;41(27):4019-4026. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dhanani LY, Franz B. A meta-analysis of COVID-19 vaccine attitudes and demographic characteristics in the United States. Public Health. Jun 2022;207:31-38. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cheong Q, Au-Yeung M, Quon S, Concepcion K, Kong JD. Predictive modeling of vaccination uptake in US counties: a machine learning-based approach. J Med Internet Res. Nov 25, 2021;23(11):e33231. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bulusu A, Segarra C, Khayat L. Analysis of COVID-19 vaccine uptake among people with underlying chronic conditions in 2022: a cross-sectional study. SSM Popul Health. Jun 2023;22:101422. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Painter EM, Ussery EN, Patel A, Hughes MM, Zell ER, Moulia DL, et al. Demographic characteristics of persons vaccinated during the first month of the COVID-19 vaccination program - United States, December 14, 2020-January 14, 2021. MMWR Morb Mortal Wkly Rep. Feb 05, 2021;70(5):174-177. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Azcona EA, Kim BW, Vike NL, Bari S, Lalvani S, Stefanopoulos L, et al. Discrete, recurrent, and scalable patterns in human judgement underlie affective picture ratings. arXiv. Preprint posted online March 12, 2022. 2022.:2203.06448. [ FREE Full text ] [ CrossRef ]
• Breiter HC, Block M, Blood AJ, Calder B, Chamberlain L, Lee N, et al. Redefining neuromarketing as an integrated science of influence. Front Hum Neurosci. Feb 12, 2014;8:1073. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schneirla TC. An evolutionary and developmental theory of biphasic processes underlying approach and withdrawal. In: Jones MR, editor. Nebraska Symposium on Motivation. Lincoln, Nebraska. University of Nebraska Press; 1959;1-42.
• Schneirla TC. Aspects of stimulation and organization in approach/withdrawal processes underlying vertebrate behavioral development. In: Advances in the Study of Behavior. Cambridge, MA. Academic Press; 1965;1-74.
• Lewin K. Psycho-sociological problems of a minority group. J Personality. Mar 1935;3(3):175-187. [ CrossRef ]
• Herrnstein RJ. Secondary reinforcement and rate of primary reinforcement. J Exp Anal Behav. Jan 1964;7(1):27-36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baum WM. On two types of deviation from the matching law: bias and undermatching. J Exp Anal Behav. Jul 1974;22(1):231-242. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim BW, Kennedy DN, Lehár J, Lee MJ, Blood AJ, Lee S, et al. Recurrent, robust and scalable patterns underlie human approach and avoidance. PLoS One. May 26, 2010;5(5):e10613. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Berridge KC, Robinson TE. The mind of an addicted brain: neural sensitization of wanting versus liking. Curr Dir Psychol Sci. 1995;4(3):71-76. [ FREE Full text ] [ CrossRef ]
• Montague PR. Free will. Curr Biol. Jul 22, 2008;18(14):R584-R585. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dai X, Brendl CM, Ariely D. Wanting, liking, and preference construction. Emotion. Jun 2010;10(3):324-334. [ CrossRef ] [ Medline ]
• Mas-Colell A, Whinston MD, Green JR. Microeconomic Theory. Oxford, UK. Oxford University Press; 1995.
• Marshall A. Principles of Economics. Buffalo, NY. Prometheus Books; 1997.
• Aharon I, Etcoff N, Ariely D, Chabris CF, O'Connor E, Breiter HC. Beautiful faces have variable reward value: fMRI and behavioral evidence. Neuron. Nov 08, 2001;32(3):537-551. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Viswanathan V, Lee S, Gilman JM, Kim BW, Lee N, Chamberlain L, et al. Age-related striatal BOLD changes without changes in behavioral loss aversion. Front Hum Neurosci. Apr 30, 2015;9:176. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Perlis RH, Holt DJ, Smoller JW, Blood AJ, Lee S, Kim BW, et al. Association of a polymorphism near CREB1 with differential aversion processing in the insula of healthy participants. Arch Gen Psychiatry. Aug 2008;65(8):882-892. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gasic GP, Smoller JW, Perlis RH, Sun M, Lee S, Kim BW, et al. BDNF, relative preference, and reward circuitry responses to emotional communication. Am J Med Genet B Neuropsychiatr Genet. Sep 05, 2009;150B(6):762-781. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Makris N, Oscar-Berman M, Jaffin SK, Hodge SM, Kennedy DN, Caviness VS, et al. Decreased volume of the brain reward system in alcoholism. Biol Psychiatry. Aug 01, 2008;64(3):192-202. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Elman I, Ariely D, Mazar N, Aharon I, Lasko NB, Macklin ML, et al. Probing reward function in post-traumatic stress disorder with beautiful facial images. Psychiatry Res. Jun 30, 2005;135(3):179-183. [ CrossRef ] [ Medline ]
• Strauss MM, Makris N, Aharon I, Vangel MG, Goodman J, Kennedy DN, et al. fMRI of sensitization to angry faces. Neuroimage. Jun 18, 2005;26(2):389-413. [ CrossRef ] [ Medline ]
• Levy B, Ariely D, Mazar N, Chi W, Lukas S, Elman I. Gender differences in the motivational processing of facial beauty. Learn Motiv. May 2008;39(2):10.1016/j.lmot.2007.09.002. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yamamoto R, Ariely D, Chi W, Langleben DD, Elman I. Gender differences in the motivational processing of babies are determined by their facial attractiveness. PLoS One. Jun 24, 2009;4(6):e6042. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Viswanathan V, Sheppard JP, Kim BW, Plantz CL, Ying H, Lee MJ, et al. A quantitative relationship between signal detection in attention and approach/avoidance behavior. Front Psychol. Feb 21, 2017;8:122. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stefanopoulos L, Lalvani S, Kim BW, Vike NL, Bari S, Emanuel A, et al. Predicting depression history from a short reward/aversion task with behavioral economic features. In: Proceedings of the International Conference on Biomedical and Health Informatics 2022. 2022. Presented at: ICBHI 2022; November 24-26, 2022; Concepcion, Chile.
• Shannon CE, Weaver W. The Mathematical Theory of Communication. Urbana, IL. The University of Illinois Press; 1949.
• Livengood SL, Sheppard JP, Kim BW, Malthouse EC, Bourne JE, Barlow AE, et al. Keypress-based musical preference is both individual and lawful. Front Neurosci. May 02, 2017;11:136. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Feynman R, Wilczek F. The Character of Physical Law. New York, NY. Modern Library; 1965.
• Kahneman D, Tversky A. Prospect theory: an analysis of decision under risk. Econometrica. Mar 1979;47(2):263-292. [ CrossRef ]
• Markowitz H. Portfolio selection*. J Finance. Apr 30, 2012;7(1):77-91. [ CrossRef ]
• Nagaya K. Why and under what conditions does loss aversion emerge? Jpn Psychol Res. Oct 15, 2021;65(4):379-398. [ CrossRef ]
• Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science. Sep 27, 1974;185(4157):1124-1131. [ CrossRef ] [ Medline ]
• Macoveanu J, Rowe JB, Hornboll B, Elliott R, Paulson OB, Knudsen GM, et al. Serotonin 2A receptors contribute to the regulation of risk-averse decisions. Neuroimage. Dec 2013;83:35-44. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Macoveanu J, Rowe JB, Hornboll B, Elliott R, Paulson OB, Knudsen GM, et al. Playing it safe but losing anyway--serotonergic signaling of negative outcomes in dorsomedial prefrontal cortex in the context of risk-aversion. Eur Neuropsychopharmacol. Aug 2013;23(8):919-930. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schultz W. Dopamine signals for reward value and risk: basic and recent data. Behav Brain Funct. Apr 23, 2010;6:24. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Manssuer L, Ding Q, Zhang Y, Gong H, Liu W, Yang R, et al. Risk and aversion coding in human habenula high gamma activity. Brain. Jun 01, 2023;146(6):2642-2653. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thrailkill EA, DeSarno M, Higgins ST. Intersections between environmental reward availability, loss aversion, and delay discounting as potential risk factors for cigarette smoking and other substance use. Prev Med. Dec 2022;165(Pt B):107270. [ CrossRef ] [ Medline ]
• Vrijen C, Hartman CA, Oldehinkel AJ. Reward-related attentional bias at age 16 predicts onset of depression during 9 years of follow-up. J Am Acad Child Adolesc Psychiatry. Mar 2019;58(3):329-338. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hansen A, Turpyn CC, Mauro K, Thompson JC, Chaplin TM. Adolescent brain response to reward is associated with a bias toward immediate reward. Dev Neuropsychol. Aug 2019;44(5):417-428. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morales S, Miller NV, Troller-Renfree SV, White LK, Degnan KA, Henderson HA, et al. Attention bias to reward predicts behavioral problems and moderates early risk to externalizing and attention problems. Dev Psychopathol. May 2020;32(2):397-409. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang S, Krajbich I, Adolphs R, Tsuchiya N. The role of risk aversion in non-conscious decision making. Front Psychol. 2012;3:50. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen X, Voets S, Jenkinson N, Galea JM. Dopamine-dependent loss aversion during effort-based decision-making. J Neurosci. Jan 15, 2020;40(3):661-670. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Trueblood JS, Sussman AB, O’Leary D. The role of risk preferences in responses to messaging about COVID-19 vaccine take-up. Soc Psychol Personal Sci. Mar 11, 2021;13(1):311-319. [ CrossRef ]
• Wagner CE, Prentice JA, Saad-Roy CM, Yang L, Grenfell BT, Levin SA, et al. Economic and behavioral influencers of vaccination and antimicrobial use. Front Public Health. Dec 21, 2020;8:614113. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hudson A, Montelpare WJ. Predictors of vaccine hesitancy: implications for COVID-19 public health messaging. Int J Environ Res Public Health. Jul 29, 2021;18(15):8054. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Horne Z, Powell D, Hummel JE, Holyoak KJ. Countering antivaccination attitudes. Proc Natl Acad Sci U S A. Aug 18, 2015;112(33):10321-10324. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Benin AL, Wisler-Scher DJ, Colson E, Shapiro ED, Holmboe ES. Qualitative analysis of mothers' decision-making about vaccines for infants: the importance of trust. Pediatrics. May 2006;117(5):1532-1541. [ CrossRef ] [ Medline ]
• Noyman-Veksler G, Greenberg D, Grotto I, Shahar G. Parents' malevolent personification of mass vaccination solidifies vaccine hesitancy. J Health Psychol. Oct 2021;26(12):2164-2172. [ CrossRef ] [ Medline ]
• Lepinteur A, Borga LG, Clark AE, Vögele C, D'Ambrosio C. Risk aversion and COVID-19 vaccine hesitancy. Health Econ. Aug 2023;32(8):1659-1669. [ CrossRef ] [ Medline ]
• Becchetti L, Candio P, Salustri F. Vaccine uptake and constrained decision making: the case of Covid-19. Soc Sci Med. Nov 2021;289:114410. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fernández A, García S, Galar M, Prati RC, Krawczyk B, Herrera F. Learning from Imbalanced Data Sets. Cham, Switzerland. Springer International Publishing; 2018.
• Bari S, Vike NL, Stetsiv K, Woodward S, Lalvani S, Stefanopoulos L, et al. The prevalence of psychotic symptoms, violent ideation, and disruptive behavior in a population with SARS-CoV-2 infection: preliminary study. JMIR Form Res. Aug 16, 2022;6(8):e36444. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Woodward SF, Bari S, Vike N, Lalvani S, Stetsiv K, Kim BW, et al. Anxiety, post-COVID-19 syndrome-related depression, and suicidal thoughts and behaviors in COVID-19 survivors: cross-sectional study. JMIR Form Res. Oct 25, 2022;6(10):e36656. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vike NL, Bari S, Stetsiv K, Woodward S, Lalvani S, Stefanopoulos L, et al. The Relationship Between a History of High-risk and Destructive Behaviors and COVID-19 Infection: Preliminary Study. JMIR Form Res. 2023;7:e40821. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lee S, Lee MJ, Kim BW, Gilman JM, Kuster JK, Blood AJ, et al. The commonality of loss aversion across procedures and stimuli. PLoS One. Sep 22, 2015;10(9):e0135216. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tversky A, Kahneman D. Advances in prospect theory: cumulative representation of uncertainty. J Risk Uncertainty. Oct 1992;5(4):297-323. [ CrossRef ]
• Sheppard JP, Livengood SL, Kim BW, Lee MJ, Blood AJ. Connecting prospect and portfolio theories through relative preference behavior. In: Proceedings of the 14th Annual Meeting on Society for NeuroEconomics. 2016. Presented at: SNE '16; August 28-30, 2016;102-103; Berlin, Germany. URL: https://staging.neuroeconomics.org/wp-content/uploads/2016/08/AbstractBookSNE2016.pdf
• Zhang R, Brennan TJ, Lo AW. The origin of risk aversion. Proc Natl Acad Sci U S A. Dec 16, 2014;111(50):17777-17782. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Iacobucci D. Mediation analysis and categorical variables: the final frontier. J Consum Psychol. Apr 12, 2012;22(4):582-594. [ CrossRef ]
• Lang PJ, Bradley MM, Cuthbert MM. Motivated attention: affect, activation and action. In: Lang PJ, Simons RF, Balaban MT, editors. Attention and Orienting: Sensory and Motivational Processes. Hillsdale, NJ. Lawrence Erlbaum Associates; 1997;97-136.
• Lang PJ, Bradley MM, Cuthbert BN. International affective picture system (IAPS): affective ratings of pictures and instruction manual. Technical Report A-8. University of Florida. 2008. URL: https://www.scirp.org/reference/referencespapers?referenceid=755311 [accessed 2021-06-01]
• Markowitz H. The Utility of Wealth. J Political Econ. Apr 1952;60(2):151-158. [ CrossRef ]
• StataCorp. Stata statistical software: release 17. StataCorp LLC. College Station, TX. StataCorp LLC; 2021. URL: https://www.stata.com/company/ [accessed 2024-02-23]
• Van Rossum G, Drake FL. Python 3 Reference Manual. Scotts Valley, CA. CreateSpace Independent Publishing Platform; 2009.
• Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in python. J Mach Learn Res. 2011;12(85):2825-2830. [ FREE Full text ]
• Lemaître G, Nogueira F, Aridas CK. Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. J Mach Learn Res. 2017;18(17):1-5. [ FREE Full text ]
• R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. 2019. URL: https://www.r-project.org/ [accessed 2024-02-23]
• David A. Kenny's homepage. David A. Kenny. URL: http://davidakenny.net/cm/mediate.htm [accessed 2024-02-22]
• MacKinnon DP, Fairchild AJ, Fritz MS. Mediation analysis. Annu Rev Psychol. 2007;58:593-614. [ FREE Full text ] [ CrossRef ] [ Medline ]
• MacKinnon DP, Cheong J, Pirlott AG. Statistical mediation analysis. In: Cooper H, Camic PM, Long DL, Panter AT, Rindskopf D, Sher KJ, editors. APA Handbook of Research Methods in Psychology. Washington, DC. American Psychological Association; 2012;313-331.
• Mackinnon DP. Introduction to Statistical Mediation Analysis. New York, NY. Lawrence Erlbaum Associates; 2008.
• Baron RM, Kenny DA. The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986;51(6):1173-1182. [ CrossRef ]
• Hayes AF. Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach. New York, NY. Guilford Press; 2018.
• Charpentier CJ, Aylward J, Roiser JP, Robinson OJ. Enhanced risk aversion, but not loss aversion, in unmedicated pathological anxiety. Biol Psychiatry. Jun 15, 2017;81(12):1014-1022. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kahneman D, Tversky A. On the interpretation of intuitive probability: a reply to Jonathan Cohen. Cognition. Jan 1979;7(4):409-411. [ CrossRef ]
• Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. Sep 2010;5(9):1315-1316. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Watt J, Borhani R, Katsaggelos AK. Machine Learning Refined: Foundations, Algorithms, and Applications. Cambridge, UK. Cambridge University Press; 2016.
• Viswanath K, Bekalu M, Dhawan D, Pinnamaneni R, Lang J, McLoud R. Individual and social determinants of COVID-19 vaccine uptake. BMC Public Health. Apr 28, 2021;21(1):818. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Judd CM, Kenny DA, McClelland GH. Estimating and testing mediation and moderation in within-subject designs. Psychol Methods. Jun 2001;6(2):115-134. [ CrossRef ] [ Medline ]
• Abdul-Mutakabbir JC, Casey S, Jews V, King A, Simmons K, Hogue MD, et al. A three-tiered approach to address barriers to COVID-19 vaccine delivery in the Black community. Lancet Glob Health. Jun 2021;9(6):e749-e750. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 11.04.23; peer-reviewed by ME Visier Alfonso, L Lapp; comments to author 18.05.23; revised version received 08.08.23; accepted 10.01.24; published 18.03.24.

©Nicole L Vike, Sumra Bari, Leandros Stefanopoulos, Shamal Lalvani, Byoung Woo Kim, Nicos Maglaveras, Martin Block, Hans C Breiter, Aggelos K Katsaggelos. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 18.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.

• Get Benzinga Pro
• Data & APIs
• Our Services
• News Earnings Guidance Dividends M&A Buybacks Legal Interviews Management Offerings IPOs Insider Trades Biotech/FDA Politics Government Healthcare Sports
• Markets Pre-Market After Hours Movers ETFs Forex Cannabis Commodities Options Binary Options Bonds Futures CME Group Global Economics Previews Small-Cap Real Estate Cryptocurrency Penny Stocks Digital Securities Volatility
• Ratings Analyst Color Downgrades Upgrades Initiations Price Target
• Ideas Trade Ideas Covey Trade Ideas Long Ideas Short Ideas Technicals From The Press Jim Cramer Rumors Best Stocks & ETFs Best Penny Stocks Best S&P 500 ETFs Best Swing Trade Stocks Best Blue Chip Stocks Best High-Volume Penny Stocks Best Small Cap ETFs Best Stocks to Day Trade Best REITs
• Yield How to Buy Corporate Bonds How to Buy Treasury Bonds How to Invest in Real Estate Online
• Money Compare Online Brokers Stock Brokers Forex Brokers Futures Brokers Crypto Brokers Options Brokers ETF Brokers Mutual Fund Brokers Index Fund Brokers Bond Brokers Short Selling Brokers Stock Apps All Broker Reviews Insurance Auto Home Medicare Life Vision Dental Business Pet Health Motorcycle Renters Workers Comp Top Stocks Penny Stocks Stocks Under $5 Stocks Under $10 Stocks Under $20 Stocks Under $50 Stocks Under $100 Alternative Investing Invest in Art Invest in Watches Invest in Land Invest in Real Estate Invest in Wine Invest in Gold Mortgages Refinance Purchase Find a Mortgage Broker Consumer Moving Living
• Alts Alternative Investment Platforms REITs Versus Crowdfunding How to Invest in Artwork How to Invest in Jewelry Best Real Estate Crowdfunding Platforms Best Alternative Investments Best Alternative Investment Platforms
• Crypto Get Started Is Bitcoin a Good Investment? Is Ethereum a Good Investment? What is Blockchain Best Altcoins How to Buy Cryptocurrency? DeFi Crypto and DeFi 101 What is DeFi? Decentralized Exchanges Best DeFi Yield Farms Digital Securities NFTs NFT Release Calendar What is a Non-Fungible Token (NFT)? How to Buy Non-Fungible Tokens (NFTs) CryptoPunks Watchlist Are NFTs a Scam or a Digital Bubble? Best In Crypto Best Crypto Apps Best Crypto Portfolio Trackers Best Crypto Day Trading Strategies Best Crypto IRA Best Cryptocurrency Scanners Best Business Crypto Accounts Best Crypto Screeners
• Cannabis Cannabis Conference News Earnings Interviews Deals Regulations Psychedelics

Apple Quietly Unveils New Multimodal AI With A Staggering 30B Parameters: Could It Power Text And Vision Features On iPhones?

Zinger key points.

• Apple has quietly released a new research paper detailing a new multimodal AI that comes with a staggering 30 billion parameters.
• Apple researchers have combined text and image to power the results, saying this approach could deliver a more flexible AI system.

Apple Inc. AAPL researchers have made significant progress in the field of artificial intelligence (AI). The iPhone maker published its latest research detailing new techniques for training large language models (LLMs) using text and images, potentially leading to more powerful and flexible AI systems.

What Happened : The new methods, outlined in a research paper titled “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training,” were quietly released on arxiv.org. The paper details how combining various training data and model architectures can result in top-notch performance across various AI benchmarks.

The researchers found that a diverse dataset incorporating visual and linguistic information was crucial for the MM1 models to excel at tasks such as image captioning, visual question answering, and natural language inference.

See Also: Ahead Of Nvidia’s ‘Grand’ Event, This Analyst Compares AI Revolution To ’10 PM In A Party That Goes To 4:30 AM'

Interestingly, researchers found that the image’s resolution also played an important role in the quality of the AI model’s output. The higher the resolution, the better the quality of the data and, therefore, the output generated.

Apple has been intensifying its investments in AI to keep up with competitors like Alphabet Inc.'s GOOG GOOGL Google , Microsoft Corp. MSFT , and Amazon.com Inc. AMZN , which have been integrating generative AI capabilities into their products.

Subscribe to the  Benzinga Tech Trends newsletter  to get all the latest tech developments delivered to your inbox.

The company is reportedly on track to spend $1 billion annually on AI development.

However, Apple is  reportedly in talks with Google  to license its  Gemini  model to power generative AI features on iPhones.

Why It Matters : Apple’s recent AI advancements align with its ongoing efforts to enhance its AI capabilities.

The company has been quietly acquiring startups to bolster its AI division.

In February, CEO Tim Cook ‘s first mention of AI during the December quarter earnings call had Wedbush’s Dan Ives excited about the iPhone maker’s prospects going forward. Ives believes Apple is entering a “new growth cycle,” with AI and iPhone being its primary growth drivers.

Apple’s plan to bolster the integration of generative AI technology into iPhones is part of the company’s larger aim to improve its devices’ capabilities and spur innovation in the tech sector.

The company’s recent hardware innovations, such as the M3 Max processor for MacBooks and the S9 chip in Apple Watch , indicate its AI ambitions.

The iPhone 15 Pro’s A17 Pro chip features a neural engine, accelerating AI processes. Apple’s breakthrough in running LLMs on-device using Flash memory facilitates faster, offline data processing.

Price Action: Apple stock closed at $172.63 on Friday, up 0.01%, according to Benzinga Pro .

Check out more of Benzinga’s Consumer Tech coverage by  following this link .

Read Next:  Nvidia CEO Jensen Huang Thinks Even Free AI Chips From Rivals Are ‘Not Cheap Enough’ Compared To Its GPUs

Disclaimer : This content was partially produced with the help of  Benzinga Neuro  and was reviewed and published by Benzinga editors.

Photo via Shutterstock

© 2024 Benzinga.com. Benzinga does not provide investment advice. All rights reserved.

Trade confidently with insights and alerts from analyst ratings, free reports and breaking news that affects the stocks you care about.

Preorder "Paper Mario: The Thousand Year Door" for Nintendo Switch

Paper Mario is good again, finally.

Updated March 13, 2024

Recommendations are independently chosen by Reviewed's editors. Purchases made through the links below may earn us and our publishing partners a commission.

Paper Mario is a sometimes-polarizing series because older fans remember how amazing the first two forays were. When Paper Mario: The Thousand-Year Door landed on the Nintendo Gamecube in 2004, it was so beloved that fans wondered where the series could even go from there. What features would newer Paper Mario entries bring? The answer: sticker, paint, and origami mechanics nobody asked for.

This is the second Switch remake of a popular Mario RPG, following last November's release of Super Mario RPG. Nintendo is striking the Mario RPG iron while it's still hot, with its remake of what many Mario Maniacs (TM) consider to be the plumber's best RPG entry.

Whether you're a diehard fan of the game, or new to the series and you want to know what all the hubbub is about, here's everything you need to know to pre-order Paper Mario: The Thousand-Year Door .

The Mario RPG classic gets a fresh coat of HD paint.

What is Paper Mario: The Thousand-Year Door ?

Paper Mario: The Thousand-Year Door is a Nintendo Switch remake of a 2004 RPG adventure bearing the same name. It's the second game in the Paper Mario series, following 2001's Paper Mario on the N64.

It's a storybook RPG that mixes platforming and character elements from Mario's mainline series, complete with friendly versions of Mario enemies like goombas and koopa troopas.

In this adventure, Mario and company sail to Rogueport, a sleazy locale that's also the destination for a legendary treasure. Like any great treasure, countless hunters are looking to get their hands on it, from our hero to his archenemy Bowser (who is also occasionally playable), and a new enemy: the nefarious X-Nauts.

The Switch version of this game offers new HD graphics, complete with new features that will make the game a little more approachable for series newbies. Just what those features are remains to be seen, but we will update this episode once those features are revealed.

When does Paper Mario: The Thousand-Year Door release?

The portable Nintendo console with a big, bright, immersive OLED screen.

Related content

Prices were accurate at the time this article was published but may change over time.

Sign up for our newsletter.

Enter your email:

Thanks for signing up.