critical review vs literature review

We use cookies on this site to enhance your experience

By clicking any link on this page you are giving your consent for us to set cookies.

A link to reset your password has been sent to your email.

Back to login

We need additional information from you. Please complete your profile first before placing your order.

Thank you. payment completed., you will receive an email from us to confirm your registration, please click the link in the email to activate your account., there was error during payment, orcid profile found in public registry, download history, difference between a literature review and a critical review.

• Charlesworth Author Services
• 08 October, 2021

As you read research papers, you may notice that there are two very different kinds of review of prior studies. Sometimes, this section of a paper is called a literature review, and at other times, it is referred to as a critical review or a critical context . These differences may be more commonly seen across different fields. Although both these sections are about reviewing prior and existing studies, this article aims to clarify the differences between the two.

Literature review

A literature review is a summary of prior or existing studies that are related to your own research paper . A literature review can be a part of a research paper or can form a paper in itself . For the former, the literature review is designed as a basis upon which your own current study is designed and built. The latter forms a synthesis of prior studies and is a way to highlight future research agendas or a framework.

Writing a literature review

In a literature review, you should attempt to discuss the arguments and findings in prior studies and then work to build on these studies as you develop your own research. You can also highlight the connection between existing and prior literature to demonstrate how the current study you are presenting can advance your knowledge in the field .

When performing a literature review, you should aim to summarise your discussions using a specific aspect of the literature, such as by topic, time, methodology/ design and findings . By doing so, you should be able to establish an effective way to present the relevant literature and demonstrate the connection between prior studies and your research.

Do note that a literature review does not include a presentation or discussion of any results or findings – this should come at a later point in the paper or study. You should also not impose your subjective viewpoints or opinions on the literature you discuss. 

Critical review

A critical review is also a popular way of reviewing prior and existing studies. It can cover and discuss the main ideas or arguments in a book or an article, or it can review a specific concept, theme, theoretical perspective or key construct found in the existing literature .

However, the key feature that distinguishes a critical review from a literature review is that the former is more than just a summary of different topics or methodologies. It offers more of a reflection and critique of the concept in question, and is engaged by authors to more clearly contextualise their own research within the existing literature and to present their opinions, perspectives and approaches .

Given that a critical review is not just a summary of prior literature, it is generally not considered acceptable to follow the same strategy as for a literature review. Instead, aim to organise and structure your critical review in a way that would enable you to discuss the key concepts, assert your perspectives and locate your arguments and research within the existing body of work. 

Structuring a critical review

A critical review would generally begin with an introduction to the concepts you would like to discuss. Depending on how broad the topics are, this can simply be a brief overview or it could set up a more complex framework. The discussion that follows through the rest of the review will then address and discuss your chosen themes or topics in more depth. 

Writing a critical review

The discussion within a critical review will not only present and summarise themes but also critically engage with the varying arguments, writings and perspectives within those themes. One important thing to note is that, similar to a literature review , you should keep your personal opinions, likes and dislikes out of a review. Whether you personally agree with a study or argument – and whether you like it or not – is immaterial. Instead, you should focus upon the effectiveness and relevance of the arguments , considering such elements as the evidence provided, the interpretations and analysis of the data, whether or not a study may be biased in any way, what further questions or problems it raises or what outstanding gaps and issues need to be addressed.

In conclusion

Although a review of previous and existing literature can be performed and presented in different ways, in essence, any literature or critical review requires a solid understanding of the most prominent work in the field as it relates to your own study. Such an understanding is crucial and significant for you to build upon and synthesise the existing knowledge, and to create and contribute new knowledge to advance the field .

Read previous (fourth) in series: How to refer to other studies or literature in the different sections of a research paper

Maximise your publication success with Charlesworth Author Services .

Charlesworth Author Services, a trusted brand supporting the world’s leading academic publishers, institutions and authors since 1928. 

To know more about our services, visit:  Our Services

Share with your colleagues

Related articles.

Conducting a Literature Review

Charlesworth Author Services 10/03/2021 00:00:00

Important factors to consider as you Start to Plan your Literature Review

Charlesworth Author Services 06/10/2021 00:00:00

How to Structure and Write your Literature Review

Charlesworth Author Services 07/10/2021 00:00:00

Related webinars

Bitesize Webinar: How to write and structure your academic article for publication- Module 3: Understand the structure of an academic paper

Charlesworth Author Services 04/03/2021 00:00:00

Bitesize Webinar: How to write and structure your academic article for publication: Module 5: Conduct a Literature Review

Bitesize Webinar: How to write and structure your academic article for publication: Module 7: Write a strong theoretical framework section

Charlesworth Author Services 05/03/2021 00:00:00

Bitesize Webinar: How to write and structure your academic article for publication: Module 11: Know when your article is ready for submission

Literature search.

Why and How to do a literature search

Charlesworth Author Services 17/08/2020 00:00:00

Best tips to do a PubMed search

Charlesworth Author Services 26/08/2021 00:00:00

Best tips to do a Scopus search

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

1 Citations

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Similar content being viewed by others

Reviewing Literature for and as Research

Discussion and Conclusion

Systematic Reviews in Educational Research: Methodology, Perspectives and Application

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

How to Write Critical Reviews

When you are asked to write a critical review of a book or article, you will need to identify, summarize, and evaluate the ideas and information the author has presented. In other words, you will be examining another person’s thoughts on a topic from your point of view.

Your stand must go beyond your “gut reaction” to the work and be based on your knowledge (readings, lecture, experience) of the topic as well as on factors such as criteria stated in your assignment or discussed by you and your instructor.

Make your stand clear at the beginning of your review, in your evaluations of specific parts, and in your concluding commentary.

Remember that your goal should be to make a few key points about the book or article, not to discuss everything the author writes.

Understanding the Assignment

To write a good critical review, you will have to engage in the mental processes of analyzing (taking apart) the work–deciding what its major components are and determining how these parts (i.e., paragraphs, sections, or chapters) contribute to the work as a whole.

Analyzing the work will help you focus on how and why the author makes certain points and prevent you from merely summarizing what the author says. Assuming the role of an analytical reader will also help you to determine whether or not the author fulfills the stated purpose of the book or article and enhances your understanding or knowledge of a particular topic.

Be sure to read your assignment thoroughly before you read the article or book. Your instructor may have included specific guidelines for you to follow. Keeping these guidelines in mind as you read the article or book can really help you write your paper!

Also, note where the work connects with what you’ve studied in the course. You can make the most efficient use of your reading and notetaking time if you are an active reader; that is, keep relevant questions in mind and jot down page numbers as well as your responses to ideas that appear to be significant as you read.

Please note: The length of your introduction and overview, the number of points you choose to review, and the length of your conclusion should be proportionate to the page limit stated in your assignment and should reflect the complexity of the material being reviewed as well as the expectations of your reader.

Write the introduction

Below are a few guidelines to help you write the introduction to your critical review.

Introduce your review appropriately

Begin your review with an introduction appropriate to your assignment.

If your assignment asks you to review only one book and not to use outside sources, your introduction will focus on identifying the author, the title, the main topic or issue presented in the book, and the author’s purpose in writing the book.

If your assignment asks you to review the book as it relates to issues or themes discussed in the course, or to review two or more books on the same topic, your introduction must also encompass those expectations.

Explain relationships

For example, before you can review two books on a topic, you must explain to your reader in your introduction how they are related to one another.

Within this shared context (or under this “umbrella”) you can then review comparable aspects of both books, pointing out where the authors agree and differ.

In other words, the more complicated your assignment is, the more your introduction must accomplish.

Finally, the introduction to a book review is always the place for you to establish your position as the reviewer (your thesis about the author’s thesis).

As you write, consider the following questions:

• Is the book a memoir, a treatise, a collection of facts, an extended argument, etc.? Is the article a documentary, a write-up of primary research, a position paper, etc.?
• Who is the author? What does the preface or foreword tell you about the author’s purpose, background, and credentials? What is the author’s approach to the topic (as a journalist? a historian? a researcher?)?
• What is the main topic or problem addressed? How does the work relate to a discipline, to a profession, to a particular audience, or to other works on the topic?
• What is your critical evaluation of the work (your thesis)? Why have you taken that position? What criteria are you basing your position on?

Provide an overview

In your introduction, you will also want to provide an overview. An overview supplies your reader with certain general information not appropriate for including in the introduction but necessary to understanding the body of the review.

Generally, an overview describes your book’s division into chapters, sections, or points of discussion. An overview may also include background information about the topic, about your stand, or about the criteria you will use for evaluation.

The overview and the introduction work together to provide a comprehensive beginning for (a “springboard” into) your review.

• What are the author’s basic premises? What issues are raised, or what themes emerge? What situation (i.e., racism on college campuses) provides a basis for the author’s assertions?
• How informed is my reader? What background information is relevant to the entire book and should be placed here rather than in a body paragraph?

Write the body

The body is the center of your paper, where you draw out your main arguments. Below are some guidelines to help you write it.

Organize using a logical plan

Organize the body of your review according to a logical plan. Here are two options:

• First, summarize, in a series of paragraphs, those major points from the book that you plan to discuss; incorporating each major point into a topic sentence for a paragraph is an effective organizational strategy. Second, discuss and evaluate these points in a following group of paragraphs. (There are two dangers lurking in this pattern–you may allot too many paragraphs to summary and too few to evaluation, or you may re-summarize too many points from the book in your evaluation section.)
• Alternatively, you can summarize and evaluate the major points you have chosen from the book in a point-by-point schema. That means you will discuss and evaluate point one within the same paragraph (or in several if the point is significant and warrants extended discussion) before you summarize and evaluate point two, point three, etc., moving in a logical sequence from point to point to point. Here again, it is effective to use the topic sentence of each paragraph to identify the point from the book that you plan to summarize or evaluate.

Questions to keep in mind as you write

With either organizational pattern, consider the following questions:

• What are the author’s most important points? How do these relate to one another? (Make relationships clear by using transitions: “In contrast,” an equally strong argument,” “moreover,” “a final conclusion,” etc.).
• What types of evidence or information does the author present to support his or her points? Is this evidence convincing, controversial, factual, one-sided, etc.? (Consider the use of primary historical material, case studies, narratives, recent scientific findings, statistics.)
• Where does the author do a good job of conveying factual material as well as personal perspective? Where does the author fail to do so? If solutions to a problem are offered, are they believable, misguided, or promising?
• Which parts of the work (particular arguments, descriptions, chapters, etc.) are most effective and which parts are least effective? Why?
• Where (if at all) does the author convey personal prejudice, support illogical relationships, or present evidence out of its appropriate context?

Keep your opinions distinct and cite your sources

Remember, as you discuss the author’s major points, be sure to distinguish consistently between the author’s opinions and your own.

Keep the summary portions of your discussion concise, remembering that your task as a reviewer is to re-see the author’s work, not to re-tell it.

And, importantly, if you refer to ideas from other books and articles or from lecture and course materials, always document your sources, or else you might wander into the realm of plagiarism.

Include only that material which has relevance for your review and use direct quotations sparingly. The Writing Center has other handouts to help you paraphrase text and introduce quotations.

Write the conclusion

You will want to use the conclusion to state your overall critical evaluation.

You have already discussed the major points the author makes, examined how the author supports arguments, and evaluated the quality or effectiveness of specific aspects of the book or article.

Now you must make an evaluation of the work as a whole, determining such things as whether or not the author achieves the stated or implied purpose and if the work makes a significant contribution to an existing body of knowledge.

Consider the following questions:

• Is the work appropriately subjective or objective according to the author’s purpose?
• How well does the work maintain its stated or implied focus? Does the author present extraneous material? Does the author exclude or ignore relevant information?
• How well has the author achieved the overall purpose of the book or article? What contribution does the work make to an existing body of knowledge or to a specific group of readers? Can you justify the use of this work in a particular course?
• What is the most important final comment you wish to make about the book or article? Do you have any suggestions for the direction of future research in the area? What has reading this work done for you or demonstrated to you?

Academic and Professional Writing

This is an accordion element with a series of buttons that open and close related content panels.

Analysis Papers

Reading Poetry

A Short Guide to Close Reading for Literary Analysis

Using Literary Quotations

Play Reviews

Writing a Rhetorical Précis to Analyze Nonfiction Texts

Incorporating Interview Data

Grant Proposals

Planning and Writing a Grant Proposal: The Basics

Additional Resources for Grants and Proposal Writing

Job Materials and Application Essays

Writing Personal Statements for Ph.D. Programs

• Before you begin: useful tips for writing your essay
• Guided brainstorming exercises
• Get more help with your essay
• Frequently Asked Questions

Resume Writing Tips

CV Writing Tips

Cover Letters

Business Letters

Proposals and Dissertations

Resources for Proposal Writers

Resources for Dissertators

Research Papers

Planning and Writing Research Papers

Quoting and Paraphrasing

Writing Annotated Bibliographies

Creating Poster Presentations

Writing an Abstract for Your Research Paper

Thank-You Notes

Advice for Students Writing Thank-You Notes to Donors

Reading for a Review

Critical Reviews

Writing a Review of Literature

Scientific Reports

Scientific Report Format

Sample Lab Assignment

Writing for the Web

Writing an Effective Blog Post

Writing for Social Media: A Guide for Academics

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

• Duke NetID Login
• 919.660.1100
• Duke Health Badge: 24-hour access
• Accounts & Access
• Databases, Journals & Books
• Request & Reserve
• Training & Consulting
• Request Articles & Books
• Renew Online
• Reserve Spaces
• Reserve a Locker
• Study & Meeting Rooms
• Course Reserves
• Digital Health Device Collection
• Pay Fines/Fees
• Recommend a Purchase
• Access From Off Campus
• Building Access
• Computers & Equipment
• Wifi Access
• My Accounts
• Mobile Apps
• Known Access Issues
• Report an Access Issue
• All Databases
• Article Databases
• Basic Sciences
• Clinical Sciences
• Dissertations & Theses
• Drugs, Chemicals & Toxicology
• Grants & Funding
• Interprofessional Education
• Non-Medical Databases
• Search for E-Journals
• Search for Print & E-Journals
• Search for E-Books
• Search for Print & E-Books
• E-Book Collections
• Biostatistics
• Global Health
• MBS Program
• Medical Students
• MMCi Program
• Occupational Therapy
• Path Asst Program
• Physical Therapy
• Researchers
• Community Partners

Conducting Research

• Archival & Historical Research
• Black History at Duke Health
• Data Analytics & Viz Software
• Data: Find and Share
• Evidence-Based Practice
• NIH Public Access Policy Compliance
• Publication Metrics
• Qualitative Research
• Searching Animal Alternatives

Systematic Reviews

• Test Instruments

Using Databases

• JCR Impact Factors
• Web of Science

Finding & Accessing

• COVID-19: Core Clinical Resources
• Health Literacy
• Health Statistics & Data
• Library Orientation

Writing & Citing

• Creating Links
• Getting Published
• Reference Mgmt
• Scientific Writing

Meet a Librarian

• Request a Consultation
• Find Your Liaisons
• Register for a Class
• Request a Class
• Self-Paced Learning

Search Services

• Literature Search
• Systematic Review
• Animal Alternatives (IACUC)
• Research Impact

Citation Mgmt

• Other Software

Scholarly Communications

• About Scholarly Communications
• Publish Your Work
• Measure Your Research Impact
• Engage in Open Science
• Libraries and Publishers
• Directions & Maps
• Floor Plans

Library Updates

• Annual Snapshot
• Conference Presentations
• Contact Information
• Gifts & Donations
• What is a Systematic Review?

Types of Reviews

• Manuals and Reporting Guidelines
• Our Service
• 1. Assemble Your Team
• 2. Develop a Research Question
• 3. Write and Register a Protocol
• 4. Search the Evidence
• 5. Screen Results
• 6. Assess for Quality and Bias
• 7. Extract the Data
• 8. Write the Review
• Additional Resources
• Finding Full-Text Articles

Review Typologies

There are many types of evidence synthesis projects, including systematic reviews as well as others. The selection of review type is wholly dependent on the research question. Not all research questions are well-suited for systematic reviews.

• Review Typologies (from LITR-EX) This site explores different review methodologies such as, systematic, scoping, realist, narrative, state of the art, meta-ethnography, critical, and integrative reviews. The LITR-EX site has a health professions education focus, but the advice and information is widely applicable.

Review the table to peruse review types and associated methodologies. Librarians can also help your team determine which review type might be appropriate for your project. 

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91-108.  doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: What is a Systematic Review?
• Next: Manuals and Reporting Guidelines >>
• Last Updated: May 22, 2024 8:22 PM
• URL: https://guides.mclibrary.duke.edu/sysreview
• Duke Health
• Duke University
• Duke Libraries
• Medical Center Archives
• Duke Directory
• Seeley G. Mudd Building
• 10 Searle Drive
• [email protected]

Literature Review vs Systematic Review

• Literature Review vs. Systematic Review
• Primary vs. Secondary Sources
• Databases and Articles
• Specific Journal or Article

Subject Guide

Definitions

It’s common to confuse systematic and literature reviews because both are used to provide a summary of the existent literature or research on a specific topic. Regardless of this commonality, both types of review vary significantly. The following table provides a detailed explanation as well as the differences between systematic and literature reviews. 

Kysh, Lynn (2013): Difference between a systematic review and a literature review. [figshare]. Available at:  http://dx.doi.org/10.6084/m9.figshare.766364

• << Previous: Home
• Next: Primary vs. Secondary Sources >>
• Last Updated: Dec 15, 2023 10:19 AM
• URL: https://libguides.sjsu.edu/LitRevVSSysRev

• Open access
• Published: 28 May 2024

Composite outcome measures in high-impact critical care randomised controlled trials: a systematic review

• Humphrey G. M. Walker 1 , 2   na1 ,
• Alastair J. Brown 1 , 2 , 3 , 4   na1 ,
• Ines P. Vaz 2 ,
• Rebecca Reed 2 ,
• Max A. Schofield 2 ,
• Jeffrey Shao 5 ,
• Vinodh B. Nanjayya 2 , 3 ,
• Andrew A. Udy 2 , 3 &
• Toby Jeffcote 2 , 3  

Critical Care volume  28 , Article number:  184 ( 2024 ) Cite this article

Metrics details

The use of composite outcome measures (COM) in clinical trials is increasing. Whilst their use is associated with benefits, several limitations have been highlighted and there is limited literature exploring their use within critical care. The primary aim of this study was to evaluate the use of COM in high-impact critical care trials, and compare study parameters (including sample size, statistical significance, and consistency of effect estimates) in trials using composite versus non-composite outcomes.

A systematic review of 16 high-impact journals was conducted. Randomised controlled trials published between 2012 and 2022 reporting a patient important outcome and involving critical care patients, were included.

8271 trials were screened, and 194 included. 39.1% of all trials used a COM and this increased over time. Of those using a COM, only 52.6% explicitly described the outcome as composite. The median number of components was 2 (IQR 2–3). Trials using a COM recruited fewer participants (409 (198.8–851.5) vs 584 (300–1566, p  = 0.004), and their use was not associated with increased rates of statistical significance (19.7% vs 17.8%, p  = 0.380). Predicted effect sizes were overestimated in all but 6 trials. For studies using a COM the effect estimates were consistent across all components in 43.4% of trials. 93% of COM included components that were not patient important.

Conclusions

COM are increasingly used in critical care trials; however effect estimates are frequently inconsistent across COM components confounding outcome interpretations. The use of COM was associated with smaller sample sizes, and no increased likelihood of statistically significant results. Many of the limitations inherent to the use of COM are relevant to critical care research.

Randomised controlled trials (RCT) are the gold standard by which clinicians assess current and emerging treatments in critical care medicine [ 1 ]. Unfortunately, RCTs are associated with significant financial and opportunity costs [ 2 ] and frequently do not identify statistically significant differences in outcomes for a given treatment strategy [ 3 ]. These issues are particularly marked when mortality is used as the primary outcome measure [ 4 , 5 , 6 ].

The selection of appropriate outcomes for critical care trials is important if they are to effectively guide clinical decision making. Trial outcomes need to be meaningful to patients, enable efficient trial design and minimise the risk of missing important treatment effects.

Composite outcome measures (COM) combine multiple clinical events (≥ 2 component outcomes) into a single outcome [ 7 ]. They are designed to capture a greater number of outcome events, and thus increase a trial’s ability to demonstrate a statistically significant treatment effect [ 8 ]. Other proposed methodological benefits include reduced sample size requirements, avoidance of the need to choose a single primary outcome, reduced use of multiple statistical comparisons [ 9 , 10 ] and to account for competing risks [ 11 ]. However, significant limitations have been identified regarding the implementation [ 11 , 12 ] and interpretation [ 8 , 10 ] of COM. A major concern is a lack of consistency of the individual component event rates, resulting in the overall event rate being driven by individual components of the COM. This imbalance risks the creation of a “misleading impression of the impact of treatment” [ 13 , 14 ]. Additionally, the outcome component driving the overall event rate may not be patient important [ 15 ].

COM are increasingly used in the design and implementation of clinical trials, and their use has been evaluated in several medical specialities [ 16 , 17 ]. Within critical care, COM, such as ventilator free days (VFD), are widely used. There is also a trend towards the analysis [ 18 ] or reanalysis [ 19 , 20 ] of trial data using more complex COM. However, their use is occurring in the setting of a paucity of literature evaluating the validity of COM use in critical care research.

Given the increasing utilisation of COM, the trend towards increasingly complex COM and the implications for trial validity, we performed a systematic review to summarise the use and reporting of COM in high-impact critical care RCTs. We focused specifically on COM that include patient important outcomes as there is increasing recognition that trial outcomes should be relevant to patients [ 21 ]. Our primary aim was to quantify COM use in this population, and compare study parameters (including sample size, statistical significance, and consistency of effect estimates) between trials using composite and non-composite outcomes.

Methods for inclusion and analysis of studies in this systematic review were pre-specified in a protocol developed in accordance with the most recent Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) [ 22 ] guidelines. This protocol was prospectively registered with PROSPERO (CRD42022380606). No funding for this study was obtained.

Search strategy

The full search strategy is described in the online data supplement and used the following databases: Ovid MEDLINE, PubMed and the Cochrane Library. RCTs published from 1st August 2012 until 31st December 2022 were included. The reference sections of selected trials from preliminary searches were reviewed to guide the refinement of search strategies.

Selection criteria

The titles and abstracts were independently screened by two of three reviewers (HW, AB, IV) and scored against prespecified eligibility criteria. Disagreements were resolved by a third senior reviewer (TJ). All trials deemed eligible underwent full text review prior to data extraction to ensure inclusion criteria were met .

We included RCTs that were conducted in the critical care population, published in the English language in sixteen high impact factor journals. Journal selection was based on a previous review by Harhay et al. [ 5 ]. The primary outcome or at least one component of the COM had to be a patient important outcome as defined by Gaudry et al. [ 23 ]: “mortality at any time and/or quality of life, functional/cognitive/neurological outcomes assessed after ICU discharge”.

Exclusion criteria included non randomised controlled trials, involving those < 18 years old or any trial that did not include a patient important outcome. The full inclusion and exclusion criteria, alongside the journals selected are published in the online data supplement (Table S1 ).

Data collection and risk of bias

All selected trials were independently reviewed by two of six reviewers (HW, AB, IV, MS, JS, RR). Risk of bias analysis was performed using The Cochrane Risk of Bias 2 (RoB2) tool [ 24 ]. Data were extracted and recorded using standardised data forms on a web-based collaboration software platform (Covidence) [ 25 ]. The data forms were designed prior to the commencement of data collection. Discrepancies in risk of bias score, methodological classification or other data points were identified by the Covidence system and were resolved in consultation with a third senior reviewer (TJ, AU, VN). No missing data were imputed.

Definitions of outcomes

Outcome measures were categorised as follows:

Categorical composites in which patients were defined as having the primary outcome if they met pre-defined criteria for one or more of the discrete components.

Continuous composites included any outcome that measured duration of organ support or disease process (e.g. VFD).

Non-COM studies consisted of trials using:

A single binary outcome e.g. mortality

Other included those single outcomes that did not fit into the above categories. They principally measured patient performance within a single domain (such as Modified Rankin Scale or Glasgow Outcome Score as a measure of functional status)

Outcomes and data synthesis

Each trial was primarily categorised as a COM or non-COM trial. COM trials were then sub-categorised according to the classification above. When classifying components of a COM, all potential possibilities of meeting the outcome were included as individual components. For example, the COM “Days Alive and Free of Life Support” included 4 potential outcomes: mortality, duration of mechanical ventilation, circulatory support, and renal replacement therapy respectively. Trials with co-primary outcomes were included. If a co-primary outcome consisted of both a non-COM and a COM e.g. Kawazoe et al. [ 26 ], then it was classified as a COM. Trials were also included if utilising both patient and non-patient important co-primary outcomes e.g. Zarbock et al. [ 27 ].

For comparison of achieved effect sizes, the difference between predicted and observed effect size was calculated. Predicted relative effect estimates were calculated to compare the predicted magnitude of effect used for power calculations. Methods for calculating the event rate gap are described in the online data supplement. Trials with more than two arms, 2 × 2 factorial designs or studies with co-primary outcomes were not included in event rate and effect size analyses. Data for comparison of event rates or effect sizes, when presented as a range of values or without reference to an absolute risk difference were excluded.

Statistical analysis

Continuous data are presented as median (interquartile range) and mean (standard deviation) where appropriate. Categorical data are presented as frequency (%). The analysis was primarily descriptive, albeit where comparisons were made, the Mann–Whitney U Test and Fisher’s Exact test have been used, as indicated. The analysis of COM use over time was conducted using the Mann Kendall trend test. To calculate the predicted relative effect estimate, the predicted effect sizes were converted into relative risk estimates by dividing the predicted absolute risk reduction by the expected event rate in the control group. The same principle was used for continuous outcomes (e.g. if an effect size of 2 VFD was predicted with a control rate of 8, then this was a 25% relative difference). These have been separated due to the differences in data and scale used (e.g. VFD and % reduction in mortality).

We conducted a sensitivity analysis in which we included ordinal outcome scales such as mRS (modified Rankin Scale) as COM, and also removed them from the analysis all together.

Significance has been assumed with a p -value < 0.05 and no adjustment for multiplicity was made. All statistical analyses were conducted using GraphPad Prism 10.0.3 [ 28 ].

12,270 trials were identified following the database search and after removal of duplicates, 8271 records underwent title and abstract screening. 194 trials were included in the final analysis. This is shown in Fig.  1 . A reference list is included in the online data supplement.

Showing PRISMA flow diagram of trial selection

Trial characteristics

Of the 194 trials included in the analysis, 76 (39.1%) RCTs were classified as using a COM for the primary outcome. For trials using a non-COM, 101 assessed mortality (or survival) at various time points as a primary outcome. The basic characteristics of the included trials are outlined in Table  1 . Figures S1 and S2 in the online data supplement show all populations and interventions studied in the included trials.

COM versus non-COM

The use of COMs in high-impact critical care RCTs has significantly increased since 2012 ( p  = 0.002 for trend over time, Fig.  2 ). Comparing RCTs using a COM vs non-COM respectively, significantly fewer patients were enrolled [409 (198.8–851.5) vs 584 (300–1566) patients, p  = 0.004] and the median number of sites was lower [12 (2–35) vs 31 (11.5–50.5) sites, p  = 0.009]. The median reported power for COM and non-COM trials were similar at 80 (IQR 80–85) % and 80 (IQR 80–90) % respectively. This is shown in Table  2 (and Fig. S3 in the online data supplement).

Showing the trend of COM use as a proportion of all trials included in the systematic review

Of note, a significant proportion of trials using a COM (n = 17, 22.4%) investigated COVID-19. Fewer (n = 7, 5.9%) non-COM trials investigated this disease. 40/76 (52.6%) of those using a COM were explicitly described as a COM in the manuscript. 49 (64.5%) had all components reported in the table of results. There was no difference in Risk of Bias assessment scores between COM and non-COM trials (see Fig. S4 in the online data supplement).

The median number of components was 2 (2–3), and 5 (6.6%) trials utilised a COM composed entirely of patient important components. All of the COM included mortality as a component of the composite outcome. Other component types utilised are shown in Fig.  3 . A comprehensive list of all the components utilised in the included trials is provide in the online data supplement (Table S2 ).

Showing a heatmap of all the components used in COMs within this systematic review. a Circulatory support includes extracorporeal membrane oxygenation (ECMO). b Includes outcomes assessing both hospital and ICU length of stay. c Operative specific components include outcomes such as re-operation. d Disease specific components includes development of specific diseases or syndromes. For example, myocardial infarction (MI), acute kidney injury (AKI), Acute Respiratory Distress Syndrome (ARDS) or venous thromboembolic event (VTE)

Categorical versus continuous COM

31 (40.8%) of the COM were categorical, with the remaining 45 (59.2%) being continuous. Of the continuous COM, 36 (80.0%) were organ support free days (OSFD). Of these, 7 (15.6%) scored death as -1. Identification within the trial manuscript that a COM was being utilised occurred less frequently in the continuous COM group (83.9% vs 31.1%, p  < 0.001). This is shown in Table  2 .

Consistency of effect estimates and effect sizes

In trials using a COM, it was only possible to assess whether point effect estimates were consistent in 58 (76.3%) of trials with the information provided. Point effect estimates were consistent (e.g. all in favour or not in favour of intervention) across all components in 33 (56.9%) of trials. In the remaining trials, point estimates were not consistent (n = 16 categorical and n = 9 continuous). There was no apparent difference between trials using a categorical or continuous COM [15/31 (48.4%) vs 18/45 (66.7%), p  = 0.192]. Mortality was the component with the highest event rate in 15/31 (48.4%) of trials using a categorical COM.

The predicted relative effect estimates are shown in Fig.  4 . For trials using a categorical COM this was 35.4 (IQR 24.8–50.0), and for trials using a continuous COM this was 20.2 (IQR 16.7–36.7). For non-COM trials that assessed mortality this was 25.0 (IQR 19.6–32.7) and for other non-COM trials this was 38.9 (IQR 22.5–56.8). Predicted relative effect estimates were significantly greater for categorical COM compared to non-COM trials assessing mortality ( p  < 0.001). Most trials had predicted effect sizes greater than the actual effect size, with only 6 trials having an effect size larger than predicted . This is shown in Fig. S5 in the online data supplement. The event rate gap was similar between COM and Non-COM trials (Table S3 and Fig. S6 in the online data supplement).

Showing the predicted relative effect estimates for COM and Non-COM studies. Error bars are median and IQR

Reported statistical significance

There was no difference in the frequency of reported statistical significance between COM and non-COM trials [17/56 (19.7%) vs 21/118 (17.8%), p  = 0.85]. Rates of statistical significance did not vary between categorical COM and continuous COM [8/31 (25.8%) vs 7/45 (15.6%), p  = 0.38].

Sensitivity analyses

19 trials used ordinal outcomes. Results in all sensitivity analyses were unchanged with respect to number of sites, number of participants, trial power and rates of statistical significance (Tables S5 –S8 in the online data supplement).

We performed a systematic review of critical care trials published in high impact journals with a specific focus on COM. We assessed the frequency of use and reporting of COM in the relevant studies. We documented the different forms of COM, the consistency of treatment effect between individual components of the composites and the proportion of patient centred outcomes within these composites. Lastly, we compared the accuracy of effect size calculations and frequency of statistically significant results between COM and non-COM studies.

The recently published CONSORT-Outcomes 2022 extension [ 7 ] calls for all components of a COM to be reported in order to clarify the interpretation of trial findings. Only 65% of trials identified in our systematic review had complete reporting of all individual components in the manuscript. Additionally, only 50% of the trials explicitly acknowledged the use of a COM in the published manuscript. There was a notable lack of acknowledgement of the composite nature of continuous COM such as organ support free days, a finding that is consistent with previous literature [ 29 ]. The median number of components for all trials with a COM was 2 (IQR 2–3) and the maximum number of components was 21 [ 30 ] . This is important as increasing components will complicate the interpretation of which components are driving any treatment effect identified by clinical trials. The number of trials using a COM composed entirely of patient important outcomes was low (7%). Treatment effects were therefore partly determined by non-patient important outcomes (e.g. asymptomatic DVT, acute kidney injury, worsening oxygenation, radiographic progression) in 93% of trials. While we recognise the clinical validity of a range of non-patient important outcomes, we would argue that these should be analysed in a way that accounts for the varying importance of specific components within a COM. One analytical method that allows for the prioritisation of outcomes within a single COM may be the Win Ratio [ 31 ] which has been used recently in the re-analysis of several large critical care trials and may mitigate the challenge of the competing risk of death and other components of composite outcomes [ 19 , 20 ].

Despite the well documented challenges associated with their use [ 29 , 32 ] continuous COM, and particularly OSFDs, were used widely (60% and 50% respectively). One specific issue with these forms of outcome measure is inconsistency of mortality scoring within the continuous COM. Our review revealed that both mortality and ongoing dependence on organ support were scored as zero in approximately 80% of studies, whereas in a further 20%, death was scored as minus one and ongoing dependence on support as zero. This heterogeneity has been previously documented [ 33 ] and is a significant barrier to study comparisons and meta-analyses. A widely accepted standard for the calculation of failure free days has not yet been established but could reduce methodological heterogeneity [ 34 ].

48% of categorical COM demonstrated consistency of treatment effect across COM components. Given the complexity of physiological derangements in critically ill patients this finding is not unexpected. There is, however, an argument for trialists to demonstrate biological plausibility for the components of a COM. Further, the explicit recognition of trial outcomes that are based on conflicting treatment effects or entirely driven by a single component of a COM should be encouraged.

Potential benefits of COM include increased power and/or reductions in sample size requirements [ 8 , 12 ]. These may be particularly valuable in critical care trials where overestimation of effect size has been consistently demonstrated [ 5 , 35 ]. However, our review revealed that only 1 trial utilising a COM had an effect size greater than predicted [ 36 ]. Greater predicted relative effect estimates for categorical COM in comparison to non-COM trials with mortality as the primary outcome was shown, and median study power for both COM and non-COM studies were similar, implying that in critical care trials COM are employed primarily to reduce sample size requirements rather than increase statistical power. In some settings (such as the recent COVID-19 pandemic) smaller sample sizes may facilitate rapid assessment of interventions. Indeed, our findings indicate increased use of continuous COM during the COVID-19 pandemic. We would caution however, that the use of COM to reduce sample size necessitates increased scrutiny of the components of the COM, the heterogeneity of component treatment effects and the degree of patient centredness. We would also argue for pre-trial documentation of COM components, expected treatment effects and the specification of what the investigators consider a minimum clinically important difference (MCID) [ 37 ] to determine if a particular result represents a genuine treatment effect or a type 2 error.

The use of COM has been associated with a trend towards increased rates of statistical significance in other specialties such as cardiology [ 38 ]. In our review there was no difference in rates of statistical significance between COM and non-COM trials. Additionally, the rate of statistical significance in our review (18.5%), is lower than previously reported for critical care trials [ 3 , 5 ]. It is noteworthy though, that previous reviews were not restricted to patient important outcomes. The observation that rates of statistical significance for clinical trials utilising COM differ between different medical specialties may reflect differences in the disease states investigated. A significant sub-set of the diseases investigated in critical care trials e.g. traumatic brain injury, acute respiratory distress syndrome, can be characterised as physiological syndromes rather than diseases with a clear pathophysiology such as acute myocardial ischaemia. The heterogeneity of critical care syndromes is increasingly recognised as a confounding factor in critical care trials [ 39 ]. A COM which reduces sample size requirements for a predicted treatment effect but fails to capture this heterogeneity will not lead to increased rates of statistically significant trials. While careful patient selection and cohort enrichment are key factors in improving trial efficiency in critical care, the flexibility of COM represents an opportunity for significantly improved critical care trial design. A thoughtfully designed COM that can represent the heterogenous nature of a critical care syndrome may be able to successfully demonstrate treatment effects with smaller sample sizes and the use of fewer resources. Recommendations based on the findings from this systematic review regarding the use of COM in critical care trials are made in Table  3 .

Strengths and limitations

To our knowledge this is the first systematic review of COM use in the critical care RCTs. We have utilised a robust and reproducible search strategy and a study selection methodology based on prior literature designed to assess the outcomes and characteristics of ICU based RCTs [ 5 ]. We have sought to distinguish between continuous and categorical COM, document the reporting of COM, assess the validity of sample size calculation and the frequency of statistically significant results. Given their increasing use, the lack of literature exploring their characteristics and the advent of more complex COM we submit that this is a timely and useful review of the literature.

Limitations of the review include the focus on 16 high impact critical care journals. There will undoubtedly be critical care trials using patient focused COM that are not captured by this review. However, we feel that practice change is predominantly based on larger trials published in high impact journals so a focus on this category of evidence is justified. Our review also omits studies utilising COM without any patient centred components. We feel that our use of a recognised definition of patient centred outcomes [ 23 ] and the increasing importance of this concept [ 40 ] adds weight to the validity of this approach. The omission of ordinal scales such as the modified Rankin scale or Glasgow outcome scale from our definition of COM could be considered inconsistent with our inclusion of continuous COM such as ventilator free days. We acknowledge the challenges and nuances associated with classification of these outcomes. Our decision to classify them as non-COM for our primary statistical analyses results from finding no examples of these scales being described as COM within the literature, which contrasts to OSFDs [ 29 ]. Additionally, it could be suggested that all levels of ordinal scales assessing neurological outcome fall within the single domain of function [ 41 ]. In recognition of the ambiguity surrounding their classification we performed multiple sensitivity analyses. The results of these analyses do not materially change our conclusions. Additionally, the sensitivity analyses of these scales reveal heterogeneity in terms of their application and analysis, a finding that has previously been described [ 42 , 43 ].

In high-impact critical care RCTs assessing patient important outcomes, COM are used frequently. The use of COM has increased over time. The inherent limitations of COM identified in trials from related medical specialties, are also relevant to critical care. Many trials using a COM did not demonstrate consistency across all components of the COM and 93% of COM included non patient-important components. The primary methodological benefit of a COM is a reduction in sample size. However, their use does not lead to increased rates of statistically significant results in critical care trials and predicted effect sizes remain grossly over optimistic. Further work to improve the use of COM in critical care should focus on the design and validation of COM that include patient important outcomes and effectively represent the heterogeneity of the pathologies studied in the critical care literature.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Randomised controlled trial

Composite outcome measure

Non-composite outcome measure

Ventilator free days

Extracorporeal membrane oxygenation

Myocardial infarction

Acute kidney injury

Acute respiratory distress syndrome

Venous thromboembolic event

Organ support free days

Modified rankin scale

Glasgow outcome score extended

Granholm A, Alhazzani W, Derde LPG, Angus DC, Zampieri FG, Hammond NE, et al. Randomised clinical trials in critical care: past, present and future. Intensive Care Med. 2022;48:164–78.

Article   CAS   PubMed   Google Scholar  

Speich B, von Niederhäusern B, Schur N, Hemkens LG, Fürst T, Bhatnagar N, et al. Systematic review on costs and resource use of randomized clinical trials shows a lack of transparent and comprehensive data. J Clin Epidemiol. 2018;96:1–11.

Article   PubMed   Google Scholar  

Kampman JM, Sperna Weiland NH, Hermanides J, Hollmann MW, Repping S, et al. Randomized controlled trials in ICU in the four highest-impact general medicine journals. Crit Care Med. 2023;51:e179–83.

Article   PubMed   PubMed Central   Google Scholar  

Veldhoen RA, Howes D, Maslove DM. Is mortality a useful primary end point for critical care trials? Chest. 2020;158:206–11.

Harhay M, Wagner J, Ratcliffe SJ, Bronheim RS, Gopal A, Green S, et al. Outcomes and statistical power in adult critical care randomized trials. Am J Respir Crit Care Med. 2014;189:1469–78.

Santacruz CA, Pereira AJ, Celis E, Vincent JL. Which multicenter randomized controlled trials in critical care medicine have shown reduced mortality? A systematic review. Crit Care Med. 2019;47:1680–91.

Butcher NJ, Monsour A, Mew EJ, Chan A-W, Moher D, Mayo-Wilson E, et al. Guidelines for reporting outcomes in trial protocols. JAMA. 2022;328:2345–56.

Palileo-Villanueva LM, Dans AL. Composite endpoints. J Clin Epidemiol. 2020;128:157–8.

Diaz-Quijano FA. Estimating and testing an index of bias attributable to composite outcomes in comparative studies. J Clin Epidemiol. 2021;132:1–9.

Dash K, Goodacre S, Sutton L. Composite outcomes in clinical prediction modeling: are we trying to predict apples and oranges? Ann Emerg Med. 2022;80:12–9.

Manja V, Al Bashir S, Guyatt G. Criteria for use of composite end points for competing risks—a systematic survey of the literature with recommendations. J Clin Epidemiol. 2017;82:4–11.

Choi SW, Cheung CW. The case of the misleading composite: one outcome is better than two. Anaesthesia. 2016;71:1101–3.

Myles PS, Devereaux PJ. Pros and cons of composite endpoints in anesthesia trials. Anesthesiology. 2010;113:776–8.

Irony TZ. The “utility” in composite outcome measures: measuring what is important to patients. JAMA. 2017;318:1820–1.

Ferreira-González I, Busse JW, Heels-Ansdell D, Montori VM, Akl EA, Bryant DM, et al. Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ. 2007;334:786.

Herman D, Lor KY, Qadree A, Horn D, D’Souza R. Composite adverse outcomes in obstetric studies: a systematic review. BMC Pregnancy Childb. 2021;21:107.

Article   Google Scholar  

Lai NM, Yap AQY, Ong HC, Wai SX, Yeo JHH, Koo CYE, et al. Use of composite outcomes in neonatal trials: an analysis of the cochrane reviews. Neonatology. 2021;118:259–63.

Freund Y, Viglino D, Cachanado M, Cassard C, Montassier E, Douay B, et al. Effect of Noninvasive airway management of comatose patients with acute poisoning: a randomized clinical trial. JAMA. 2023;330:2267–74.

Zampieri FG, Damiani LP, Biondi RS, Freitas FGR, Veiga VC, Figueiredo RC, et al. Hierarchical endpoint analysis using win ratio in critical care: an exploration using the balanced solutions in intensive care study (BaSICS). J Crit Care. 2022;71:154113.

Zampieri FG, Serpa-Neto A, Wald R, Bellomo R, Bagshaw SM. Hierarchical endpoints in critical care: A post-hoc exploratory analysis of the standard versus accelerated initiation of renal-replacement therapy in acute kidney injury and the intensity of continuous renal-replacement therapy in critically ill patients trials. J Crit Care. 2024;82:154767.

Iwashyna TJ, McPeake J. Choosing outcomes for clinical trials: a pragmatic perspective. Curr Opin Crit Care. 2018;24:428–33.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Gaudry S, Messika J, Ricard JD, Guillo S, Pasquet B, Dubief E, et al. Patient-important outcomes in randomized controlled trials in critically ill patients: a systematic review. Ann Intensive Care. 2017;7:28.

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.

Covidence Systematic Review Software (2023). Melbourne, Australia: Veritas Health Innovation at https://www.covidence.org

Kawazoe Y, Miyamoto K, Morimoto T, Yamamoto T, Fuke A, Hashimoto A, et al. Effect of dexmedetomidine on mortality and ventilator-free days in patients requiring mechanical ventilation with sepsis: a randomized clinical trial. JAMA. 2017;317:1321–8.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Zarbock A, Küllmar M, Kindgen-Milles D, Wempe C, Gerss J, Brandenburger T, et al. Effect of regional citrate anticoagulation vs systemic heparin anticoagulation during continuous kidney replacement therapy on dialysis filter life span and mortality among critically Ill patients with acute kidney injury: a randomized clinical trial. JAMA. 2020;324:1629–39.

GraphPad Prism version 10.0.3 for Windows (2023). Boston, Massachusetts, USA: Graph Pad Software at www.graphpad.com

Yehya N, Harhay MO, Curley MAQ, Schoenfeld DA, Reeder RW. Reappraisal of ventilator-free days in critical care research. Am J Resp Crit Care. 2019;200:828–36.

Chesnut RM, Temkin N, Carney N, Dikmen S, Rondina C, et al. A trial of intracranial-pressure monitoring in traumatic brain injury. N Engl J Med. 2012;367:2471–81.

Pocock SJ, Ariti CA, Collier TJ, Wang D. The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities. Eur Heart J. 2012;33:176–82.

Verghis RM, McDowell C, Blackwood B, Lee B, McAuley DF, Clarke M. Re-analysis of ventilator-free days (VFD) in acute respiratory distress syndrome (ARDS) studies. Trials. 2023;24:183.

Contentin L, Ehrmann S, Giraudeau B. Heterogeneity in the definition of mechanical ventilation duration and ventilator-free days. Am J Respir Crit Care Med. 2014;189:998–1002.

Triché LR, Futier E, Carvalho MD, Piñol-Domenech N, Bodet-Contentin L, Jabaudon M, et al. Sample size estimation in clinical trials using ventilator-free days as the primary outcome: a systematic review. Crit Care. 2023;27:303.

Abrams D, Montesi SB, Moore SKL, Manson DK, Klipper KM, Case MA, et al. Powering bias and clinically important treatment effects in randomized trials of critical illness. Crit Care Med. 2020;48:1710–9.

Zhou Y, Jin X, Lv Y, Wang P, Yang Y, et al. Early application of airway pressure release ventilation may reduce the duration of mechanical ventilation in acute respiratory distress syndrome. Intens Care Med. 2017;43:1648–59.

Draak THP, de Greef BTA, Faber CG, Merkies ISJ. The minimum clinically important difference: which direction to take. Eur J Neurol. 2019;26:850–5.

Tan NS, Ali SH, Lebovic G, Mamdani M, Laupacis A, Yan AT. Temporal trends in use of composite end points in major cardiovascular randomized clinical trials in prominent medical journals. Circ Cardiovasc Qual Outcomes. 2018;10:e003753.

Laffey JG, Kavanagh BP. Negative trials in critical care: why most research is probably wrong. Lancet Respir Med. 2018;6:659–60.

Dinglas VD, Faraone LN, Needham DM. Understanding patient-important outcomes after critical illness. Curr Opin Crit Care. 2018;24:401–9.

Broderick JP, Adeoye O, Elm J. Evolution of the modified rankin scale and its use in future stroke trials. Stroke. 2017;48:2007–12.

Saver JL, Chaisinanunkul N, Campbell BCV, Grotta JC, Hill MD, Khatri P, et al. Standardized nomenclature for modified rankin scale global disability outcomes: consensus recommendations from stroke therapy academic industry roundtable XI. Stroke. 2021;52:3054–62.

Selman CJ, Lee KJ, Ferguson KN, et al. Statistical analyses of ordinal outcomes in randomised controlled trials: a scoping review. Trials. 2024;25:241.

Download references

Acknowledgements

Not applicable.

No funding was obtained for this study.

Author information

Humphrey G. M. Walker and Alastair J. Brown are joint first authors.

Authors and Affiliations

Department of Critical Care, St Vincent’s Hospital, Melbourne, VIC, Australia

Humphrey G. M. Walker & Alastair J. Brown

Department of Intensive Care and Hyperbaric Medicine, The Alfred Hospital, Melbourne, Australia

Humphrey G. M. Walker, Alastair J. Brown, Ines P. Vaz, Rebecca Reed, Max A. Schofield, Vinodh B. Nanjayya, Andrew A. Udy & Toby Jeffcote

Australian and New Zealand Intensive Care Research Centre, School of Public Health and Preventive Medicine, Monash University, Prahran, VIC, Australia

Alastair J. Brown, Vinodh B. Nanjayya, Andrew A. Udy & Toby Jeffcote

Department of Critical Care, University of Melbourne, Melbourne, VIC, Australia

Alastair J. Brown

Monash University, Melbourne, VIC, Australia

Jeffrey Shao

You can also search for this author in PubMed   Google Scholar

Contributions

H.W., T.J., V.N., A.U. and A.B. designed the study. H.W., I.V., M.S., J.S., R.R., and A.B. screened trials, reviewed trial abstracts, and collected data. H.W., A.B., and T.J. drafted the initial manuscript. H.W. produced all the figures. H.W. and V.N. conducted the statistical analysis. A.B., A.U., V.N., and T.J. provided senior support. All authors reviewed and approved the manuscript.

Corresponding author

Correspondence to Humphrey G. M. Walker .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

No potential conflicts of interest relevant to this article were reported however AU reports grants (paid to Monash University) from NHMRC Australia, MRFF Australia and the Department of Health, Commonwealth of Australia that are unrelated to this work and the receipt of in-kind support (trial consumables) from Integra Lifesciences for a project outside of the current work.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Walker, H.G.M., Brown, A.J., Vaz, I.P. et al. Composite outcome measures in high-impact critical care randomised controlled trials: a systematic review. Crit Care 28 , 184 (2024). https://doi.org/10.1186/s13054-024-04967-3

Download citation

Received : 08 April 2024

Accepted : 22 May 2024

Published : 28 May 2024

DOI : https://doi.org/10.1186/s13054-024-04967-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Critical care outcomes
• Outcome assessment
• Critical care
• Randomized controlled trials as topic

Critical Care

ISSN: 1364-8535

• Submission enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Mark Access Health Policy
• v.11(1); 2023
• PMC10392303

Rapid literature review: definition and methodology

Beata smela.

a Assignity, Cracow, Poland

Mondher Toumi

b Public Health Department, Aix-Marseille University, Marseille, France

Karolina Świerk

Clement francois, małgorzata biernikiewicz.

c Studio Slowa, Wroclaw, Poland

Emilie Clay

d Clever-Access, Paris, France

Laurent Boyer

Introduction: A rapid literature review (RLR) is an alternative to systematic literature review (SLR) that can speed up the analysis of newly published data. The objective was to identify and summarize available information regarding different approaches to defining RLR and the methodology applied to the conduct of such reviews.

Methods: The Medline and EMBASE databases, as well as the grey literature, were searched using the set of keywords and their combination related to the targeted and rapid review, as well as design, approach, and methodology. Of the 3,898 records retrieved, 12 articles were included.

Results: Specific definition of RLRs has only been developed in 2021. In terms of methodology, the RLR should be completed within shorter timeframes using simplified procedures in comparison to SLRs, while maintaining a similar level of transparency and minimizing bias. Inherent components of the RLR process should be a clear research question, search protocol, simplified process of study selection, data extraction, and quality assurance.

Conclusions: There is a lack of consensus on the formal definition of the RLR and the best approaches to perform it. The evidence-based supporting methods are evolving, and more work is needed to define the most robust approaches.

Introduction

A systematic literature review (SLR) summarizes the results of all available studies on a specific topic and provides a high level of evidence. Authors of the SLR have to follow an advanced plan that covers defining a priori information regarding the research question, sources they are going to search, inclusion criteria applied to choose studies answering the research question, and information regarding how they are going to summarize findings [ 1 ].

The rigor and transparency of SLRs make them the most reliable form of literature review [ 2 ], providing a comprehensive, objective summary of the evidence for a given topic [ 3 , 4 ]. On the other hand, the SLR process is usually very time-consuming and requires a lot of human resources. Taking into account a high increase of newly published data and a growing need to analyze information in the fastest possible way, rapid literature reviews (RLRs) often replace standard SLRs.

There are several guidelines on the methodology of RLRs [ 5–11 ]; however, only recently, one publication from 2021 attempted to construct a unified definition [ 11 ]. Generally, by RLRs, researchers understand evidence synthesis during which some of the components of the systematic approach are being used to facilitate answering a focused research question; however, scope restrictions and a narrower search strategy help to make the project manageable in a shorter time and to get the key conclusions faster [ 4 ].

The objective of this research was to collect and summarize available information on different approaches to the definition and methodology of RLRs. An RLR has been run to capture publications providing data that fit the project objective.

To find publications reporting information on the methodology of RLRs, searches were run in the Medline and EMBASE databases in November 2022. The following keywords were searched for in titles and abstracts: ‘targeted adj2 review’ OR ‘focused adj2 review’ OR ‘rapid adj2 review’, and ‘methodology’ OR ‘design’ OR ‘scheme’ OR ‘approach’. The grey literature was identified using Google Scholar with keywords including ‘targeted review methodology’ OR ‘focused review methodology’ OR ‘rapid review methodology’. Only publications in English were included, and the date of publication was restricted to year 2016 onward in order to identify the most up-to-date literature. The reference lists of each included article were searched manually to obtain the potentially eligible articles. Titles and abstracts of the retrieved records were first screened to exclude articles that were evidently irrelevant. The full texts of potentially relevant papers were further reviewed to examine their eligibility.

A pre-defined Excel grid was developed to extract the following information related to the methodology of RLR from guidelines:

• Definition,
• Research question and searches,
• Studies selection,
• Data extraction and quality assessment,
• Additional information.

There was no restriction on the study types to be analyzed; any study reporting on the methodology of RLRs could be included: reviews, practice guidelines, commentaries, and expert opinions on RLR relevant to healthcare policymakers or practitioners. The data extraction and evidence summary were conducted by one analyst and further examined by a senior analyst to ensure that relevant information was not omitted. Disagreements were resolved by discussion and consensus.

Studies selection

A total of 3,898 records (3,864 articles from a database search and 34 grey literature from Google Scholar) were retrieved. After removing duplicates, titles and abstracts of 3,813 articles were uploaded and screened. The full texts of 43 articles were analyzed resulting in 12 articles selected for this review, including 7 guidelines [ 5–11 ] on the methodology of RLRs, together with 2 papers summarizing the results of the Delphi consensus on the topic [ 12 , 13 ], and 3 publications analyzing and assessing different approaches to RLRs [ 4 , 14 , 15 ].

Overall, seven guidelines were identified: from the World Health Organization (WHO) [ 5 ], National Collaborating Centre for Methods and Tools (NCCMT) [ 7 ], the UK government [ 8 ], the Oxford Centre for Evidence Based Medicine [ 9 ], the Cochrane group [ 6 , 11 ], and one multi-national review [ 10 ]. Among the papers that did not describe the guidelines, Gordon et al. [ 4 ] proposed 12 tips for conducting a rapid review in the right settings and discussed why these reviews may be more beneficial in some circumstances. The objective of work conducted by Tricco et al. [ 13 ] and Pandor et al. [ 12 ] was to collect and compare perceptions of rapid reviews from stakeholders, including researchers, policymakers, industry, journal editors, and healthcare providers, and to reach a consensus outlining the domains to consider when deciding on approaches for RLRs. Haby et al. [ 14 ] run a rapid review of systematic reviews and primary studies to find out the best way to conduct an RLR in health policy and practice. In Tricco et al. (2022) [ 15 ], JBI position statement for RLRs is presented.

From all the seven identified guidelines information regarding definitions the authors used for RLRs, approach to the PICOS criteria and search strategy development, studies selection, data extractions, quality assessment, and reporting were extracted.

Cochrane Rapid Reviews Methods Group developed methods guidance based on scoping review of the underlying evidence, primary methods studies conducted, as well as surveys sent to Cochrane representative and discussion among those with expertise [ 11 ]. They analyzed over 300 RLRs or RLR method papers and based on the methodology of those studies, constructed a broad definition RLR, one that meets a minimum set of requirements identified in the thematic analysis: ‘ A rapid review is a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting a variety of methods to produce evidence in a resource-efficient manner .’ This interpretation aligns with more than 50% of RLRs identified in this study. The authors additionally provided several other definitions, depending on specific situations or requirements (e.g., when RLR is produced on stakeholder’s request). It was additionally underlined that RLRs should be driven by the need of timely evidence for decision-making purposes [ 11 ].

Rapid reviews vary in their objective, format, and methods used for evidence synthesis. This is a quite new area, and still no agreement on optimal methods can be found [ 5 ]. All of the definitions are highlighting that RLRs are completed within shorter timeframes than SLRs, and also lack of time is one of the main reasons they are conducted. It has been suggested that most rapid reviews are conducted within 12 weeks; however, some of the resources suggest time between a few weeks to no more than 6 months [ 5 , 6 ]. Some of the definitions are highlighting that RLRs follow the SLR process, but certain phases of the process are simplified or omitted to retrieve information in a time-saving way [ 6 , 7 ]. Different mechanisms are used to enhance the timeliness of reviews. They can be used independently or concurrently: increasing the intensity of work by intensifying the efforts of multiple analysts by parallelization of tasks, using review shortcuts whereby one or more systematic review steps may be reduced, automatizing review steps by using new technologies [ 5 ]. The UK government report [ 8 ] referred to two different RLRs: in the form of quick scoping reviews (QSR) or rapid evidence assessments (REA). While being less resource and time-consuming compared to standard SLRs, QSRs and REAs are designed to be similarly transparent and to minimize bias. QSRs can be applied to rather open-ended questions, e.g., ‘what do we know about something’ but both, QSRs and REAs, provide an understanding of the volume and characteristics of evidence on a specific topic, allowing answering questions by maximizing the use of existing data, and providing a clear picture of the adequacy of existing evidence [ 8 ].

Research questions and searches

The guidelines suggest creating a clear research question and search protocol at the beginning of the project. Additionally, to not duplicate RLRs, the Cochrane Rapid Reviews Methods Group encourages all people working on RLRs to consider registering their search protocol with PROSPERO, the international prospective register of reviews; however, so far they are not formally registered in most cases [ 5 , 6 ]. They also recommend involving key stakeholders (review users) to set and refine the review question, criteria, and outcomes, as well as consulting them through the entire process [ 11 ].

Regarding research questions, it is better to structure them in a neutral way rather than focus on a specific direction for the outcome. By doing so, the researcher is in a better position to identify all the relevant evidence [ 7 ]. Authors can add a second, supportive research question when needed [ 8 ]. It is encouraged to limit the number of interventions, comparators and outcomes, to focus on the ones that are most important for decision-making [ 11 ]. Useful could be also reviewing additional materials, e.g., SLRs on the topic, as well as conducting a quick literature search to better understand the topic before starting with RLRs [ 7 ]. In SLRs researchers usually do not need to care a lot about time spent on creating PICOS, they need to make sure that the scope is broad enough, and they cannot use many restrictions. When working on RLRs, a reviewer may spend more or less time defining each of the components of the study question, and the main step is making sure that PICOS addresses the needs of those who requested the rapid review, and at the same time, it is feasible within the required time frame [ 7 ]. Search protocol should contain an outline of how the following review steps are to be carried out, including selected search keywords and a full strategy, a list of data sources, precise inclusion and exclusion criteria, a strategy for data extraction and critical appraisal, and a plan of how the information will be synthesized [ 8 ].

In terms of searches running, in most cases, an exhaustive process will not be feasible. Researchers should make sure that the search is effective and efficient to produce results in a timely manner. Cochrane Rapid Reviews Methods Group recommends involving an information specialist and conducting peer review of at least one search strategy [ 11 ]. According to the rapid review guidebook by McMaster University [ 7 ], it is important that RLRs, especially those that support policy and program decisions, are being fed by the results of a body of literature, rather than single studies, when possible. It would result in more generalizable findings applied at the level of a population and serve more realistic findings for program decisions [ 7 ]. It is important to document the search strategy, together with a record of the date and any date limits of the search, so that it can easily be run again, modified, or updated. Furthermore, the information on the individual databases included in platform services should always be reported, as this depends on organizations’ subscriptions and must be included for transparency and repeatability [ 7 , 8 ]. Good solution for RLRs is narrowing the scope or searching a limited number of databases and other sources [ 7 ]. Often, the authors use the PubMed/MEDLINE, Cochrane Library, and Embase databases. In most reviews, two or more databases are searched, and common limits are language (usually restricted to English), date, study design, and geographical area. Some RLRs include searching of grey literature; however, contact with authors is rather uncommon [ 5 , 8 ]. According to the flexible framework for restricted systematic review published by the University of Oxford, the search should be run in at least one major scientific database such as PubMed, and one other source, e.g., Google Scholar [ 9 ]. Grey literature and unpublished evidence may be particularly needed and important for intervention questions. It is related to the fact that studies that do not report the effects of interventions are less likely to be published [ 8 ]. If there is any type of evidence that will not be considered by the RLRs, e.g., reviews or theoretical and conceptual studies, it should also be stated in the protocol together with justification [ 8 ]. Additionally, authors of a practical guide published by WHO suggest using a staged search to identify existing SLRs at the beginning, and then focusing on studies with other designs [ 5 ]. If a low number of citations have been retrieved, it is acceptable to expand searches, remove some of the limits, and add additional databases and sources [ 7 ].

Searching for RLRs is an iterative process, and revising the approach is usually needed [ 7 ]. Changes should be confirmed with stakeholders and should be tracked and reflected in the final report [ 5 ].

The next step in the rapid review is the selection of studies consisting of two phases: screening of titles and abstracts, and analysis of full texts. Prior to screening initiation, it is recommended to conduct a pilot exercise using the same 30–50 abstracts and 5–10 full-texts for the entire screening team in order to calibrate and test the review form [ 11 ]. In contrast to SLRs, it can be done by one reviewer with or without verification by a second one. If verification is performed, usually the second reviewer checks only a subset of records and compares them. Cochrane Group, in contrast, recommends a stricter approach: at least 20% of references should be double-screened at titles and abstracts stage, and while the rest of the references may be screened by one reviewer, the excluded items need to be re-examined by second reviewer; similar approach is used in full-text screening [ 11 ]. This helps to ensure that bias was reduced and that the PICOS criteria are applied in a relevant way [ 5 , 8 , 9 , 11 ]. During the analysis of titles and abstracts, there is no need to report reasons for exclusion; however, they should be tracked for all excluded full texts [ 7 ].

Data extraction and quality assessment

According to the WHO guide, the most common method for data extraction in RLRs is extraction done by a single reviewer with or without partial verification. The authors point out that a reasonable approach is to use a second reviewer to check a random sample of at least 10% of the extractions for accuracy. Dual performance is more necessary for the extraction of quantitative results than for descriptive study information. In contrast, Cochrane group recommends that second reviewer should check the correctness and completeness of all data [ 11 ]. When possible, extractions should be limited to key characteristics and outcomes of the study. The same approach to data extraction is also suggested for a quality assessment process within rapid reviews [ 5 , 9 , 11 ]. Authors of the guidebook from McMaster University highlight that data extraction should be done ideally by two reviewers independently and consensus on the discrepancies should always be reached [ 7 ]. The final decision on the approach to this important step of review should depend on the available time and should also reflect the complexity of the research question [ 9 ].

For screening, analysis of full texts, extractions, and quality assessments, researchers can use information technologies to support them by making these review steps more efficient [ 5 ].

Before data reporting, a reviewer should prepare a document with key message headings, executive summary, background related to the topic and status of the current knowledge, project question, synthesis of findings, conclusions, and recommendations. According to the McMaster University guidebook, a report should be structured in a 1:2:20 format, that is, one page for key messages, two pages for an executive summary, and a full report of up to 20 pages [ 7 ]. All the limitations of the RLRs should be analyzed, and conclusions should be drawn with caution [ 5 ]. The quality of the accumulated evidence and the strength of recommendations can be assessed using, e.g., the GRADE system [ 5 ]. When working on references quoting, researchers should remember to use a primary source, not secondary references [ 7 ]. It would be worth considering the support of some software tools to automate reporting steps. Additionally, any standardization of the process and the usage of templates can support report development and enhance the transparency of the review [ 5 ].

Ideally, all the review steps should be completed during RLRs; however, often some steps may need skipping or will not be completed as thoroughly as should because of time constraints. It is always crucial to decide which steps may be skipped, and which are the key ones, depending on the project [ 7 ]. Guidelines suggest that it may be helpful to invite researchers with experience in the operations of SLRs to participate in the rapid review development [ 5 , 9 ]. As some of the steps will be completed by one reviewer only, it is important to provide them with relevant training at the beginning of the process, as well as during the review, to minimize the risk of mistakes [ 5 ].

Additional information

Depending on the policy goal and available resources and deadlines, methodology of the RLRs may be modified. Wilson et al. [ 10 ] provided extensive guidelines for performing RLR within days (e.g., to inform urgent internal policy discussions and/or management decisions), weeks (e.g., to inform public debates), or months (e.g., to inform policy development cycles that have a longer timeline, but that cannot wait for a traditional full systematic review). These approaches vary in terms of data synthesis, types of considered evidence and project management considerations.

In shortest timeframes, focused questions and subquestions should be formulated, typically to conduct a policy analysis; the report should consist of tables along with a brief narrative summary. Evidence from SLRs is often considered, as well as key informant interviews may be conducted to identify additional literature and insights about the topic, while primary studies and other types of evidence are not typically feasible due to time restrictions. The review would be best conducted with 1–2 reviewers sharing the work, enabling rapid iterations of the review. As for RLRs with longer timeline (weeks), these may use a mix of policy, systems and political analysis. Structure of the review would be similar to shorter RLRs – tabular with short narrative summary, as the timeline does not allow for comprehensive synthesis of data. Besides SLRs, primary studies and other evidence may be feasible in this timeframe, if obtained using the targeted searches in the most relevant databases. The review team should be larger, and standardized procedures for reviewing of the results and data extraction should be applied. In contrast to previous timeframe, merit review process may be feasible. For both timeframes, brief consultations with small transdisciplinary team should be conducted at the beginning and in the final stage of the review to discuss important matters.

For RLRs spanning several months, more comprehensive methodology may be adapted in terms of data synthesis and types of evidence. However, authors advise that review may be best conducted with a small review team in order to allow for more in-depth interpretation and iteration.

Studies analyzing methodology

There have been two interesting publications summarizing the results of Delphi consensus on the RLR methodology identified and included in this review [ 12 , 13 ].

Tricco et al. [ 13 ] first conducted an international survey and scoping review to collect information on the possible approaches to the running of rapid reviews, based on which, they employed a modified Delphi method that included inputs from 113 stakeholders to explore the most optimized approach. Among the six most frequent rapid review approaches (not all detailed here) being evaluated, the approach that combines inclusion of published literature only, a search of more than one database and limitations by date and language, study selection by one analyst, data extraction, and quality assessment by one analyst and one verifier, was perceived as the most feasible approach (72%, 81/113 responses) with the potentially lowest risk of bias (12%, 12/103). The approach ranked as the first one when considering timelines assumes updating of the search from a previously published review, no additional limits on search, studies selection and data extraction done by one reviewer, and no quality assessment. Finally, based on the publication, the most comprehensive RLRs can be made by moving on with the following rules: searching more than one database and grey literature and using date restriction, and assigning one reviewer working on screening, data extraction, and risk of bias assessment ( Table 1 ). Pandor et al. [ 12 ] introduced a decision tool for SelecTing Approaches for Rapid Reviews (STARR) that were produced through the Delphi consensus of international experts through an iterative and rigorous process. Participants were asked to assess the importance of predefined items in four domains related to the rapid review process: interaction with commissioners, understanding the evidence base, data extraction and synthesis methods, and reporting of rapid review methods. All items assigned to four domains achieved > 70% of consensus, and in that way, the first consensus-driven tool has been created that supports authors of RLRs in planning and deciding on approaches.

Six most frequent approaches to RLRs (adapted from Tricco et al. [ 13 ]).

Haby et al. [ 14 ] run searches of 11 databases and two websites and developed a comprehensive overview of the methodology of RLRs. With five SLRs and one RCT being finally included, they identified the following approaches used in RLRs to make them faster than full SLRs: limiting the number and scope of questions, searching fewer databases, limited searching of grey literature, restrictions on language and date (e.g., English only, most recent publications), updating the existing SLRs, eliminating or limiting hand searches of reference lists, noniterative search strategies, eliminating consultation with experts, limiting dual study selection, data extraction and quality assessment, minimal data synthesis with short concise conclusions or recommendations. All the SLRs included in this review were consistent in stating that no agreed definition of rapid reviews is available, and there is still no final agreement on the best methodological rules to be followed.

Gordon et al. [ 4 ] explained the advantages of performing a focused review and provided 12 tips for its conduction. They define focused reviews as ‘a form of knowledge synthesis in which the components of the systematic process are applied to facilitate the analysis of a focused research question’. The first tip presented by the authors is related to deciding if a focused review is a right solution for the considered project. RLRs will suit emerging topics, approaches, or assessments where early synthesis can support doctors, policymakers, etc., but also can direct future research. The second, third, and fourth tips highlight the importance of running preliminary searches and considering narrowing the results by using reasonable constraints taking into account the local context, problems, efficiency perspectives, and available time. Further tips include creating a team of experienced reviewers working on the RLRs, thinking about the target journal from the beginning of work on the rapid review, registering the search protocol on the PROSPERO registry, and the need for contacting authors of papers when data available in publications are missing or incongruent. The last three tips are related to the choice of evidence synthesis method, using the visual presentation of data, and considering and describing all the limitations of the focused review.

Finally, a new publication by Tricco et al. from 2022, describing JBI position statement [ 15 ] underlined that for the time being, there is no specific tool for critical appraisal of the RLR’s methodological quality. Instead, reviewers may use available tools to assess the risk of bias or quality of SLRs, like ROBIS, the JBI critical appraisal tools, or the assessment of multiple systematic reviews (AMSTAR).

Inconsistency in the definitions and methodologies of RLR

Although RLR was broadly perceived as an approach to quicken the conduct of conventional SLR, there is a lack of consensus on the formal definition of the RLR, so as to the best approaches to perform it. Only in 2021, a study proposing unified definition was published; however, it is important to note that the most accurate definition was only matching slightly over 50% of papers analysed by the authors, which underlines the lack of homogeneity in the field [ 11 ]. The evidence-based supporting methods are evolving, and more evidence is needed to define the most robust approaches [ 5 ].

Diverse terms are used to describe the RLR, including ‘rapid review’, focused systematic review’, ‘quick scoping reviews’, and ‘rapid evidence assessments’. Although the general principles of conducting RLR are to accelerate the whole process, complexity was seen in the methodologies used for RLRs, as reflected in this study. Also, inconsistencies related to the scope of the questions, search strategies, inclusion criteria, study screening, full-text review, quality assessment, and evidence presentation were implied. All these factors may hamper decision-making about optimal methodologies for conducting rapid reviews, and as a result, the efficiency of RLR might be decreased. Additionally, researchers may tend to report the methodology of their reviews without a sufficient level of detail, making it difficult to appraise the quality and robustness of their work.

Advantages and weaknesses of RLR

Although RLR used simplified approaches for evidence synthesis compared with SLR, the methodologies for RLR should be replicable, rigorous, and transparent to the greatest extent [ 16 ]. When time and resources are limited, RLR could be a practical and efficient tool to provide the summary of evidence that is critical for making rapid clinical or policy-related decisions [ 5 ]. Focusing on specific questions that are of controversy or special interest could be powerful in reaffirming whether the existing recommendation statements are still appropriate [ 17 ].

The weakness of RLR should also be borne in mind, and the trade-off of using RLR should be carefully considered regarding the thoroughness of the search, breadth of a research question, and depth of analysis [ 18 ]. If allowed, SLR is preferred over RLR considering that some relevant studies might be omitted with narrowed search strategies and simplified screening process [ 14 ]. Additionally, omitting the quality assessment of included studies could result in an increased risk of bias, making the comprehensiveness of RLR compromised [ 13 ]. Furthermore, in situations that require high accuracy, for example, where a small relative difference in an intervention has great impacts, for the purpose of drafting clinical guidelines, or making licensing decisions, a comprehensive SLR may remain the priority [ 19 ]. Therefore, clear communications with policymakers are recommended to reach an agreement on whether an RLR is justified and whether the methodologies of RLR are acceptable to address the unanswered questions [ 18 ].

Disclosure statement

No potential conflict of interest was reported by the author(s).