published literature review

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Literature Reviews

What this handout is about.

This handout will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.

Introduction

OK. You’ve got to write a literature review. You dust off a novel and a book of poetry, settle down in your chair, and get ready to issue a “thumbs up” or “thumbs down” as you leaf through the pages. “Literature review” done. Right?

Wrong! The “literature” of a literature review refers to any collection of materials on a topic, not necessarily the great literary texts of the world. “Literature” could be anything from a set of government pamphlets on British colonial methods in Africa to scholarly articles on the treatment of a torn ACL. And a review does not necessarily mean that your reader wants you to give your personal opinion on whether or not you liked these sources.

What is a literature review, then?

A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period.

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

But how is a literature review different from an academic research paper?

The main focus of an academic research paper is to develop a new argument, and a research paper is likely to contain a literature review as one of its parts. In a research paper, you use the literature as a foundation and as support for a new insight that you contribute. The focus of a literature review, however, is to summarize and synthesize the arguments and ideas of others without adding new contributions.

Why do we write literature reviews?

Literature reviews provide you with a handy guide to a particular topic. If you have limited time to conduct research, literature reviews can give you an overview or act as a stepping stone. For professionals, they are useful reports that keep them up to date with what is current in the field. For scholars, the depth and breadth of the literature review emphasizes the credibility of the writer in his or her field. Literature reviews also provide a solid background for a research paper’s investigation. Comprehensive knowledge of the literature of the field is essential to most research papers.

Who writes these things, anyway?

Literature reviews are written occasionally in the humanities, but mostly in the sciences and social sciences; in experiment and lab reports, they constitute a section of the paper. Sometimes a literature review is written as a paper in itself.

Let’s get to it! What should I do before writing the literature review?

If your assignment is not very specific, seek clarification from your instructor:

• Roughly how many sources should you include?
• What types of sources (books, journal articles, websites)?
• Should you summarize, synthesize, or critique your sources by discussing a common theme or issue?
• Should you evaluate your sources?
• Should you provide subheadings and other background information, such as definitions and/or a history?

Find models

Look for other literature reviews in your area of interest or in the discipline and read them to get a sense of the types of themes you might want to look for in your own research or ways to organize your final review. You can simply put the word “review” in your search engine along with your other topic terms to find articles of this type on the Internet or in an electronic database. The bibliography or reference section of sources you’ve already read are also excellent entry points into your own research.

Narrow your topic

There are hundreds or even thousands of articles and books on most areas of study. The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the material. Your instructor will probably not expect you to read everything that’s out there on the topic, but you’ll make your job easier if you first limit your scope.

Keep in mind that UNC Libraries have research guides and to databases relevant to many fields of study. You can reach out to the subject librarian for a consultation: https://library.unc.edu/support/consultations/ .

And don’t forget to tap into your professor’s (or other professors’) knowledge in the field. Ask your professor questions such as: “If you had to read only one book from the 90’s on topic X, what would it be?” Questions such as this help you to find and determine quickly the most seminal pieces in the field.

Consider whether your sources are current

Some disciplines require that you use information that is as current as possible. In the sciences, for instance, treatments for medical problems are constantly changing according to the latest studies. Information even two years old could be obsolete. However, if you are writing a review in the humanities, history, or social sciences, a survey of the history of the literature may be what is needed, because what is important is how perspectives have changed through the years or within a certain time period. Try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to consider what is currently of interest to scholars in this field and what is not.

Strategies for writing the literature review

Find a focus.

A literature review, like a term paper, is usually organized around ideas, not the sources themselves as an annotated bibliography would be organized. This means that you will not just simply list your sources and go into detail about each one of them, one at a time. No. As you read widely but selectively in your topic area, consider instead what themes or issues connect your sources together. Do they present one or different solutions? Is there an aspect of the field that is missing? How well do they present the material and do they portray it according to an appropriate theory? Do they reveal a trend in the field? A raging debate? Pick one of these themes to focus the organization of your review.

Convey it to your reader

A literature review may not have a traditional thesis statement (one that makes an argument), but you do need to tell readers what to expect. Try writing a simple statement that lets the reader know what is your main organizing principle. Here are a couple of examples:

The current trend in treatment for congestive heart failure combines surgery and medicine. More and more cultural studies scholars are accepting popular media as a subject worthy of academic consideration.

Consider organization

You’ve got a focus, and you’ve stated it clearly and directly. Now what is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? And in what order should you present them? Develop an organization for your review at both a global and local level:

First, cover the basic categories

Just like most academic papers, literature reviews also must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper. The following provides a brief description of the content of each:

• Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern.
• Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).
• Conclusions/Recommendations: Discuss what you have drawn from reviewing literature so far. Where might the discussion proceed?

Organizing the body

Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further.

To help you come up with an overall organizational framework for your review, consider the following scenario:

You’ve decided to focus your literature review on materials dealing with sperm whales. This is because you’ve just finished reading Moby Dick, and you wonder if that whale’s portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 1980’s. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 1968 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick, so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel.

Now consider some typical ways of organizing the sources into a review:

• Chronological: If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (1968), and finally the biology articles (1980s) and the recent articles on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus.
• By publication: Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies.
• By trend: A better way to organize the above sources chronologically is to examine the sources under another trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Under this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote a century apart.
• Thematic: Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For instance, the sperm whale review could focus on the development of the harpoon for whale hunting. While the study focuses on one topic, harpoon technology, it will still be organized chronologically. The only difference here between a “chronological” and a “thematic” approach is what is emphasized the most: the development of the harpoon or the harpoon technology.But more authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as “evil” in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.
• Methodological: A methodological approach differs from the two above in that the focusing factor usually does not have to do with the content of the material. Instead, it focuses on the “methods” of the researcher or writer. For the sperm whale project, one methodological approach would be to look at cultural differences between the portrayal of whales in American, British, and French art work. Or the review might focus on the economic impact of whaling on a community. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed. Once you’ve decided on the organizational method for the body of the review, the sections you need to include in the paper should be easy to figure out. They should arise out of your organizational strategy. In other words, a chronological review would have subsections for each vital time period. A thematic review would have subtopics based upon factors that relate to the theme or issue.

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

• Current Situation: Information necessary to understand the topic or focus of the literature review.
• History: The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Methods and/or Standards: The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

Begin composing

Once you’ve settled on a general pattern of organization, you’re ready to write each section. There are a few guidelines you should follow during the writing stage as well. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as “writer,” “pedestrian,” and “persons.” The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine “generic” condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, “Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense,” Women and Language19:2).

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review’s focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though, if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton’s study, but then synthesize it by rephrasing the study’s significance and relating it to their own work.

Keep your own voice

While the literature review presents others’ ideas, your voice (the writer’s) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author’s information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil’s. For more information, please see our handout on plagiarism .

Revise, revise, revise

Draft in hand? Now you’re ready to revise. Spending a lot of time revising is a wise idea, because your main objective is to present the material, not the argument. So check over your review again to make sure it follows the assignment and/or your outline. Then, just as you would for most other academic forms of writing, rewrite or rework the language of your review so that you’ve presented your information in the most concise manner possible. Be sure to use terminology familiar to your audience; get rid of unnecessary jargon or slang. Finally, double check that you’ve documented your sources and formatted the review appropriately for your discipline. For tips on the revising and editing process, see our handout on revising drafts .

Works consulted

We consulted these works while writing this handout. This is not a comprehensive list of resources on the handout’s topic, and we encourage you to do your own research to find additional publications. Please do not use this list as a model for the format of your own reference list, as it may not match the citation style you are using. For guidance on formatting citations, please see the UNC Libraries citation tutorial . We revise these tips periodically and welcome feedback.

Anson, Chris M., and Robert A. Schwegler. 2010. The Longman Handbook for Writers and Readers , 6th ed. New York: Longman.

Jones, Robert, Patrick Bizzaro, and Cynthia Selfe. 1997. The Harcourt Brace Guide to Writing in the Disciplines . New York: Harcourt Brace.

Lamb, Sandra E. 1998. How to Write It: A Complete Guide to Everything You’ll Ever Write . Berkeley: Ten Speed Press.

Rosen, Leonard J., and Laurence Behrens. 2003. The Allyn & Bacon Handbook , 5th ed. New York: Longman.

Troyka, Lynn Quittman, and Doug Hesse. 2016. Simon and Schuster Handbook for Writers , 11th ed. London: Pearson.

You may reproduce it for non-commercial use if you use the entire handout and attribute the source: The Writing Center, University of North Carolina at Chapel Hill

Make a Gift

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

• Library Homepage

Literature Review: The What, Why and How-to Guide: Literature Reviews?

• Literature Reviews?
• Strategies to Finding Sources
• Keeping up with Research!
• Evaluating Sources & Literature Reviews
• Organizing for Writing
• Writing Literature Review
• Other Academic Writings

What is a Literature Review?

So, what is a literature review .

"A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available or a set of summaries." - Quote from Taylor, D. (n.d)."The Literature Review: A Few Tips on Conducting it".

• Citation: "The Literature Review: A Few Tips on Conducting it"

What kinds of literature reviews are written?

Each field has a particular way to do reviews for academic research literature. In the social sciences and humanities the most common are:

• Narrative Reviews: The purpose of this type of review is to describe the current state of the research on a specific research topic and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weaknesses, and gaps are identified. The review ends with a conclusion section that summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.
• Book review essays/ Historiographical review essays : A type of literature review typical in History and related fields, e.g., Latin American studies. For example, the Latin American Research Review explains that the purpose of this type of review is to “(1) to familiarize readers with the subject, approach, arguments, and conclusions found in a group of books whose common focus is a historical period; a country or region within Latin America; or a practice, development, or issue of interest to specialists and others; (2) to locate these books within current scholarship, critical methodologies, and approaches; and (3) to probe the relation of these new books to previous work on the subject, especially canonical texts. Unlike individual book reviews, the cluster reviews found in LARR seek to address the state of the field or discipline and not solely the works at issue.” - LARR

What are the Goals of Creating a Literature Review?

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 
• Baumeister, R.F. & Leary, M.R. (1997). "Writing narrative literature reviews," Review of General Psychology , 1(3), 311-320.

When do you need to write a Literature Review?

• When writing a prospectus or a thesis/dissertation
• When writing a research paper
• When writing a grant proposal

In all these cases you need to dedicate a chapter in these works to showcase what has been written about your research topic and to point out how your own research will shed new light into a body of scholarship.

Where I can find examples of Literature Reviews?

Note:  In the humanities, even if they don't use the term "literature review", they may have a dedicated  chapter that reviewed the "critical bibliography" or they incorporated that review in the introduction or first chapter of the dissertation, book, or article.

• UCSB electronic theses and dissertations In partnership with the Graduate Division, the UC Santa Barbara Library is making available theses and dissertations produced by UCSB students. Currently included in ADRL are theses and dissertations that were originally filed electronically, starting in 2011. In future phases of ADRL, all theses and dissertations created by UCSB students may be digitized and made available.

Where to Find Standalone Literature Reviews

Literature reviews are also written as standalone articles as a way to survey a particular research topic in-depth. This type of literature review looks at a topic from a historical perspective to see how the understanding of the topic has changed over time. 

• Find e-Journals for Standalone Literature Reviews The best way to get familiar with and to learn how to write literature reviews is by reading them. You can use our Journal Search option to find journals that specialize in publishing literature reviews from major disciplines like anthropology, sociology, etc. Usually these titles are called, "Annual Review of [discipline name] OR [Discipline name] Review. This option works best if you know the title of the publication you are looking for. Below are some examples of these journals! more... less... Journal Search can be found by hovering over the link for Research on the library website.

Social Sciences

• Annual Review of Anthropology
• Annual Review of Political Science
• Annual Review of Sociology
• Ethnic Studies Review

Hard science and health sciences:

• Annual Review of Biomedical Data Science
• Annual Review of Materials Science
• Systematic Review From journal site: "The journal Systematic Reviews encompasses all aspects of the design, conduct, and reporting of systematic reviews" in the health sciences.
• << Previous: Overview
• Next: Strategies to Finding Sources >>
• Last Updated: Mar 5, 2024 11:44 AM
• URL: https://guides.library.ucsb.edu/litreview

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 6 May 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

How To Write An A-Grade Literature Review

3 straightforward steps (with examples) + free template.

By: Derek Jansen (MBA) | Expert Reviewed By: Dr. Eunice Rautenbach | October 2019

Quality research is about building onto the existing work of others , “standing on the shoulders of giants”, as Newton put it. The literature review chapter of your dissertation, thesis or research project is where you synthesise this prior work and lay the theoretical foundation for your own research.

Long story short, this chapter is a pretty big deal, which is why you want to make sure you get it right . In this post, I’ll show you exactly how to write a literature review in three straightforward steps, so you can conquer this vital chapter (the smart way).

Overview: The Literature Review Process

• Understanding the “ why “
• Finding the relevant literature
• Cataloguing and synthesising the information
• Outlining & writing up your literature review
• Example of a literature review

But first, the “why”…

Before we unpack how to write the literature review chapter, we’ve got to look at the why . To put it bluntly, if you don’t understand the function and purpose of the literature review process, there’s no way you can pull it off well. So, what exactly is the purpose of the literature review?

Well, there are (at least) four core functions:

• For you to gain an understanding (and demonstrate this understanding) of where the research is at currently, what the key arguments and disagreements are.
• For you to identify the gap(s) in the literature and then use this as justification for your own research topic.
• To help you build a conceptual framework for empirical testing (if applicable to your research topic).
• To inform your methodological choices and help you source tried and tested questionnaires (for interviews ) and measurement instruments (for surveys ).

Most students understand the first point but don’t give any thought to the rest. To get the most from the literature review process, you must keep all four points front of mind as you review the literature (more on this shortly), or you’ll land up with a wonky foundation.

Okay – with the why out the way, let’s move on to the how . As mentioned above, writing your literature review is a process, which I’ll break down into three steps:

• Finding the most suitable literature
• Understanding , distilling and organising the literature
• Planning and writing up your literature review chapter

Importantly, you must complete steps one and two before you start writing up your chapter. I know it’s very tempting, but don’t try to kill two birds with one stone and write as you read. You’ll invariably end up wasting huge amounts of time re-writing and re-shaping, or you’ll just land up with a disjointed, hard-to-digest mess . Instead, you need to read first and distil the information, then plan and execute the writing.

Step 1: Find the relevant literature

Naturally, the first step in the literature review journey is to hunt down the existing research that’s relevant to your topic. While you probably already have a decent base of this from your research proposal , you need to expand on this substantially in the dissertation or thesis itself.

Essentially, you need to be looking for any existing literature that potentially helps you answer your research question (or develop it, if that’s not yet pinned down). There are numerous ways to find relevant literature, but I’ll cover my top four tactics here. I’d suggest combining all four methods to ensure that nothing slips past you:

Method 1 – Google Scholar Scrubbing

Google’s academic search engine, Google Scholar , is a great starting point as it provides a good high-level view of the relevant journal articles for whatever keyword you throw at it. Most valuably, it tells you how many times each article has been cited, which gives you an idea of how credible (or at least, popular) it is. Some articles will be free to access, while others will require an account, which brings us to the next method.

Method 2 – University Database Scrounging

Generally, universities provide students with access to an online library, which provides access to many (but not all) of the major journals.

So, if you find an article using Google Scholar that requires paid access (which is quite likely), search for that article in your university’s database – if it’s listed there, you’ll have access. Note that, generally, the search engine capabilities of these databases are poor, so make sure you search for the exact article name, or you might not find it.

Method 3 – Journal Article Snowballing

At the end of every academic journal article, you’ll find a list of references. As with any academic writing, these references are the building blocks of the article, so if the article is relevant to your topic, there’s a good chance a portion of the referenced works will be too. Do a quick scan of the titles and see what seems relevant, then search for the relevant ones in your university’s database.

Method 4 – Dissertation Scavenging

Similar to Method 3 above, you can leverage other students’ dissertations. All you have to do is skim through literature review chapters of existing dissertations related to your topic and you’ll find a gold mine of potential literature. Usually, your university will provide you with access to previous students’ dissertations, but you can also find a much larger selection in the following databases:

• Open Access Theses & Dissertations
• Stanford SearchWorks

Keep in mind that dissertations and theses are not as academically sound as published, peer-reviewed journal articles (because they’re written by students, not professionals), so be sure to check the credibility of any sources you find using this method. You can do this by assessing the citation count of any given article in Google Scholar. If you need help with assessing the credibility of any article, or with finding relevant research in general, you can chat with one of our Research Specialists .

Alright – with a good base of literature firmly under your belt, it’s time to move onto the next step.

Need a helping hand?

Step 2: Log, catalogue and synthesise

Once you’ve built a little treasure trove of articles, it’s time to get reading and start digesting the information – what does it all mean?

While I present steps one and two (hunting and digesting) as sequential, in reality, it’s more of a back-and-forth tango – you’ll read a little , then have an idea, spot a new citation, or a new potential variable, and then go back to searching for articles. This is perfectly natural – through the reading process, your thoughts will develop , new avenues might crop up, and directional adjustments might arise. This is, after all, one of the main purposes of the literature review process (i.e. to familiarise yourself with the current state of research in your field).

As you’re working through your treasure chest, it’s essential that you simultaneously start organising the information. There are three aspects to this:

• Logging reference information
• Building an organised catalogue
• Distilling and synthesising the information

I’ll discuss each of these below:

2.1 – Log the reference information

As you read each article, you should add it to your reference management software. I usually recommend Mendeley for this purpose (see the Mendeley 101 video below), but you can use whichever software you’re comfortable with. Most importantly, make sure you load EVERY article you read into your reference manager, even if it doesn’t seem very relevant at the time.

2.2 – Build an organised catalogue

In the beginning, you might feel confident that you can remember who said what, where, and what their main arguments were. Trust me, you won’t. If you do a thorough review of the relevant literature (as you must!), you’re going to read many, many articles, and it’s simply impossible to remember who said what, when, and in what context . Also, without the bird’s eye view that a catalogue provides, you’ll miss connections between various articles, and have no view of how the research developed over time. Simply put, it’s essential to build your own catalogue of the literature.

I would suggest using Excel to build your catalogue, as it allows you to run filters, colour code and sort – all very useful when your list grows large (which it will). How you lay your spreadsheet out is up to you, but I’d suggest you have the following columns (at minimum):

• Author, date, title – Start with three columns containing this core information. This will make it easy for you to search for titles with certain words, order research by date, or group by author.
• Categories or keywords – You can either create multiple columns, one for each category/theme and then tick the relevant categories, or you can have one column with keywords.
• Key arguments/points – Use this column to succinctly convey the essence of the article, the key arguments and implications thereof for your research.
• Context – Note the socioeconomic context in which the research was undertaken. For example, US-based, respondents aged 25-35, lower- income, etc. This will be useful for making an argument about gaps in the research.
• Methodology – Note which methodology was used and why. Also, note any issues you feel arise due to the methodology. Again, you can use this to make an argument about gaps in the research.
• Quotations – Note down any quoteworthy lines you feel might be useful later.
• Notes – Make notes about anything not already covered. For example, linkages to or disagreements with other theories, questions raised but unanswered, shortcomings or limitations, and so forth.

If you’d like, you can try out our free catalog template here (see screenshot below).

2.3 – Digest and synthesise

Most importantly, as you work through the literature and build your catalogue, you need to synthesise all the information in your own mind – how does it all fit together? Look for links between the various articles and try to develop a bigger picture view of the state of the research. Some important questions to ask yourself are:

• What answers does the existing research provide to my own research questions ?
• Which points do the researchers agree (and disagree) on?
• How has the research developed over time?
• Where do the gaps in the current research lie?

To help you develop a big-picture view and synthesise all the information, you might find mind mapping software such as Freemind useful. Alternatively, if you’re a fan of physical note-taking, investing in a large whiteboard might work for you.

Step 3: Outline and write it up!

Once you’re satisfied that you have digested and distilled all the relevant literature in your mind, it’s time to put pen to paper (or rather, fingers to keyboard). There are two steps here – outlining and writing:

3.1 – Draw up your outline

Having spent so much time reading, it might be tempting to just start writing up without a clear structure in mind. However, it’s critically important to decide on your structure and develop a detailed outline before you write anything. Your literature review chapter needs to present a clear, logical and an easy to follow narrative – and that requires some planning. Don’t try to wing it!

Naturally, you won’t always follow the plan to the letter, but without a detailed outline, you’re more than likely going to end up with a disjointed pile of waffle , and then you’re going to spend a far greater amount of time re-writing, hacking and patching. The adage, “measure twice, cut once” is very suitable here.

In terms of structure, the first decision you’ll have to make is whether you’ll lay out your review thematically (into themes) or chronologically (by date/period). The right choice depends on your topic, research objectives and research questions, which we discuss in this article .

Once that’s decided, you need to draw up an outline of your entire chapter in bullet point format. Try to get as detailed as possible, so that you know exactly what you’ll cover where, how each section will connect to the next, and how your entire argument will develop throughout the chapter. Also, at this stage, it’s a good idea to allocate rough word count limits for each section, so that you can identify word count problems before you’ve spent weeks or months writing!

PS – check out our free literature review chapter template…

3.2 – Get writing

With a detailed outline at your side, it’s time to start writing up (finally!). At this stage, it’s common to feel a bit of writer’s block and find yourself procrastinating under the pressure of finally having to put something on paper. To help with this, remember that the objective of the first draft is not perfection – it’s simply to get your thoughts out of your head and onto paper, after which you can refine them. The structure might change a little, the word count allocations might shift and shuffle, and you might add or remove a section – that’s all okay. Don’t worry about all this on your first draft – just get your thoughts down on paper.

Once you’ve got a full first draft (however rough it may be), step away from it for a day or two (longer if you can) and then come back at it with fresh eyes. Pay particular attention to the flow and narrative – does it fall fit together and flow from one section to another smoothly? Now’s the time to try to improve the linkage from each section to the next, tighten up the writing to be more concise, trim down word count and sand it down into a more digestible read.

Once you’ve done that, give your writing to a friend or colleague who is not a subject matter expert and ask them if they understand the overall discussion. The best way to assess this is to ask them to explain the chapter back to you. This technique will give you a strong indication of which points were clearly communicated and which weren’t. If you’re working with Grad Coach, this is a good time to have your Research Specialist review your chapter.

Finally, tighten it up and send it off to your supervisor for comment. Some might argue that you should be sending your work to your supervisor sooner than this (indeed your university might formally require this), but in my experience, supervisors are extremely short on time (and often patience), so, the more refined your chapter is, the less time they’ll waste on addressing basic issues (which you know about already) and the more time they’ll spend on valuable feedback that will increase your mark-earning potential.

Literature Review Example

In the video below, we unpack an actual literature review so that you can see how all the core components come together in reality.

Let’s Recap

In this post, we’ve covered how to research and write up a high-quality literature review chapter. Let’s do a quick recap of the key takeaways:

• It is essential to understand the WHY of the literature review before you read or write anything. Make sure you understand the 4 core functions of the process.
• The first step is to hunt down the relevant literature . You can do this using Google Scholar, your university database, the snowballing technique and by reviewing other dissertations and theses.
• Next, you need to log all the articles in your reference manager , build your own catalogue of literature and synthesise all the research.
• Following that, you need to develop a detailed outline of your entire chapter – the more detail the better. Don’t start writing without a clear outline (on paper, not in your head!)
• Write up your first draft in rough form – don’t aim for perfection. Remember, done beats perfect.
• Refine your second draft and get a layman’s perspective on it . Then tighten it up and submit it to your supervisor.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

38 Comments

Thank you very much. This page is an eye opener and easy to comprehend.

This is awesome!

I wish I come across GradCoach earlier enough.

But all the same I’ll make use of this opportunity to the fullest.

Thank you for this good job.

Keep it up!

You’re welcome, Yinka. Thank you for the kind words. All the best writing your literature review.

Thank you for a very useful literature review session. Although I am doing most of the steps…it being my first masters an Mphil is a self study and one not sure you are on the right track. I have an amazing supervisor but one also knows they are super busy. So not wanting to bother on the minutae. Thank you.

You’re most welcome, Renee. Good luck with your literature review 🙂

This has been really helpful. Will make full use of it. 🙂

Thank you Gradcoach.

Really agreed. Admirable effort

thank you for this beautiful well explained recap.

Thank you so much for your guide of video and other instructions for the dissertation writing.

It is instrumental. It encouraged me to write a dissertation now.

Thank you the video was great – from someone that knows nothing thankyou

an amazing and very constructive way of presetting a topic, very useful, thanks for the effort,

It is timely

It is very good video of guidance for writing a research proposal and a dissertation. Since I have been watching and reading instructions, I have started my research proposal to write. I appreciate to Mr Jansen hugely.

I learn a lot from your videos. Very comprehensive and detailed.

Thank you for sharing your knowledge. As a research student, you learn better with your learning tips in research

I was really stuck in reading and gathering information but after watching these things are cleared thanks, it is so helpful.

Really helpful, Thank you for the effort in showing such information

This is super helpful thank you very much.

Thank you for this whole literature writing review.You have simplified the process.

I’m so glad I found GradCoach. Excellent information, Clear explanation, and Easy to follow, Many thanks Derek!

You’re welcome, Maithe. Good luck writing your literature review 🙂

Thank you Coach, you have greatly enriched and improved my knowledge

Great piece, so enriching and it is going to help me a great lot in my project and thesis, thanks so much

This is THE BEST site for ANYONE doing a masters or doctorate! Thank you for the sound advice and templates. You rock!

Thanks, Stephanie 🙂

This is mind blowing, the detailed explanation and simplicity is perfect.

I am doing two papers on my final year thesis, and I must stay I feel very confident to face both headlong after reading this article.

thank you so much.

if anyone is to get a paper done on time and in the best way possible, GRADCOACH is certainly the go to area!

This is very good video which is well explained with detailed explanation

Thank you excellent piece of work and great mentoring

Thanks, it was useful

Thank you very much. the video and the information were very helpful.

Good morning scholar. I’m delighted coming to know you even before the commencement of my dissertation which hopefully is expected in not more than six months from now. I would love to engage my study under your guidance from the beginning to the end. I love to know how to do good job

Thank you so much Derek for such useful information on writing up a good literature review. I am at a stage where I need to start writing my one. My proposal was accepted late last year but I honestly did not know where to start

Like the name of your YouTube implies you are GRAD (great,resource person, about dissertation). In short you are smart enough in coaching research work.

This is a very well thought out webpage. Very informative and a great read.

Very timely.

I appreciate.

Very comprehensive and eye opener for me as beginner in postgraduate study. Well explained and easy to understand. Appreciate and good reference in guiding me in my research journey. Thank you

Thank you. I requested to download the free literature review template, however, your website wouldn’t allow me to complete the request or complete a download. May I request that you email me the free template? Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Literature Review: Examples of Published Literature Reviews

• Sample Searches
• Examples of Published Literature Reviews
• Researching Your Topic
• Subject Searching
• Google Scholar
• Track Your Work
• Citation Managers This link opens in a new window
• Citation Guides This link opens in a new window
• Tips on Writing Your Literature Review This link opens in a new window
• Research Help

Ask a Librarian

Chat with a Librarian

Lisle: (630) 829-6057 Mesa: (480) 878-7514 Toll Free: (877) 575-6050 Email: [email protected]

Book a Research Consultation Library Hours

• Example of a Literature Review in History
• Comparing the Annotated Bibliography to the Literature Review
• Literature Review Guidelines Literature Review (Historiographic Essay): Making sense of what has been written on your topic.

To find examples of published literature reviews in your field or niche, try searching ProQuest Dissertations and Theses by keyword, advisor, or subject. 

• Dissertations & Theses Theses and dissertations from around the world covering all areas of study.
• << Previous: Sample Searches
• Next: Researching Your Topic >>
• Last Updated: Apr 25, 2024 3:34 PM
• URL: https://researchguides.ben.edu/lit-review

Kindlon Hall 5700 College Rd. Lisle, IL 60532 (630) 829-6050

Gillett Hall 225 E. Main St. Mesa, AZ 85201 (480) 878-7514

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: May 9, 2024 11:05 AM
• URL: https://libguides.usc.edu/writingguide

Get science-backed answers as you write with Paperpal's Research feature

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

• What is the purpose of literature review? 
• a. Habitat Loss and Species Extinction: 
• b. Range Shifts and Phenological Changes: 
• c. Ocean Acidification and Coral Reefs: 
• d. Adaptive Strategies and Conservation Efforts: 
• How to write a good literature review 
• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• Frequently asked questions 

What is a literature review?

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

What is the purpose of literature review?

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

• Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 
• Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field. 
• Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 
• Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 
• Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 
• Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

a. Habitat Loss and Species Extinction:

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

b. Range Shifts and Phenological Changes:

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

c. Ocean Acidification and Coral Reefs:

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

d. Adaptive Strategies and Conservation Efforts:

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

How to write a good literature review

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. 

Frequently asked questions

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• Life Sciences Papers: 9 Tips for Authors Writing in Biological Sciences
• What is an Argumentative Essay? How to Write It (With Examples)

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, how paperpal’s research feature helps you develop and..., how paperpal is enhancing academic productivity and accelerating..., how to write a successful book chapter for..., academic editing: how to self-edit academic text with..., 4 ways paperpal encourages responsible writing with ai, what are scholarly sources and where can you..., how to write a hypothesis types and examples , measuring academic success: definition & strategies for excellence, what is academic writing: tips for students, why traditional editorial process needs an upgrade.

Writing a Literature Review: Home

      A literature review is a guide to the published information on a topic. While a literature review summarizes each author’s ideas and contributions, it is not just an alphabetical or numbered list. Sources are usually grouped into subtopics or ideas important to the topic. For example, a review of the literature on crop circles might be divided into specific geographic areas or it might represent the skeptic’s viewpoint as well as the believer’s.

     Here is an excerpt from The Writing Center at UNC-Chapel Hill:

A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period.

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

     Literature reviews are very valuable to researchers who need an overview of what’s been written about a topic, but they do not count as scholarly journal articles when you are collecting sources for a term paper.

WRITING A LITERATURE REVIEW: CHOOSING A TOPIC

      I f you have to write a literature review, you need to take care that the topic you choose is neither too broad or too narrow. You could spend the rest of your life doing a literature review on war whereas a literature review on women in the Gulf War might be adequately addressed in several pages. Conversely, you might choose a topic and discover that almost nothing has been written about it. Consult with a reference librarian or your professor to be certain you’ve chosen wisely.

WRITING A LITERATURE REVIEW: SELECTION CRITERIA

      N ext, decide what types of literature will be included. Will you use books, journal articles, magazine articles, newspaper articles, web sites, and/or something else? How old should the material be? How many items will be included?

WRITING A LITERATURE REVIEW: THESIS STATEMENT

      T he thesis statement for a literature review argues neither for or against a particular position but for a particular perspective on the literature. Continuing on with our example of crop circles, a possible thesis statement might be:

The phenomenon known as “crop circles” has become worthy of investigation by the scientific community.

The hallmarks of scientific literature on hoaxes are revealed in the literature on the phenomenon known as “crop circles.”

SAMPLE LITERATURE REVIEWS

• Enrichment for Captive TIgers(Panthera tigris): Current Knowledge and Future Directions

Notice how the literature reviews below address different, specific aspects of oil spills:

• Offshore Oil Spill Response Practices and Emerging Challenges
• Oil Spills and Their Impacts on Sand Beach Invertebrate Communities: A Literature Review
• Oil Spill Detection by Satellite Remote Sensing

A Crop Circle

• What is "The Literature"?

FURTHER ASSISTANCE

      T here is a lot more to writing a literature review than can be easily covered in this guide. Get all the details you can from your professor. Find a literature review to use as a model. Consult one of these books or Web sites:

• Preparing Literature Reviews : Qualitative and Quantitative Approaches by M. Ling Pan.
• Writing Literature Reviews : a Guide for Students of the Social and Behavioral Sciences by Jose L. Galvan.
• The UNC Writing Center: Literature Reviews
• The Literature Review: A Few Tips On Conducting It
• Social Work Literature Review Guidelines

• Last Updated: Nov 23, 2020 12:18 PM
• URL: https://libguides.csusb.edu/literaturereview

• UNC Libraries
• HSL Academic Process
• Searching PubMed
• Literature Reviews

Searching PubMed: Literature Reviews

Created by health science librarians.

• Basic Searches
• Filters and Narrowing Searches
• Find Full-Text Articles
• Save Search Results
• Saving Searches & Creating Alerts
• My NCBI Accounts

Section Objective

What is a literature review, clearly stated research question, search terms, searching worksheets, boolean and / or.

The content in the Literature Review section defines the literature review purpose and process, explains using the PICO format to ask a clear research question, and demonstrates how to evaluate and modify search results to improve the accuracy of the retrieval.

A literature review seeks to identify, analyze and summarize the published research literature about a specific topic.  Literature reviews are assigned as course projects; included as the introductory part of master's and PhD theses; and are conducted before undertaking any new scientific research project.

The purpose of a literature review is to establish what is currently known about a specific topic and to evaluate the strength of the evidence upon which that knowledge is based. A review of a clinical topic may identify implications for clinical practice. Literature reviews also identify areas of a topic that need further research.

A systematic review is a literature review that follows a rigorous process to find all of the research conducted on a topic and then critically appraises the research methods of the highest quality reports. These reviews track and report their search and appraisal methods in addition to providing a summary of the knowledge established by the appraised research.

The UNC Writing Center provides a nice summary of what to consider when writing a literature review for a class assignment. The online book, Doing a literature review in health and social care : a practical guide (2010), is a good resource for more information on this topic.

Obviously, the quality of the search process will determine the quality of all literature reviews. Anyone undertaking a literature review on a new topic would benefit from meeting with a librarian to discuss search strategies. A consultaiton with a librarian is strongly recommended for anyone undertaking a systematic review.

Use the email form on our Ask a Librarian page to arrange a meeting with a librarian.

The first step to a successful literature review search is to state your research question as clearly as possible.

It is important to:

• be as specific as possible
• include all aspects of your question

Clinical and social science questions often have these aspects (PICO):

• People/population/problem  (What are the characteristics of the population?  What is the condition or disease?)
• Intervention (What do you want to do with this patient?  i.e. treat, diagnose)
• Comparisons [not always included]  (What is the alternative to this intervention?  i.e. placebo, different drug, surgery)
• Outcomes  (What are the relevant outcomes?  i.e. morbidity, death, complications)

If the PICO model does not fit your question, try to use other ways to help be sure to articulate all parts of your question. Perhaps asking yourself Who, What, Why, How will help.  

Example Question:  Is acupuncture as effective of a therapy as triptans in the treament of adult migraine?

Note that this question fits the PICO model.

• Population: Adults with migraines
• Intervention: Acupuncture
• Comparison: Triptans/tryptamines
• Outcome: Fewer Headache days, Fewer migraines

A literature review search is an iterative process. Your goal is to find all of the articles that are pertinent to your subject. Successful searching requires you to think about the complexity of language. You need to match the words you use in your search to the words used by article authors and database indexers. A thorough PubMed search must identify the author words likely to be in the title and abstract or the indexer's selected MeSH (Medical Subject Heading) Terms.

Start by doing a preliminary search using the words from the key parts of your research question.

Step #1: Initial Search

Enter the key concepts from your research question combined with the Boolean operator AND. PubMed does automatically combine your terms with AND. However, it can be easier to modify your search if you start by including the Boolean operators.

migraine AND acupuncture AND tryptamines

The search retrieves a number of relevant article records, but probably not everything on the topic.

Step #2: Evaluate Results

Use the Display Settings drop down in the upper left hand corner of the results page to change to Abstract display.

Review the results and move articles that are directly related to your topic to the Clipboard .

Go to the Clipboard to examine the language in the articles that are directly related to your topic.

• look for words in the titles and abstracts of these pertinent articles that differ from the words you used
• look for relevant MeSH terms in the list linked at the bottom of each article

The following two articles were selected from the search results and placed on the Clipboard.

Here are word differences to consider:

• Initial search used acupuncture. MeSH Terms use Acupuncture therapy.
• Initial search used migraine.  Related word from MeSH Terms is Migraine without Aura and Migraine Disorders.
• Initial search used tryptamines. Article title uses sumatriptan. Related word from MeSH is Sumatriptan or Tryptamines.

With this knowledge you can reformulate your search to expand your retrieval, adding synonyms for all concepts except for manual and plaque.

#3 Revise Search

Use the Boolean OR operator to group synonyms together and use parentheses around the OR groups so they will be searched properly. See the image below to review the difference between Boolean OR / Boolean AND.

Here is what the new search looks like:

(migraine OR migraine disorders) AND (acupuncture OR acupuncture therapy) AND (tryptamines OR sumatriptan)

• Search Worksheet Example: Acupuncture vs. Triptans for Migraine
• Search Worksheet

• << Previous: My NCBI Accounts
• Last Updated: Dec 19, 2023 3:00 PM
• URL: https://guides.lib.unc.edu/search-pubmed

Search & Find

• E-Research by Discipline
• More Search & Find

Places & Spaces

• Places to Study
• Book a Study Room
• Printers, Scanners, & Computers
• More Places & Spaces
• Borrowing & Circulation
• Request a Title for Purchase
• Schedule Instruction Session
• More Services

Support & Guides

• Course Reserves
• Research Guides
• Citing & Writing
• More Support & Guides
• Mission Statement
• Diversity Statement
• Staff Directory
• Job Opportunities
• Give to the Libraries
• News & Exhibits
• Reckoning Initiative
• More About Us

• Search This Site
• Privacy Policy
• Accessibility
• Give Us Your Feedback
• 208 Raleigh Street CB #3916
• Chapel Hill, NC 27515-8890
• 919-962-1053

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7 Writing a Literature Review

Hundreds of original investigation research articles on health science topics are published each year. It is becoming harder and harder to keep on top of all new findings in a topic area and – more importantly – to work out how they all fit together to determine our current understanding of a topic. This is where literature reviews come in.

In this chapter, we explain what a literature review is and outline the stages involved in writing one. We also provide practical tips on how to communicate the results of a review of current literature on a topic in the format of a literature review.

7.1 What is a literature review?

Literature reviews provide a synthesis and evaluation  of the existing literature on a particular topic with the aim of gaining a new, deeper understanding of the topic.

Published literature reviews are typically written by scientists who are experts in that particular area of science. Usually, they will be widely published as authors of their own original work, making them highly qualified to author a literature review.

However, literature reviews are still subject to peer review before being published. Literature reviews provide an important bridge between the expert scientific community and many other communities, such as science journalists, teachers, and medical and allied health professionals. When the most up-to-date knowledge reaches such audiences, it is more likely that this information will find its way to the general public. When this happens, – the ultimate good of science can be realised.

A literature review is structured differently from an original research article. It is developed based on themes, rather than stages of the scientific method.

In the article Ten simple rules for writing a literature review , Marco Pautasso explains the importance of literature reviews:

Literature reviews are in great demand in most scientific fields. Their need stems from the ever-increasing output of scientific publications. For example, compared to 1991, in 2008 three, eight, and forty times more papers were indexed in Web of Science on malaria, obesity, and biodiversity, respectively. Given such mountains of papers, scientists cannot be expected to examine in detail every single new paper relevant to their interests. Thus, it is both advantageous and necessary to rely on regular summaries of the recent literature. Although recognition for scientists mainly comes from primary research, timely literature reviews can lead to new synthetic insights and are often widely read. For such summaries to be useful, however, they need to be compiled in a professional way (Pautasso, 2013, para. 1).

An example of a literature review is shown in Figure 7.1.

Video 7.1: What is a literature review? [2 mins, 11 secs]

Watch this video created by Steely Library at Northern Kentucky Library called ‘ What is a literature review? Note: Closed captions are available by clicking on the CC button below.

Examples of published literature reviews

• Strength training alone, exercise therapy alone, and exercise therapy with passive manual mobilisation each reduce pain and disability in people with knee osteoarthritis: a systematic review
• Traveler’s diarrhea: a clinical review
• Cultural concepts of distress and psychiatric disorders: literature review and research recommendations for global mental health epidemiology

7.2 Steps of writing a literature review

Writing a literature review is a very challenging task. Figure 7.2 summarises the steps of writing a literature review. Depending on why you are writing your literature review, you may be given a topic area, or may choose a topic that particularly interests you or is related to a research project that you wish to undertake.

Chapter 6 provides instructions on finding scientific literature that would form the basis for your literature review.

Once you have your topic and have accessed the literature, the next stages (analysis, synthesis and evaluation) are challenging. Next, we look at these important cognitive skills student scientists will need to develop and employ to successfully write a literature review, and provide some guidance for navigating these stages.

Analysis, synthesis and evaluation

Analysis, synthesis and evaluation are three essential skills required by scientists  and you will need to develop these skills if you are to write a good literature review ( Figure 7.3 ). These important cognitive skills are discussed in more detail in Chapter 9.

The first step in writing a literature review is to analyse the original investigation research papers that you have gathered related to your topic.

Analysis requires examining the papers methodically and in detail, so you can understand and interpret aspects of the study described in each research article.

An analysis grid is a simple tool you can use to help with the careful examination and breakdown of each paper. This tool will allow you to create a concise summary of each research paper; see Table 7.1 for an example of  an analysis grid. When filling in the grid, the aim is to draw out key aspects of each research paper. Use a different row for each paper, and a different column for each aspect of the paper ( Tables 7.2 and 7.3 show how completed analysis grid may look).

Before completing your own grid, look at these examples and note the types of information that have been included, as well as the level of detail. Completing an analysis grid with a sufficient level of detail will help you to complete the synthesis and evaluation stages effectively. This grid will allow you to more easily observe similarities and differences across the findings of the research papers and to identify possible explanations (e.g., differences in methodologies employed) for observed differences between the findings of different research papers.

Table 7.1: Example of an analysis grid

Table 7.3: Sample filled-in analysis grid for research article by Ping and colleagues

Source: Ping, WC, Keong, CC & Bandyopadhyay, A 2010, ‘Effects of acute supplementation of caffeine on cardiorespiratory responses during endurance running in a hot and humid climate’, Indian Journal of Medical Research, vol. 132, pp. 36–41. Used under a CC-BY-NC-SA licence.

Step two of writing a literature review is synthesis.

Synthesis describes combining separate components or elements to form a connected whole.

You will use the results of your analysis to find themes to build your literature review around. Each of the themes identified will become a subheading within the body of your literature review.

A good place to start when identifying themes is with the dependent variables (results/findings) that were investigated in the research studies.

Because all of the research articles you are incorporating into your literature review are related to your topic, it is likely that they have similar study designs and have measured similar dependent variables. Review the ‘Results’ column of your analysis grid. You may like to collate the common themes in a synthesis grid (see, for example Table 7.4 ).

Step three of writing a literature review is evaluation, which can only be done after carefully analysing your research papers and synthesising the common themes (findings).

During the evaluation stage, you are making judgements on the themes presented in the research articles that you have read. This includes providing physiological explanations for the findings. It may be useful to refer to the discussion section of published original investigation research papers, or another literature review, where the authors may mention tested or hypothetical physiological mechanisms that may explain their findings.

When the findings of the investigations related to a particular theme are inconsistent (e.g., one study shows that caffeine effects performance and another study shows that caffeine had no effect on performance) you should attempt to provide explanations of why the results differ, including physiological explanations. A good place to start is by comparing the methodologies to determine if there are any differences that may explain the differences in the findings (see the ‘Experimental design’ column of your analysis grid). An example of evaluation is shown in the examples that follow in this section, under ‘Running performance’ and ‘RPE ratings’.

When the findings of the papers related to a particular theme are consistent (e.g., caffeine had no effect on oxygen uptake in both studies) an evaluation should include an explanation of why the results are similar. Once again, include physiological explanations. It is still a good idea to compare methodologies as a background to the evaluation. An example of evaluation is shown in the following under ‘Oxygen consumption’.

7.3 Writing your literature review

Once you have completed the analysis, and synthesis grids and written your evaluation of the research papers , you can combine synthesis and evaluation information to create a paragraph for a literature review ( Figure 7.4 ).

The following paragraphs are an example of combining the outcome of the synthesis and evaluation stages to produce a paragraph for a literature review.

Note that this is an example using only two papers – most literature reviews would be presenting information on many more papers than this ( (e.g., 106 papers in the review article by Bain and colleagues discussed later in this chapter). However, the same principle applies regardless of the number of papers reviewed.

The next part of this chapter looks at the each section of a literature review and explains how to write them by referring to a review article that was published in Frontiers in Physiology and shown in Figure 7.1. Each section from the published article is annotated to highlight important features of the format of the review article, and identifies the synthesis and evaluation information.

In the examination of each review article section we will point out examples of how the authors have presented certain information and where they display application of important cognitive processes; we will use the colour code shown below:

This should be one paragraph that accurately reflects the contents of the review article.

Introduction

The introduction should establish the context and importance of the review

Body of literature review

The reference section provides a list of the references that you cited in the body of your review article. The format will depend on the journal of publication as each journal has their own specific referencing format.

It is important to accurately cite references in research papers to acknowledge your sources and ensure credit is appropriately given to authors of work you have referred to. An accurate and comprehensive reference list also shows your readers that you are well-read in your topic area and are aware of the key papers that provide the context to your research.

It is important to keep track of your resources and to reference them consistently in the format required by the publication in which your work will appear. Most scientists will use reference management software to store details of all of the journal articles (and other sources) they use while writing their review article. This software also automates the process of adding in-text references and creating a reference list. In the review article by Bain et al. (2014) used as an example in this chapter, the reference list contains 106 items, so you can imagine how much help referencing software would be. Chapter 5 shows you how to use EndNote, one example of reference management software.

Click the drop down below to review the terms learned from this chapter.

Copyright note:

• The quotation from Pautasso, M 2013, ‘Ten simple rules for writing a literature review’, PLoS Computational Biology is use under a CC-BY licence. 
• Content from the annotated article and tables are based on Schubert, MM, Astorino, TA & Azevedo, JJL 2013, ‘The effects of caffeinated ‘energy shots’ on time trial performance’, Nutrients, vol. 5, no. 6, pp. 2062–2075 (used under a CC-BY 3.0 licence ) and P ing, WC, Keong , CC & Bandyopadhyay, A 2010, ‘Effects of acute supplementation of caffeine on cardiorespiratory responses during endurance running in a hot and humid climate’, Indian Journal of Medical Research, vol. 132, pp. 36–41 (used under a CC-BY-NC-SA 4.0 licence ). 

Bain, A.R., Morrison, S.A., & Ainslie, P.N. (2014). Cerebral oxygenation and hyperthermia. Frontiers in Physiology, 5 , 92.

Pautasso, M. (2013). Ten simple rules for writing a literature review. PLoS Computational Biology, 9 (7), e1003149.

How To Do Science Copyright © 2022 by University of Southern Queensland is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Publishing Literature Reviews

• First Online: 11 August 2022

Cite this chapter

• Rob Dekkers 4 ,
• Lindsey Carey 5 &
• Peter Langhorne 6  

1604 Accesses

1 Citations

With the credo of academics being ‘publish or perish’, much attention is given to publishing output of studies, which includes literature reviews. The foundation for conducting literature reviews has been laid in the previous chapters from setting the topic in Chapter 4 to the analysis and synthesis in later chapters and the reporting in Chapter 13 .

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Paraphrased from electronic messages to Rob Dekkers dated 6th June 2020 and 19th October 2020.

Predatory journals are used by exploitive publishers that charge publication fees to authors without reviewing articles for quality and legitimacy, and without providing the other editorial and publishing services that legitimate academic journals provide. They are considered as predatory, because scholars can be tricked into publishing with them. See Beall ( 2016 ) for how these predatory journals operate and also which points to pay attention to.

Note that ranking, although often used for all kinds of purposes, is not directly a reflection of the importance or relevance of journals. An increasing number of academics and scientific institutions have signed the San Francisco Declaration on Research Assessment, published in 2013, also known by its abbreviation DORA. Specifically, it states that the impact factor (or any other ranking) is not to be used as a substitute ‘measure of the quality of individual research articles, or in hiring, promotion, or funding decisions.’

In addition to predatory journals there are predatory conferences, such as those organised by WASET (World Academy of Science, Engineering and Technology); there have been numerous publications in the media about these type of organisations and how they falsely claim to be credible conferences. Therefore, always check the credentials of a conference before submitting papers to avoid becoming victim of these.

These key tests have been adapted from a presentation given by Prof. David Bennett for a select group of doctoral students at the University of the West of Scotland in 2012.

Careful reading of the comments in the review is necessary, including the advice of the editor or associate editor, because sometimes the decision for a major revision can be interpreted as a polite way of rejecting (McKercher et al. 2007 , p. 458). If there is any doubt, it is better to contact the editor before submitting a revision; note that not all editors might reply.

Beall J (2016) Essential information about predatory publishers and journals. Int Higher Educ 86:2–3. https://doi.org/10.6017/ihe.2016.86.9358

Article   Google Scholar  

Bodolica V, Spraggon M (2018) An end-to-end process of writing and publishing influential literature review articles: do’s and don’ts. Manag Decis 56(11):2472–2486. https://doi.org/10.1108/MD-03-2018-0253

Callahan JL (2014) Writing literature reviews: a reprise and update. Hum Resour Dev Rev 13(3):271–275. https://doi.org/10.1177/1534484314536705

Green BN, Johnson CD, Adams A (2006) Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med 5(3):101–117. https://doi.org/10.1016/S0899-3467(07)60142-6

Holweg M (2007) The genealogy of lean production. J Oper Manag 25(2):420–437. https://doi.org/10.1016/j.jom.2006.04.001

Jennex ME (2015) Literature reviews and the review process: an editor-in-chief’s perspective. Commun Assoc Inf Syst 36:139–146. https://doi.org/10.17705/1CAIS.03608

MacLure M (2005) Clarity bordering on stupidity: where’s the quality in systematic review? J Educ Policy 20(4):393–416. https://doi.org/10.1080/02680930500131801

McKercher B, Law R, Weber K, Song H, Hsu C (2007) Why referees reject manuscripts. J Hosp Tour Res 31(4):455–470. https://doi.org/10.1177/1096348007302355

Rocco TS, Plakhotnik MS (2009) Literature reviews, conceptual frameworks, and theoretical frameworks: terms, functions, and distinctions. Hum Resour Dev Rev 8(1):120–130. https://doi.org/10.1177/1534484309332617

Shah R, Ward PT (2007) Defining and developing measures of lean production. J Oper Manag 25(4):785–805. https://doi.org/10.1016/j.jom.2007.01.019

Soilemezi D, Linceviciute S (2018) Synthesizing qualitative research: reflections and lessons learnt by two new reviewers. Int J Qual Methods 17(1):1–14. https://doi.org/10.1177/1609406918768014

Taylor EW, Beck J, Ainsworth E (2001) Publishing qualitative adult education research: a peer review perspective. Stud Educ Adults 33(2):163–179. https://doi.org/10.1080/02660830.2001.11661452

Torraco RJ (2005) Writing integrative literature reviews: guidelines and examples. Hum Resour Dev Rev 4(3):356–367. https://doi.org/10.1177/1534484305278283

Toussaint JS, Berry LL (2013) The promise of lean in health care. Mayo Clin Proc 88(1):74–82. https://doi.org/10.1016/j.mayocp.2012.07.025

Trafford V, Leshem S, Bitzer E (2014) Conclusion chapters in doctoral theses: some international findings. High Educ Rev 46(3):52–81

Google Scholar  

Womack JP, Jones DT (1996) Lean thinking. Simon & Schuster, New York

Womack JP, Jones DT, Roos D (1991) The machine that changed the world: the story of lean production. Free Press, New York

Young TP, McClean SI (2008) A critical look at lean thinking in healthcare. Qual Saf Health Care 17(5):382–386. https://doi.org/10.1136/qshc.2006.020131

Download references

Author information

Authors and affiliations.

University of Glasgow, Glasgow, UK

Rob Dekkers

Glasgow Caledonian University, Glasgow, UK

Lindsey Carey

Peter Langhorne

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Dekkers, R., Carey, L., Langhorne, P. (2022). Publishing Literature Reviews. In: Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches. Springer, Cham. https://doi.org/10.1007/978-3-030-90025-0_16

Download citation

DOI : https://doi.org/10.1007/978-3-030-90025-0_16

Published : 11 August 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-90024-3

Online ISBN : 978-3-030-90025-0

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Want to publish a literature review? Think of it as an empirical paper

What to consider if you want to publish a literature review paper

Tatiana Andreeva - Fri 23 Apr 2021 07:50 (updated Fri 27 Oct 2023 17:26)

[Guest post by CYGNA member Tatiana Andreeva ]

When you’ve been reading a lot on a particular topic – for example, reviewing the literature for your research project or for your PhD – at some point it looks like you have enough material and reflections to publish this piece of work as a separate paper. Recognize this? If you ever tried it, you might know that publishing a literature review paper in an academic journal is a tricky task. The literature review publications come in so many forms, and there is no single cheat-sheet or established format like for empirical papers that you could follow to ensure success in publication.

Through my own journey of trial-and-error on this path, as well as through reviewing for journals and for PhD students in my course, I came up with an idea that will help you to increase the chances of publishing a literature review: think of a literature review as simply another empirical research project. Think of it as an empirical study, in which your data comes not from your usual fieldwork but from the articles that you review.

Many literature reviews can be thought of as a qualitative empirical study, in which the papers included in the review substitute interviews or field observations that you would usually collect and code. Some literature reviews, e.g., meta-analyses, are more like a quantitative empirical paper, in which various numbers you extract from the papers in your dataset substitute your survey data.

Seeing literature review in this way has three important implications for how we think about our literature review, and how we can design it to increase its chances of being interesting to others - that is, of being published.

Start with a relevant research problem and an interesting research question

We learn early in our academic career that any empirical paper should have a clear research problem and a clear research question. We frequently hear from journal editors and reviewers that just having a gap in the literature, or the fact that something has not been researched before, are not good enough to justify doing yet another empirical study. They say: you need to have a problem that your study can address, and you need to have a question that we currently don’t have an answer to. Only then your empirical study can add value to existing research. 

When we think of a literature review as of an empirical study, just with the different type of data at hand, we realize that the very same rationale applies. From this perspective the arguments that I often see in literature reviews – that there is no literature review in this particular area or that the existing literature reviews are quite dated – are not sufficient in the journal’s eyes to justify the publication of a literature review on a topic. If you aim to publish your literature review, start by thinking – what is the problem I would like to address? What would be my research question about this problem, that other readers would find interesting?

Design a methodologically-sound data collection and analysis protocol

When we think of any empirical study, we know that if we want to have reliable findings that will be accepted by our peers as trustworthy, we need to follow a transparent and well-thought data collection protocol. We also need to carefully choose and correctly apply relevant data analysis method. This goes without saying, right?

The same applies to the literature review! If we want our readers to trust our conclusions from the literature review, we need to make sure that the data we collect speaks to our research question, is of good quality, representative of the field, etc. The growing attention in business and management field to the systematic approach to literature reviews (Denyer & Tranfield, 2009; Rojon et al., 2021) reflects the rising expectations of the quality of the data used in literature review papers. Indeed, this approach offers exactly that: a clear data collection protocol, transparently communicated, so that someone else could replicate your study. For example, do the very same thing in 10 years and see how thinking on the topic has changed.

In the literature on doing literature reviews you will read that systematic literature review is only one of the types of literature reviews. Yet all recommendations on doing different types of the literature reviews share the idea that the data that you base your conclusions on has to be collected in a rigorous and transparent way (e.g., Callahan, 2014).  In this post you can find more references on how to ensure that your literature review “data collection” protocol meets the quality expectations.

So now you have all the papers you have carefully selected, how do you go about analysing them, so that peer academics would recognize your conclusions as reliable and robust? This is the trickiest part, and we have limited methodological advice published on this. In this post I’ve mentioned some papers that discuss specific methods of literature analysis. For example, I found that a sophisticated coding rubric leveraged our literature analysis to a different level (Sergeeva & Andreeva, 2016), but must acknowledge that developing this rubric was one of the most challenging tasks of this review paper. In O’Higgins et al. (forthcoming) we used a combination of qualitative content analysis with Pearson’s chi-squared (χ²) goodness of fit test in order to validate some of our conclusions. The trick is - as with any empirical study - your choice of the analytical method needs to fit with your research question. In sum, the message is: choose your method for analysis of the selected literature carefully, apply it rigorously, and explain it transparently.

Think of the theoretical contribution beyond description of the findings

When we think of our usual empirical work, be it qualitative or quantitative, we are well-aware that just the description of our data wouldn’t do. We know that we need to leverage what our data shows to explain how it informs the broader theory, how it compares to previous studies, what is new that we see from this data?

Again, the same logic applies to the literature reviews. In practice though, we often find it difficult to apply this advice to our literature review papers, because the description of the field in itself seems to be novel, especially if nobody did such a review before. In my experience, this argument does not persuade editors and reviewers of the journals, and often rightfully so.

For example, think of a typical quantitative empirical paper: a descriptive statistics table must be provided, but no one would claim a contribution based on it, right? Cropanzano (2009:1306-1307) offers a good exercise that explains why reviewers often don’t buy the description of the field as a novel contribution. He suggests: imagine somebody who read all the primary articles in your dataset, would they still learn anything from your literature review? And if the answer is “no”, then it’s likely that your review paper doesn’t have yet the level of contribution that is needed to turn it into a publication.

I think this exercise can also help to stimulate your thinking of what a theoretical contribution of your literature review could be. For example, think – what it is that I see in this literature that others are not likely to see?  In this blogpost you can find some papers that offer insights on how to leverage your literature review to have a theoretical contribution.  

Callahan, J.L. (2014). Writing literature reviews: A reprise and update. Human Resource Development Review , 13(3), 271–275. https://doi.org/10.1177/1534484314536705

Cropanzano, R. (2009). Writing nonempirical articles for Journal of Management: General thoughts and suggestions. Journal of Management , 35(6), 1304–1311. https://doi.org/10.1177/0149206309344118

Denyer, D., Tranfield, D. (2009). Producing a systematic review. In Buchanan, D., Bryman, A. (Eds.), The Sage handbook of organizational research methods (pp. 671–689). London, UK: Sage.

O’Higgins, C., Andreeva, T., Aramburu, N. (forthcoming). International management challenges of professional service firms: a synthesis of the literature. Review of International Business and Strategy.

Rojon, C., Okupe, A., McDowall, A. (2021). Utilization and development of systematic reviews in management research: What do we know and where do we go from here? International Journal of Management Reviews, 1– 33. https://doi.org/10.1111/ijmr.12245

Sergeeva, A., Andreeva, T. (2016). Knowledge sharing: bringing the context back in, Journal of Management Inquiry , 25, 240-261. https://doi.org/10.1177/1056492615618271

Related blogposts

• Resources on doing a literature review
• Do you really want to publish your literature review? Advice for PhD students
• How to keep up-to-date with the literature, but avoid information overload?
• Is a literature review publication a low-cost project?
• Using Publish or Perish to do a literature review
• How to conduct a longitudinal literature review?
• New: Publish or Perish now also exports abstracts

Find the resources on my website useful?

I cover all the expenses of operating my website privately. If you enjoyed this post and want to support me in maintaining my website, consider buying a copy of one of my books (see below) or  supporting the Publish or Perish software .

A story about perseverance and publication failures

Copyright © 2023 Tatiana Andreeva . All rights reserved. Page last modified on Fri 27 Oct 2023 17:26

Tatiana Andreeva's profile and contact details >>

• Search Menu
• Advance articles
• Editor's Choice
• Supplement Archive
• Article Collection Archive
• Author Guidelines
• Submission Site
• Open Access
• Call for Papers
• Why Publish?
• About Nutrition Reviews
• About International Life Sciences Institute
• Editorial Board
• Early Career Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, acknowledgments, supporting information, mapping the evidence of novel plant-based foods: a systematic review of nutritional, health, and environmental impacts in high-income countries.

• Article contents
• Figures & tables
• Supplementary Data

Sarah Nájera Espinosa, Genevieve Hadida, Anne Jelmar Sietsma, Carmelia Alae-Carew, Grace Turner, Rosemary Green, Silvia Pastorino, Roberto Picetti, Pauline Scheelbeek, Mapping the evidence of novel plant-based foods: a systematic review of nutritional, health, and environmental impacts in high-income countries, Nutrition Reviews , 2024;, nuae031, https://doi.org/10.1093/nutrit/nuae031

• Permissions Icon Permissions

Shifting from current dietary patterns to diets rich in plant-based (PB) foods and lower in animal-based foods (ABFs) is generally regarded as a suitable strategy to improve nutritional health and reduce environmental impacts. Despite the recent growth in supply of and demand for novel plant-based foods (NPBFs), a comprehensive overview is lacking.

This review provides a synthesis of available evidence, highlights challenges, and informs public health and environmental strategies for purposeful political decision-making by systematically searching, analyzing, and summarizing the available literature.

Five peer-reviewed databases and grey literature sources were rigorously searched for publications.

Study characteristics meeting the inclusion criteria regarding NPBF nutrient composition and health and environmental outcomes in high-income countries were extracted.

Fifty-seven peer-reviewed and 36 grey literature sources were identified; these were published in 2016–2022. NPBFs typically have substantially lower environmental impacts than ABFs, but the nutritional contents are complex and vary considerably across brands, product type, and main primary ingredient. In the limited evidence on the health impacts, shifts from ABFs to PB meats were associated with positive health outcomes. However, results were mixed for PB drinks, with links to micronutrient deficiencies.

If carefully selected, certain NPBFs have the potential to be healthier and nutrient-rich alternatives to ABFs and typically have smaller environmental footprints. More disaggregated categorization of various types of NPBFs would be a helpful step in guiding consumers and key stakeholders to make informed decisions. To enable informed policymaking on the inclusion of NPBFs in dietary transitions as part of a wider net-zero and health strategy, future priorities should include nutritional food standards, labelling, and subdivisions or categorizations of NPBFs, as well as short- and long-term health studies evaluating dietary shifts from ABFs to NPBFs and standardized environmental impact assessments, ideally from independent funders.

The fragile interconnection between food systems and the environment is increasingly evident. 1–3 While current agricultural practices are damaging the environment, environmental change is putting food supplies at risk of disruption if timely adaptation strategies are not used. 4–8 This relationship exists at a time when food systems are already struggling to provide healthy diets for all, with many populations experiencing a coexistence of undernutrition and obesity. 1 , 3

Structural changes in food systems are critical to both safeguard people’s health and accomplish the climate adaptation and mitigation commitments mentioned in The United Nations Framework Convention on Climate Change 9 and the United Nations’ Sustainable Development Goals. 10 While production-side strategies can contribute toward climate mitigation, substantial opportunities for further emission reductions and acceleration toward net-zero targets can be achieved through dietary changes and the resulting lower demand for foods with a large environmental footprint.

In food-secure and high-income settings, a shift from “conventional diets” (which typically contain high amounts of animal-based foods [ABFs]) to predominantly plant-based (PB) diets could improve population and planetary health. 2 , 11 Dietary change has many obstacles, with diets influenced by many factors 12 , 13 that act as barriers to increasing consumption of minimally processed PB foods (eg, legumes, vegetables). If common barriers are removed, such as the need for additional cooking skills, major changes in taste and appearance of commonly consumed dishes, and fear of social stigma, 14 , 15 novel plant-based foods (NPBFs), products designed to mimic and replace ABFs to allow easy incorporation into habitual diets (eg, vegan and vegetarian meat and dairy) (see Box 1 ), may offer an easier option to facilitate this shift.

In recent years, the NPBF landscape has expanded rapidly. Several new types of NPBFs (eg, PB drinks, yogurts, eggs, meats) were introduced to the market, and trends showed increasing sales, volume, and investment growth across many countries. 16–21 In 2023, data suggested a possible slowdown, especially for PB meats, with some consumers criticizing their cost and taste, 22 and some NPBF manufacturers reporting net losses. 23 , 24 However, sales of supermarkets’ own-label PB meat alternatives have seen growth, 23 alongside consistent increases in sales of PB dairy and eggs 25 (see Supplementary file 1, section 1.1, in the Supporting Information online for detailed information on costs).

According to a global survey focusing on individuals following vegan or vegetarian diets most or all of the time, 22.0% of consumers reported adhering to a meat-free diet, and there is growing interest in embracing PB eating, with approximately 42.0% of consumers anticipating that PB foods will replace most meat within a decade. 26 With consumption of NPBFs in the United Kingdom doubling between 2008 and 2019, particularly among women and younger generations, and the fact that in 2022, 60.0% of US households purchased at least 1 type of NPBF, verification of any health and sustainability claims in marketed products is of vital importance. 22 , 27 , 28 Currently, various NPBFs are advertised as potential dietary “game changers,” with claims that they would play an important and positive role in sustainability and health, 29 , 30 and, thus, could play a pivotal role in the so-called consumption corridors. 31 However, because of their novelty, some consumers question these positive claims. 32 Although NPBFs are generally regarded as a low-carbon alternative to ABFs, their nutrient and health profiles remain largely unknown and are often criticized. This is primarily related to concerns regarding micronutrient and protein content, along with higher content of saturated fats and sodium in comparison to ABFs, and level of processing. 33 , 34

Previous reviews have primarily focused on single aspects of NPBFs 17 , 19 , 22 , 25 , 29 , 34–46 or ingredients of NPBFs 39 , 47–50 ; a few recent reviews explored the positive health and environmental outcomes of consuming selected NPBFs. 51–53 However, research quantifying the potential impacts of NPBFs is still in its infancy, and an overview that is both systematic and comprehensive, comprising health, nutrient, and environmental outcomes from peer-reviewed and grey literature of different types of NPBFs, does not yet exist, to our knowledge. This lack makes it difficult for policy makers and consumers to assess the trade-offs between nutrient composition and the environmental and health impacts of NPBFs, and hinders the potential inclusion of NPBFs in sustainable and healthy dietary recommendations.

To synthesize available evidence, highlight challenges, inform public health and environmental strategies, and inform purposeful political decision-making, we aimed, in this study, to systematically search, analyze, and summarize the available grey and peer-reviewed literature on the nutrient composition, environmental footprints, and health effects of NPBFs sold and consumed in high-income countries, and to quantify and summarize their reported results.

The full-study protocol we followed is published elsewhere (see Nájera Espinosa et al 54 and Supplementary file 1, section 2, in the Supporting Information online for more details on the methods). Briefly, a systematic search was performed to identify peer-reviewed journal articles and grey literature that contained data on the nutrient composition, health impacts, and environmental impacts of NPBFs. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. 55

Peer-reviewed literature

Five scientific databases were systematically searched (MEDLINE, Embase, Global Health, GreenFILE, and the Web of Science Core Collection) on August 29, 2021; we conducted an updated search on June 29, 2022. The search was limited to articles published and accepted after January 2016 until June 29, 2022, because of the substantial growth in supply and demand of NPBFs in the past 7 years. 16–19 In addition to database searching, citation lists from identified systematic literature reviews were handsearched (see Supplementary file 1, section 2.6, in the Supporting Information online for the full search strategy). After the quality criteria were applied (described in Supplementary file 1 , Table S1 in the Supporting Information online ), titles were manually and triple screened. Abstracts were manually double screened after application of a supervised machine-learning algorithm (ie, a support vector machine 56 ) through Scikit Learn 57 that ranked and highlighted likely relevant articles (ie, conducted priority screening). This approach is described elsewhere in detail (see Supplementary file 1, section 2.1, in the Supporting Information online ). 58 Full texts were manually screened by 2 authors and data were also double extracted.

Grey literature

To capture grey literature in a systematic way, a manual search was conducted on Google (see Supplementary file 1, section 2.3 , and Table S3 in the Supporting Information online ). Text from the webpages was then scraped and a state-of-the-art, pretrained language model from Hugging Face 59 was used to create a summary of each web link. Results were exported into a comma-separated value, or CSV, file. Additionally, a manual search in Google of relevant websites from the top NPBF producers in the United Kingdom and United States was conducted. 60–63 And literature from relevant websites that promote NPBFs, such as the Good Food Institute and Green Queen, were searched and screened manually (see Supplementary file 1, section 2.3 , and Tables S4 and S5 in the Supporting Information online ).

Data analysis, categorization, and key definitions: nutrient, health, and environmental outcomes

The PICO (population, intervention, comparison, and outcome) criteria are defined in Table 1 (see Supplementary file 1, Table S1 in the Supporting Information online for a detailed list of the inclusion and exclusion criteria). Main study characteristics and nutrient, health, and environmental outcomes were extracted (see Supplementary files 1 and 3 in the Supporting Information online for more details).

PICO criteria for inclusion of studies

PB drinks and milk reported in 100 ml of product.

NPBFs and their ABF counterparts were categorized into food groups on the basis of their primary ingredient ( Table 2 ). See Supplementary file 1, sections 2.4 and 2.5, in the Supporting Information online for more details on the selection of nutrients, data analysis assumptions, and ABF baseline comparators). The following terms for each NPBF type are used in this review:

Food groups for novel plant-based foods and animal-based foods and their respective reported main primary ingredient

For the purposes of this review, peanuts were included in the Nuts and Seeds group because they are typically consumed as such.

Blended or mixed products, if reported, the first ingredient was taken as the primary ingredient. For example, soy & almond PB drinks were labelled as legumes.

If a product did not report any ingredients, they were categorised as unknown.

PB meat products or alternatives: include different types of PB meats (eg, PB chicken, sausages, mincemeat), categories (eg, mycoprotein, legumes), and brands

PB drink products or alternatives: include different PB drink categories (eg, legumes, nuts, seeds) and brands

PB yogurt products or alternatives: include different PB yogurt categories (eg, legumes, coconut) and brands

PB cheese products or alternatives: include different types of PB cheese categories (eg, coconut, nuts, seeds) and brands

PB egg products or alternatives: include different types of PB egg categories and brands

Mention of PB products (without further specification) refers to all the listed product subcategories mentioned, except for PB eggs.

Assessment of robustness and relevance

A modified version of the Critical Appraisal Skills Program checklist for randomized controlled trials 64 was adapted to assess robustness and relevance of the studies in the full-text reviewing stage. The modifications involved the exclusion of the randomization, blinding, and cost-effectiveness criteria on the Critical Appraisal Skills Program checklist, and funding source was added as a criterion. Studies were assessed by 4 reviewers (G.H., R.P., S.P., and S.N.E.). Studies were assessed as follows: (1) clear description of the study design, (2) appropriate comparison group, (3) clear description of the methods, (4) rigorous and clearly described analysis, (5) funding source, and (6) precision of measure of effect. Studies with a minimum score of 1 were included, and sensitivity analysis was performed by funding source (see Supplementary file 1, section 2.2, in the Supporting Information online for more details).

Fruit, vegetable, legume, and nut content in novel plant-based foods

In addition to the review component, a cross-sectional analysis was conducted to examine the total fruit, vegetable, legume, and nut content (percentage estimate) of each type of NPBF sold in the United Kingdom. For this, a time-stamped data set of observations from UK supermarkets generated by FoodDB in October 2021 was used. Details are described elsewhere 65 and in Supplementary file 1, section 2.7, in the Supporting Information online . Detailed data at the global level are not available to date; hence, this part of the analysis is limited to the United Kingdom only.

Sensitivity analysis

A common concern about studies on the health impacts and environmental sustainability of NPBFs is that they can be funded by the industry that produces them; hence, we conducted a sensitivity analysis by funding source. Furthermore, given that relative improvements in health and environmental sustainability depend on the baseline comparator used ( Supplementary file 1, section 2.2, in the Supporting Information online ), the sensitivity analysis based on the main primary ingredient of a given NPBF and its respective ABF comparator was also performed. The Wilcoxon test for sensitivity analysis with a significance level set at P  ≤ 0.05 was used.

Systematic search results

A total of 49 563 peer-reviewed and 891 grey literature records were identified from the initial search. After unique literature sources were screened, 57 peer-reviewed articles and 36 grey literature studies met the inclusion criteria ( Figure 1 ). Supplementary 1, section 2, in the Supporting Information online provides further details on the screening process. The study characteristics that were extracted included basic study details (eg, authors, year, type of study, country, number of participants, follow-up period), relevant macro- and micronutrient content (eg, those related to common deficiencies, such as iron, calcium, vitamin B 12 ), health and health proxy data (eg, obesity, micronutrient status, risk factors related to noncommunicable diseases), and environmental variables (eg, carbon, water, and land-use data).

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of systematic review process reporting nutrient composition, and environmental and health outcomes of novel plant-based products in high-income countries . Abbreviations : IPCC, Intergovernmental Panel on Climate Change.

Nutrient composition of novel plant-based foods

The nutrient content of NPBFs was the most frequently studied outcome (n = 56 studies). Nutrient data were typically collected through supermarket cross-sectional surveys or manufacturers’ websites. PB meat alternatives (n = 35) and PB drink alternatives (n = 19) were most frequently reported; fewer studies researched PB cheese (n = 5) and yogurt alternatives (n = 4). No studies were found that assessed PB egg alternatives. The nutritional profile of NPBFs varied greatly by manufacturing process, including the main base ingredient (eg, soy, almond); the processing techniques, time, and temperature applied; and the type of product manufactured (ie, PB drinks, PB meats). 39 , 40 , 66 , 67

Energy density, saturated fat, fiber, sugar, sodium, and micronutrient content of plant-based meat alternatives

The 35 publications evaluating PB meat alternatives reported on 508 PB meat products with 66 ABF comparators. Where the median values for meat comparators were reported to be 221.0 kcal/100 g (interquartile range [IQR], 186.6–246.7), 5.7 g/100 g saturated fat (IQR, 3.2–7.1), and very low fiber (<0.1 g/100 g; IQR, 0.0–0.5), most meat-alternative groups were reported to have lower energy density, lower saturated-fat content, and more fiber ( Figure 2 and Supplementary file 2: Table S1 for detailed macronutrient information disaggregated by main ingredient). Mycoprotein-based meat alternatives were reported to be the least energy dense, with a median energy value of 123.0 kcal/100 g (IQR, 94.0–198.5; with ABFs, P value of difference [ P d ] < 0.001), whereas meat alternatives based on cereals and grain had the highest energy density of all PB meats (226.0 kcal/100 g [IQR, 189.8–268.5]; P d < 0.360), with values very similar to those of meat and poultry. Mycoprotein-based meats were also reported to be lowest in saturated fat (0.8 g/100 g [IQR, 0.5–1.3]; P d < 0.001), whereas nut- and seed-based meats had the highest saturated fat content (1.4 g/100 g [IQR, 1.1–1.7]; P d = 0.003) of all PB meats, which still was significantly lower than saturated fat content in meat and poultry. Finally, mycoprotein-based meat was reported to contain the highest fiber content (median, 6.0 g/100 g [IQR, 5.2–7.1]; P d  < 0.001), whereas cereal- and grain-based meats had the lowest fiber content of all PB meats (3.1 g/100 g [IQR, 2.3–3.9]; P d < 0.001), which still was significantly higher than in meat and poultry.

Macronutrient, sodium, and energy content in plant-based meat and drink alternatives in their respective food group based on main primary ingredient   (ie, predominant or core food item on the ingredient list) compared with meat and poultry, and dairy, respectively . Data were limited to raw products only. Abbreviation : M, median of each category.

Meat and poultry contained a median of 0.5 g/100 g total sugar (IQR, 0.0–0.9) and 426.7 mg/100 g sodium content (IQR, 101.0–672.8). All PB meats contained more total sugar but had similar levels of sodium in comparison with meat and poultry. Mycoprotein-based meats had the lowest total sugar content of all PB meats (median, 0.8 g/100 g [IQR: 0.5–1.8]; P d < 0.001], and nut- and seed-based meats contained the highest total sugar amount (median, 4.2 g/100 g [IQR, 2.3–6.6]; P d = 0.002); both showed strong evidence of being higher in total sugar content than meat and poultry. This is equivalent to 0.4 g and 3.4 g of total sugar/80.0 g serving size, or, if these sugars are considered free, 1.6% and 13.4% of the maximum recommended approximately 25.0 g average daily sugar intake. 68 Finally, the median sodium values for all PB meat groups did not show strong evidence of a difference from meat and poultry, except for legume-based meats (median, 520.0 mg/100 g [IQR, 400.0–636.0]; P d = 0.011). This is equivalent to 416.0 mg of sodium (or 1.0 g of salt) per 80.0 g serving size, or 20.8% of the maximum recommended 5.0 g average daily salt intake. Moreover, there were extreme outliers, with some PB meats reported to contain more than 1400.00 mg sodium (equivalent to 2.8 g salt) per 80.0 g; thus, consumption of 1 portion of this PB meat alternatives is more than half the recommended maximum daily intake of salt. 69

Only a few studies (n = 9) evaluated micronutrient data; these reported on 250 PB meat products and 24 ABF comparators. Micronutrient content ranged vastly across all groups: whereas some products would provide substantial contributions to average daily requirements, others were much less nutritious ( Table 3 and Supplementary file 2: Table S2 ). 69–83 For example, the median iron content for cereal- and grain-based PB meats (5.4 mg/100 g [IQR, 4.2–5.4]) was higher than the median of meat and poultry (1.3 mg/100 g [IQR, 1.1–1.6]). On the contrary, vitamin B 12 levels were lower for PB meat alternatives (medians ranged from 0.1 μg/100 g [IQR: 0.0–0.9] to 0.3 μg/100 g [IQR: 0.3–0.3]) as compared with 1.2 μg/100 g (IQR: 0.6–1.6) in meat and poultry. However, certain individual products had a comparable or higher vitamin B 12 content than their ABF comparator.

Summarized micronutrient values for PB meat and drinks and animal-based foods a

Values are compared with global average daily requirements (see Supplementary file 2 in the Supporting Information online for detailed information containing all disaggregated numbers by main ingredient of each novel plant-based food and animal-based foods). The table only reports micronutrients commonly found in meat and dairy. PB products also provided other micronutrients not commonly found in meat and dairy (ie, calcium in PB meats).

Abbreviations : ADR, average daily requirement; max, maximum; min, minimum; IQR, interquartile range; PB, plant-based.

No studies reported nutrient data from organic products. Although protein levels were not the main focus of this study, protein results are reported in Supplementary file 1: Figure S2 and Supplementary file 2: Table S1 , and show that, particularly, legume- and mycoprotein-based PB meats typically match meat and poultry in protein content.

Energy density, saturated fat, fiber, sugar, sodium, and micronutrient content of plant-based drinks

The 19 studies evaluating PB drinks reported on 397 PB drinks (unflavored and unsweetened) and 52 dairy milk products. Where dairy milk comparators were reported to contain median values of 50.1 kcal/100 mL energy density (IQR, 39.3–63.0), 1.1 g/100 mL saturated fat (IQR, 0.9–2.2), and no fiber (0.0 g/100 mL; IQR, 0.0–0.0), most PB drink groups were reported to have lower energy density, lower saturated fat content, and more fiber ( Figure 2 and Supplementary file 2: Table S1 ). Coconut-based drinks were reported to be the least energy dense (median energy value, 20.0 kcal/100 g [IQR: 19.0–33.7]; P d < 0.001), whereas drinks based on cereals and grains had the highest energy density of all PB drinks (median, 59.0 kcal/100 mL [IQR: 43.0–57.0]; P d = 0.566) but not significantly higher than dairy milks. PB drinks made of cereals and grains, fruits and vegetables, and nuts and seeds were reported to be lowest in saturated fat (median, 0.2 g/100 mL; IQRs, 0.1–0.2, 0.2–0.2, and 0.1–0.3, respectively; P d < 0.001), whereas coconut-based drinks had the highest saturated fat content (median, 1.1 g/100 mL; IQR, 0.9–1.7; P d = 0.952) of all PB drinks, but this was not significantly different than dairy milks. All PB drinks contained more fiber than dairy milks; however, only the drinks based on cereals and grains, legumes, and nuts and seeds were significantly higher in fiber when compared with dairy milks (for cereals and grains, and for legumes: median, 0.5 g/100 mL [IQRs, 0.2–0.8 and 0.2–0.6, respectively]; and for nuts and seeds, 0.3 g/100 mL [IQR, 0.3–0.5]; P d < 0.001).

Dairy milks contained a median of 4.7 g/100 mL total sugar (IQR, 4.3–5.0) and 39.1 mg/100 mL sodium (IQR, 33.6–43.3). Most PB drinks contained less total sugar than did dairy milks, but they had similar levels of sodium. However, the total sugar content was only significantly lower for coconut (median, 1.9 g/100 mL; IQR, 1.5–2.5), legumes (median, 1.9 g/100 mL; IQR, 0.5–2.6), and nut- and seed-based drinks (median, 2.4 g/100 mL; IQR, 0.2–3.3) when compared with dairy milks ( P d < 0.001). This is equivalent to 3.8 g and 4.8 g of total sugar/200.0 mL serving size, or, if these sugars are considered free, 15.2% and 19.2% of the maximum recommended 25.0 g average daily sugar intake. 68 The only PB drink group that was statistically different in sodium content compared with dairy milks was the group based on nuts and seeds (median, 47.2 mg/100 mL [IQR, 34.0–60.0]; P d = 0.032). This is equivalent to 94.4 mg of sodium (0.2 g of salt) per 200.0 mL serving size, or 4.0% of the maximum recommended 5.0 g average daily salt intake. 69 However, there were also some extreme outliers, some of which reported containing more than 3 times this amount of sodium per 200.0 mL, the equivalent of approximately 12.0% of the daily World Health Organization recommendation. 69

A few studies (n = 16) evaluated micronutrient data of PB drinks, reporting on 249 PB alternative products and 37 ABF comparators. Iodine was only reported in PB drinks, not in other types of PB products. Like PB meat alternatives, micronutrient content ranged vastly across all groups: some products contributed to the average daily requirement, whereas others were much less nutritious ( Table 3 and Supplementary file 2: Table S2 ). For example, the median calcium content for all PB drink categories was 120.0 mg/100 mL (IQRs as follows: cereals and grains, 120.0–120.0; coconut, 120.0–120.0; fruits and vegetables, 120.0–120.0; legumes, 120.0–120.0; nuts and seeds, 114.5–120.0) as compared with 116.7 mg/100 mL (IQR, 109.3–124.0) for dairy milks. However, none of the PB products (median, 0.0 μg/100 mL; IQR, 0.0–1.4) matched the iodine content of dairy milks (median, 24.9 μg/100 mL; IQR, 20.0–36.5).

Only 4 studies (evaluating 29 PB drinks and 11 dairy milk products) reported nutrient data from organic PB products. All evaluated different nutrients, hence no further pooling of results was possible for organic products as a subgroup. Protein results are reported in Supplementary file 1: Figure S2 and Supplementary file 2: Table S1 in the Supporting Information online, which show that, particularly, legume-based PB drinks typically match dairy milk in protein content.

Energy density, saturated fat, fiber, sugar, sodium and micronutrient content of plant-based yogurt alternatives

The 4 studies on PB yogurt alternatives evaluated 191 PB yogurt products with 90 dairy-based comparator products (unflavored and unsweetened). The overall nutritional composition of PB yogurts appears to show some variation by main primary ingredient (see Supplementary file 2 in the Supporting Information online ); however, formal disaggregated assessment of PB yogurts by primary ingredient was not possible, because that information was often not reported by authors. At an aggregate level, PB yogurts typically contained less saturated fat and sodium but had a higher energy density and higher total sugar and fiber content.

Only 2 studies evaluated micronutrient data of PB yogurts (excluding sodium) and, therefore, no further pooling of results was possible. No studies reported nutrient data from organic PB yogurts. Protein results are reported in Supplementary file 2: Table S1 in the Supporting Information online . Only the sample of a legume-based PB yogurts came close to matching dairy yogurt in protein content.

Energy density, saturated fat, fiber, sugar, sodium and micronutrient content of plant-based cheese alternatives

The 5 studies evaluating PB cheese alternatives reported on 163 PB cheese products with 143 dairy-based comparator products. PB cheese alternatives were the least nutritionally diverse foods. Where the primary ingredient of PB cheeses was known, this was mostly coconut oil ( Supplementary file 2 in the Supporting Information online ); however, like PB yogurts, the main ingredient was often not reported by authors.

The cheese comparators were reported to contain median values of 284.0 kcal/100 g energy density (IQR, 108.0–330.1), 14.0 g/100 g saturated fat (IQR, 11.0–17.3), and no fiber (0.0 g/100 g; IQR, 0.0–0.0). Most PB cheese subgroups were reported to have higher energy densities and higher saturated fat and fiber content. PB cheese based on nuts and seeds had the highest energy density (328.0 kcal/100 g [IQR, 306.0–328.0]; P d = 0.334]), whereas coconut oil-based cheese had the highest saturated fat content (21.0 g/100 g [IQR, 19.7–22.0]; P d < 0.001]), a significant difference, with 50.0% more than dairy cheese. Unlike PB drinks, PB meat, and PB yogurt alternatives, not all PB cheese contained fiber. Nut- and seed-based cheese had the highest fiber content (median, 2.5 g/100 g [IQR, 2.4–2.7]; P d < 0.001). Although the median fiber content of PB cheese made from coconut oil was 0.0 g/100 g (IQR, 0.0–1.7; P d = 0.011), some products did contain up to 5.9 g/100 g and, therefore, strong evidence was found that both PB cheese based on nuts and seeds and on coconut oil had significantly higher fiber content than did dairy cheese.

Most PB cheese contained less sugar and sodium than did dairy cheese, which had a median of 2.0 g/100 g (IQR, 0.5–5.0) and 720.0 mg/100 g (IQR, 560.0–1000.0), respectively, across the identified studies. In general, PB cheese alternatives had either no or minimal total sugar content. Finally, coconut oil–based cheese had the highest sodium content across all PB cheese (median, 714.0 mg/100 g [IQR, 600.0–880.0]; P d = 0.897), but this was similar to dairy cheese. PB cheese made of nuts and seeds had the lowest median sodium content (240.0 mg/100 g [IQR, 200.0–240.0]; P d = 0.001), which would equal 48.0 mg of sodium (0.1 g of salt) per 20.0 g serving size, or 2.0% of the recommended maximum daily salt intake 69 ; hence, this type of PB cheese had a large reduction in sodium compared with dairy cheese.

The micronutrient content of PB cheese was evaluated by only 2 studies. Only 1 product made of nuts and seeds was fortified with calcium, whereas coconut-based PB cheese was typically fortified with vitamin B 12 (median, 2.5 μg/100 g; IQR, 2.5–2.5). For dairy cheeses, these medians were 815.0 mg/100 g (IQR, 463.0–930.0) for calcium and 2.5 μg/100 g (IQR, 1.8–2.5) for vitamin B 12 .

No studies reported nutrient data from organic products. Protein results are reported in Supplementary file 2: Table S1 in the Supporting Information online. Nut- and seed-based cheese typically had the highest protein content, though it did not match the protein content of dairy cheese.

Health impacts and risk factors of novel plant-based foods

Eleven peer-reviewed studies were included in this review, 9 of which evaluated PB meat alternatives and 3 evaluated PB drinks ( Table 4 ) 84–94 (see Supplementary file 1, section 3.3, in the Supporting Information online for further details on the health outcomes). No health studies were found that evaluated consumption of PB cheese, yogurt, or egg alternatives; links between NPBFs and mental health outcomes; nor any grey literature evaluating any health outcomes.

Summary of the evidence on the health impacts and risks of novel plant-based foods

Health impacts and risk factors of plant-based meat alternatives

Studies of PB meats (n = 9) showed positive health outcomes when individuals switched from consuming ABFs. Three studies on mycoprotein consumption by both healthy and overweight adults found a positive association with lower glycemic markers, 84 reduced energy intake, 84 , 85 and insulin release. 85 Moreover, mycoprotein consumption was hypothesized to have a beneficial impact on the plasma lipidome. 86

Four studies with healthy adults evaluated PB meat alternatives consumption (other than mycoprotein). When considering the same caloric intake, consumption of PB meats was associated with a lower risk of cardiovascular disease than was consumption of ABFs, mostly by reducing fasting serum levels of trimethylamine- N -oxide, and low-density lipoprotein cholesterol concentrations, compared with ABF consumption. 87 Furthermore, consumption of PB meats was associated with a reduction in body weight as compared with meat consumers. 87 , 88 Lysine-enriched PB meat as a substitute for ABFs was reported to increase muscle protein synthesis rates, which is a biological process of building new protein cells via amino acids. 89 Last, the replacement of 4 meat-containing meals per week with PB meat alternatives elicited positive changes in the gut microbiome, with changes in the presence of butyrate-producing pathways and increased taxa. 90

Health impacts and risk factors of plant-based drinks

Studies assessing PB drinks (n = 3) only focused on almond and soy drinks. The main focus and health outcomes of these studies varied. Sun et al 91 researched the reduction in glycemic response in young adults consuming soy drink or bovine milk together with white bread. These authors found that both products had a similar glycemic response through different biological pathways. Dineva et al 92 assessed micronutrient content in PB drinks and found significantly lower iodine intake and urinary iodine concentration in people consuming only PB drinks, 93 highlighting the need for appropriate fortification as more people transition to eat more NPBFs. Finally, Shen et al 93 evaluated the impact of PB drinks on dental health and found that a soy drink with added sugar caused enamel demineralization, compared with dairy milk, which promoted remineralization.

Environmental impacts of novel plant-based foods

A total of 53 studies evaluated at least 1 environmental outcome, using the life cycle assessment method, evaluating 209 PB products and 91 ABFs as comparators. Most studies used life cycle assessment inventories, and some relied on data providers (n = 32) to calculate environmental footprints. System boundaries varied across studies, with the majority evaluating category impacts from cradle-to-retail (see Supplementary file 3 in the Supporting Information online ). Studies mainly assessed the effect of substituting ABFs with NPBFs on greenhouse gas emissions (GHGE) (n = 50), followed by blue-water footprint (WF) (n = 39) and land use (LU) (n = 17) ( Figure 3 and Supplementary file 1: Table S11 in the Supporting Information online ). Although methods, assumptions, and inventory data varied from 1 study to another, most studies consistently reported percentage reductions in GHGE and LU for the production of NPBFs as compared with ABFs. Wider differences were observed in blue WF.

Reduction of environmental impacts by respective funding source . Calculated as a percentage difference between each novel plant-based (PB) product (by product type and food group based on main primary ingredient [ie, predominant or core food item on the ingredient list]) in comparison with their respective reported baseline (eg, dairy milk and cheese, meat and poultry). See Supplementary file 3 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only. Studies reporting data on cooked PB products also found reductions in environmental impacts.

Environmental footprints of plant-based meat alternatives replacing meat and poultry

The 34 publications evaluating PB meat alternatives reported on 135 PB meat products with 53 ABF comparators. The percentage difference showed reductions of more than 70% in GHGE, LU, and WF for most products when shifting from ABFs to PB meat alternatives. GHGE reductions across PB meat groups, based on primary ingredients, were similar, with the largest reduction in GHGE seen for nut- and seed-based meats, with a median value of –94.2% (IQR, –94.4 to –93.4), whereas PB meats based on legumes had the smallest reduction (–86.1%; IQR, –88.6 to –77.5). Only 2 of 134 PB products had higher levels of GHGE than their ABF comparator. For LU, mycoprotein (median, 89.0%; IQR, –92.3 to –76.5) and nut- and seed-based meats (median, 89.5%; IQR, –90.0 to –89.0) had the largest reduction. Alternatively, legume-based meats had the smallest LU reductions (median, –71.2%; IQR, –84.7 to –47.6). Only 3 of 55 products had higher LU than their ABF comparator. Finally, the largest reduction of WF was observed in PB meats made of cereals and grains (median, –92.6%; IQR, –94.1 to –92.0), and the smallest was observed with products made of mycoprotein (median, –73.7%; IQR, –84.4 to –55.2). Nine of 51 products had a higher WF than their respective ABF counterparts. Specifically, when certain individual legume- and mycoprotein-based meats were compared with chicken, PB meat alternatives reported requiring between 2.7% and 339.0% more water, with the largest difference observed in a Swedish chicken comparator to mycoprotein-based meats. This variation was attributed to differences between feed types, rearing systems, and farm efficiency across countries. 74 Comparisons were also made between the upper limit footprint of mycoprotein-based items and the average or lower limit footprint of the ABF. Moreover, there were extreme outliers, with some PB meats reporting a water percentage difference of 8006.9%. The authors attributed this to soybeans’ substantial water demand during processing and lower yield per soybean. 74

Environmental footprints of plant-based drinks alternatives replacing dairy milk

The 21 publications evaluating PB drinks reported on 51 PB drink products with 13 ABF comparators. PB drinks also were associated with reductions in GHGE and LU when shifting from dairy milk to PB drinks. Fruit- and vegetable-based drinks had the largest reduction of GHGE (median, –90.2%; IQR, –90.8 to –90.2]), whereas PB drinks based on cereals and grains had the smallest reduction (median, –76.9%; IQR, –88.8 to –56.0). Only 2 products of 36 had an increase of GHGE when comparing soy- (40.0%) and almond-based (18.9%) drinks with dairy milk (equivalent to 0.3, 0.4, and 0.3 kg CO 2 eq/100 g, respectively). 95 Wider differences were observed on the LU percentage difference; however, reductions were found for all products (n = 13 PB drinks).

Cereal- and grain-based drinks had the largest reduction (median, –86.4%; IQR, –92.7 to –76.0), whereas legume-based drinks had the smallest LU reductions (median, –56.6%; IQR, –75.5 to –38.8). The magnitude of change in the percentage difference for WF varied considerably, although, these data were less frequently reported by authors (n = 11 PB drinks). Cereal- and grain-based drinks had the largest reduction (median, –85.0%; IQR, –88.7 to –71.0), whereas legume-based drinks had the smallest WF reductions (median, –67.6%; IQR, –73.9 to –42.2). Nut- and seed-based drinks presented contradictory evidence. For example, Grant and Hicks 95 observed that almond drinks (9241.9%) required considerably more water than soy (–35.6%) and dairy milks (equivalent to 109.3, 0.8, and 1.2 L/100 g, respectively); whereas Ritchie 96 found that an almond drink required half the amount of water (–40.87%) than dairy milk (equivalent to 37.2 and 62.8 L/100 g, respectively). Data were limited to these 2 products; hence, no further pooling of results was possible.

Environmental footprints of plant-based yogurt alternatives replacing dairy yogurt

The 2 publications evaluating 2 PB yogurt alternatives compared to 2 dairy yogurts. They reported GHGE reductions ranging between –64.7% and –52.9%. Analysis of LU and WF was not possible due to lack of a baseline, differences in methods, and system boundaries.

Environmental footprints of plant-based cheese alternatives replacing dairy cheese

The 2 publications evaluating PB cheese alternatives reported on 21 PB cheese products with 23 ABF comparators. Data on the environmental impacts were particularly from coconut oil–based cheese alternatives (n = 20). All coconut oil–based cheese alternatives had a large reduction in amounts of GHGE and LU (GHGE: median, –75.4% [IQR, –77.4 to –59.3]; LU: median, –83.1% [IQR, –83.8 to –80.6]). A smaller reduction was observed in WF (median, –45.1%; IQR, –52.0 to 38.5), with a higher WF being reported than for the ABF comparator for only 3 products.

Health effects and environmental impacts of novel plant-based foods

Studies that simultaneously assessed both health and environmental outcomes and/or nutrient profiles of NPBFs were pooled ( Figure 4 ). Only 1 study reported environmental outcomes together with diet-related health effects of PB meat alternatives, and this study found that free access to NPBFs was associated with greater weight loss and reduced dietary carbon and LU, as compared with a control arm. 88 From 93 references, 20 studies assessed the environmental outcome and nutrient content of NPBFs; only 6 studies evaluated the health effects and nutrient content of NPBFs (see Supplementary file 1: Table S9 in the Supporting Information online ).

Reduction of environmental outcomes and their associated nutrient outcomes of novel plant-based foods (NPBFs) compared with baseline (eg, dairy milk and cheese, meat and poultry), expressed in percentage difference . The y -axis shows the increase or decrease of the nutrient content (energy, fiber, sodium, and saturated fat) in comparison with baseline; and the x -axis shows the reduction (or increase) of the environmental categories. Three environmental categories are reported: greenhouse gas emissions (circles), land use (triangles), and blue-water use (squares). Three NPBFs are reported: plant-based (PB) cheese alternatives (brown), PB meat alternatives (purple), and PB drinks (orange). PB yogurts were not included due to the limited amount of data. See Supplementary file 2 in the Supporting Information online for detailed information on the baseline used for each reference. Data were limited to raw products only.

When compared with ABF counterparts, data suggest NPBFs are overwhelmingly associated with smaller environmental footprints. Data on nutritional profiles of NPBF were mixed: nutritional profiles for some NPBF groups were better aligned with healthy diets, but not for others. Clear co-benefits were observed for fiber intake from NPBFs. However, for the other nutrients, the picture was much more mixed due to the variability in content arising from differences in the main primary ingredients and the type of NPBFs.

Fruit, vegetable, legume, and nut content of novel plant-based foods

The percentage of fruit, vegetable, legume, and nut content in each NPBF in the United Kingdom was estimated as a case study ( Figure 5 ). Most NPBFs had at least 1 fruit, vegetable, legume, or nut, ranging from 0.0% to 100.0% of their weight. Overall, median content was low, with a few exceptions. PB meat alternatives had the highest content of vegetables and legumes, and PB cheese alternatives had the lowest content ( Supplementary file 1: Figure S5 and Supplementary file 2: Table S5 in the Supporting Information online ).

Estimated fruit, vegetable, legume, and nut content (%) in each novel plant-based foods product from time-stamped data from UK supermarkets . Panels show (a) plant-based (PB) drink alternatives; (b) PB meat alternatives; (c) PB cheese alternatives; and (d) PB yogurt alternatives.

Assessment of robustness and relevance of the included studies

For results on the assessment of robustness and relevance of the included studies see Supplementary file 1 : Table S12 in the Supporting Information online in section 3.6 .

Sensitivity analysis of funding sources of nutrient composition studies

Almost half of the nutrition studies included (n = 26; 46.4%) were funded by academic funders; 44.6% (n = 25) were fully funded or partially funded by industry; and 10.0% (n = 5) did not state their funding source. NPBF manufacturers were the support for the majority of industry-funded studies (n = 21; 37.5%), followed by the livestock industry (n = 3; 5.4%), and both (n = 1; 1.8%). The sensitivity analysis of the percentage difference for all the nutrients associated with the burden of disease, except total sugar, revealed that studies funded by industry were more likely to find differences than those funded by academia, with the former typically reporting more positive results on lower energy and saturated fat ( Table 5 and see Supplementary file 1: Table S13 in the Supporting Information online for sensitivity analysis on studies partially funded by the industry). However, the direction across all studies was the same: reductions in energy and saturated fat content, and increases in fiber, total sugar, and sodium content.

Sensitivity analysis, based on funding source, of the percentage difference between novel plant-based foods vs animal-based foods in nutrient content and environmental impacts a

The funding source of 6 articles were unknown, so they were excluded from this analysis. The superscript b and c indicate the direction and dimension of the association.

Industry-funded studies show a more positive impact on health and environmental outcomes of their PB products (vs animal sourced foods) as compared with academically funded studies.

Industry-funded studies show a less positive impact on health and environmental outcomes of their PB products (vs animal-sourced foods) as compared with academically funded studies.

Abbreviations : ABF, animal-based food; IQR, interquartile range; NPBF, novel plant-based food.

Sensitivity analysis of funding sources of health studies

Only 2 health studies were funded by academia; the rest of the studies were either partially or wholly funded by industry (n = 9). Most industry-funded studies were from NPBF manufacturers (n = 8); 1 study was partially funded by Dairy Australia.

Sensitivity analysis of funding sources of environmental studies

Compared with nutritional studies, a greater percentage of environmental studies were by industry researchers, particularly from NPBF manufacturers (67.9%). Approximately 71.7% of studies (n = 38) were fully funded or partially funded by industry; 26.4% (n = 14) were supported by academic funders; and 1.9% (n = 1) did not state their funding source. Of the industry-funded studies, only 2 (3.8%) were funded by the livestock industry. The sensitivity analysis revealed that the percentage differences were significantly larger between academic and industry funders in terms of GHGE and LU. Studies funded by industry typically reported more positive results on LU than did studies funded by academic funders, and the opposite was observed for GHGE. Like nutrient studies, the direction (decreases in GHGE, LU, and WF) was the same regardless of the funding source ( Table 5 , and see Supplementary file 1: Table S13 in the Supporting Information online for the sensitivity analysis of studies partially funded by the industry).

Research findings

We reviewed evidence from high-income countries that was published in peer-reviewed and grey literature within the past 7 years on nutrient content, and environmental and health outcomes of consuming NPBFs. Most NPBFs typically have much lower environmental impacts compared with ABFs, particularly with respect to GHGE and, to a lesser extent, to LU and WF. The nutrient content of NPBFs is highly variable in comparison to the nutrient profiles of ABFs. Although several individual NPBFs had positive health and environmental outcomes, co-benefits identified were not universal across all NPBFs and several trade-offs were identified. The main primary ingredient, type of product, processing techniques, and brand were all important determinants of health, and nutritional and environmental outcomes, findings that show the need for further subcategorization of NPBFs to better educate consumers and enable them to take informed decisions regarding the healthiness and sustainability of their diets and (potential) dietary changes.

Research in context

If carefully selected, certain NPBFs (particularly certain PB drinks and meat alternatives) could be an effective part of interventions to achieve net-zero and health targets in high-income countries. By applying a combination of strategies, enhanced uptake of these foods could improve the nutritional quality of diets, improve health, and contribute to tackling climate change impacts.

At the macronutrient level, NPBFs are generally the healthier option, given their higher fiber content and typically lower saturated fat and calorie contents, which could be advantageous in high-income (often obesogenic) settings. Certain types of NPBFs, particularly mycoprotein and legume-based meats, often also contain a substantial amount of fruit, vegetables, legumes, and/or nuts, which are food groups that are typically underconsumed in high-income settings. Composition of legume and fruit and vegetable-based drinks, were also typically consistent with healthier diets in high-income food secure settings, including low energy density, low total sugar, high fiber and low saturated fat content. Caution is recommended in the selection of these products if they were to be part of dietary recommendations, or standard institutional procurement for example, as certain NPBFs can also have higher levels of total sugar, sodium, and saturated fats in comparison to their respective ABF. This is particularly true for certain cereal and grain-based drinks, and coconut-based cheese and yogurts. Although the specific type of oil used in each NPBF product was not analyzed, coconut oil, which is high in saturated fatty acids, is often the ingredient that increases saturated fat levels in NPBFs to levels similar to its ABF counterparts. 51 , 75 Indeed, coconut oil-based cheese had approximately 50% more saturated fat than dairy cheese, and typically contained the least amount of fruit, vegetables, legumes or nuts, with the majority being absent.

In line with other evidence, 39 , 97 , 98 fortified NPBFs, in some cases, can be nutritionally comparable to their respective ABFs. Some individual NPBFs contained even higher concentrations of iron, vitamin B 12 , and calcium, whereas others did not. However, micronutrient assessment was difficult because not all included studies reported micronutrients. This could be because either NPBFs were unfortified or the information simply was not reported. Especially when nutrient information is gathered from supermarket websites for individual studies, micronutrient data are generally not reported.

The highly varying nutrient content across and within all PB products and categories may cause consumer confusion when individuals are looking for healthy and environmentally friendly alternatives to ABFs. Clearer front-of-package labelling of certain nutrients and information campaigns could reduce such confusion and better enable the consumer to make informed decisions about food purchases. 99 Potential development of rules and regulations on the food standards of NPBFs could also be a step forward in having a larger range of “healthy” NPBFs, because such regulations could potentially encourage reformulation of NPBFs, including the reduction of sodium, total sugar, and saturated fat content, and increased micronutrients. From a technological perspective, this is certainly possible. For example, new biotechnological techniques have been developed that enable companies to reduce sugar content and improve palatability, nutrient profile, and digestibility of PB drinks. 67 , 100–103 Some processing techniques can also decrease levels of anti-nutrients and polyphenols, which commonly are associated with low mineral and vitamin bioavailability, 35 , 98 , 101 , 104–107 and increase protein yield. 101 Given that specific raw materials, isolated proteins, processing levels, and fortification methods, often used in NPBFs, as compared with ABF nutrient profiles, are still debated in the scientific community, further research on the nutrient content and health risks related to bioavailability, bioaccessibility, and byproduct formation during industrial processes will reveal whether there are differences in terms of health impacts of “natural” vs more “isolated” nutrients. 30 , 108 , 109 More research into the metabolic profiles of NPBFs is imperative, particularly in light of a recent study identifying differences in the abundance of profiled metabolites between beef and PB burgers, despite their labelled nutritional similarities. 110 Instead of continuing the debate between the superiority of ABFs vs NPBFs, or vice versa, acknowledging and embracing their complementary differences can contribute to a less polarized dietary transition. This is especially relevant because emerging evidence has suggested that people who consume NPBFs also tend to purchase ABFs. 111

From the limited evidence on health, the inclusion of NPBFs into diets appears to typically have beneficial health effects, particularly the consumption of PB meat alternatives. The positive health effects mostly relate to better weight management and associated reduced risk of noncommunicable diseases in high-income (and often obesogenic) countries. This is aligned with a recently published meta-analysis that found positive outcomes on total cholesterol, low-density lipoprotein cholesterol and triglycerides when consuming PB meat alternatives as replacements for meat. 51 Furthermore, a few older studies also found positive health outcomes when assessing consumption of mycoprotein-based foods (eg, drinks, cookies, milkshakes, crisps) 112–115 and soy protein with isoflavones, 50 compared with consumption of dairy milk and/or meat products.

Previous evidence revealed that NPBFs are often regarded as healthier alternatives to ABFs 116 ; hence, it could be hypothesized that people may consume NPBFs in larger quantities than they would otherwise have done when eating ABFs. This may have negative health implications, especially if consumed regularly. Establishing a clear division in PB foods classifications, including ultraprocessed and less processed PB alternative foods, could enable better assessment of short- and long-term health impacts of NPBFs if they were to be consumed at an even larger scale. 116

Ultraprocessed foods have been associated with many diet-related diseases because these foods are generally energy dense and hyperpalatable. 117 , 118 Almost all NPBFs fall, technically, within this category; however, in this review, we found that the nutritional composition of some NPBFs aligns well with healthy dietary recommendations, such as having a high fiber content, low energy density, and low saturated fat content. Additionally, 1 of the included studies 90 also found positive associations with the gut microbiome when substituting meat in certain meals with PB meat alternatives. To get a better overview of the overall effect of NPBFs on health, more information and detailed analyses are needed regarding level of processing and gastrointestinal fate.

Consistent evidence was found regarding environmental outcomes, similar to previous research. 52 , 53 , 108 , 119–121 Most NPBFs had smaller environmental footprints than their ABF counterparts, with median reductions reported of up to 94.3%, 89.5%, and 92.6% for GHGE, LU, and WF, respectively. Nevertheless, some PB products had greater environmental impacts than their ABF counterparts, with some extreme outliers particularly in terms of WF. Although evidence was rather consistent, and the direction of effect appears to be clear, care should be taken not to overinterpret the exact numerical results: environmental impact calculations are notoriously context dependent and sensitive to methodological and data choices. This makes it impossible to come up with a summary figure that is representative for all products, produced in all countries; generally, however, there is a broad body of evidence demonstrating a reduction in GHGE, LU, and WF for a wide range of PB products in a wide variety of contexts compared with their ABF equivalents.

To improve the strategic use of NPBFs to achieve more sustainable food systems, life cycle assessments of these products should incorporate the full range of environmental impact categories, as well as sociocultural, economic, and health impacts with harmonized methods and assumptions across studies.

This study revealed an evidence gap for health impacts of NPBFs, including mental and dental health, and other risks associated with micronutrient deficiencies. There is also a lack of health studies on PB yogurts, PB cheese, and PB egg alternatives. Research on the health effects of PB drinks has been conducted with only certain products, “generally soy and almond drinks,” but there is a gap in knowledge about other PB drinks, such as those made from oat, potato, and hazelnut, among others. Furthermore, some concerns have been raised about the carbohydrate content in some PB drinks. A study by Jeske et al 122 revealed that the presence of β-glucan in many oat-based drinks causes a moderate glycemic index, despite the high carbohydrate content. In fact, Dhankhar 104 associated the consumption of oat drinks with high β-glucan levels with a reduction in cholesterol levels in study participants. However, this evidence needs to be updated to reflect the potential benefits of different types of PB drinks and current market brands. Although dairy products contain naturally occurring sugars from lactose, it is difficult to determine the breakdown of “natural” vs added sugars in NPBFs from the available literature. More research is also required on dental health to assess the potential risks of increased dental cavities due to lower calcium bioavailability, and the effects of free sugar content, pH levels, and buffering capacity in NPBFs.

Additional research is needed to provide more nutrient environmental and health evidence for PB yogurts, cheese, and egg alternatives. Last, although this review assessment focused on 3 environmental outcomes, evidence on other environmental impacts, including biodiversity loss and socioeconomic implications, is scarce. Across the 3 themes assessed in this review, better standardization and clear reporting of results in NPBF studies in the future would facilitate updates of this review.

Relevance for policy and practice

Minimally processed PB foods are still considered the gold standard for healthier and more sustainable diets. However, shifts from ABFs to PB whole foods remain problematic because, despite all the scientific knowledge about healthy eating, dietary change toward minimally processed PB foods has not been achieved. This review revealed that NPBFs can be healthier and more environmentally friendly alternatives to ABF consumption, if carefully selected. Although behavioral aspects are embedded in this transition, NPBFs could offer a convenient, novel, and potentially more realistic option to facilitate dietary transitions at large scale, diversifying diets, and increasing consumption of fruits, vegetables, legumes, and nuts without the need for significant individual dietary habits.

For potential promotion of the inclusion of NPBFs as part of public procurement or embedding them into food-based dietary guidelines, some of the consideration regarding varying healthiness of specific types of NPBFs and the need for further subclassifications needs to be carefully addressed. Furthermore, affordability is a concern because NPBFs often are more expensive than their ABF counterparts. Although comprehensively synthesizing price data was outside of the scope of this study, in the United Kingdom, the Food Foundation found that PB drinks are, on average, 50.0% more expensive than dairy milk. 71

Active promotion of NPBFs would require more detailed analysis of consumer behavior: current consumption of NPBFs is generally higher among younger generations, women, White populations, and those with higher education and incomes. 28 Better understanding of main drivers and barriers of consumption of NPBFs would allow targeted promotion to widen this consumer group. 71 NPBFs could play an additional role in reducing the prevalence of micronutrient deficiencies, especially given their reformulation and fortification potential. For example, in Finland, a mass fortification strategy of vitamin D across dairy and nondairy products has shown positive health outcomes over the past decade. 123 Finally, formalization, standardization, and accountability of environmental labelling could help consumers making informed decisions and avoid misinformation.

Strengths and limitations

To our knowledge, this is the first systematic review assessing the published peer-reviewed and grey literature evidence from studies that evaluated nutrient, and health and environmental impacts or benefits of NPBFs. A strict and comprehensive search string was developed to assess the full breadth of studies and reports, and machine-learning models were used to filter the large number of studies and systematically present all the available evidence on various NPBFs.

This study only covered the past 7 years to assess the current evidence, and an exhaustive cross-check of references was not performed, which possibly introduces reporting bias for missed relevant studies from previous years. However, it was assumed that only a small amount of additional findings had been missed, given the recent emergence of the variety and types of these novel products. Second, only 3 environmental impact categories were examined: carbon footprint, LU, and blue-water consumption. However, the heterogeneity of study designs, from system boundaries to geographical location, agricultural inputs, and methods used to calculate environmental footprints, made the review process too time consuming to expand on other environmental impacts in this particular study. Reliable reporting of environmental impacts of novel ingredients used in NPBFs, including added minerals and vitamins for fortification purposes, are generally missing in many studies. All the data reported by authors were collected and each study was compared individually against its own baseline (ie, the ABF comparator provided by author). Given the large spectrum of methods to determine environmental footprints, this could have introduced some bias; however, the alternative (using a standardized comparator) would equally have its limitations (eg, this would not be representative for all farming systems and products). Third, products and nutrients were assessed individually. Although the nutrient content gives some guidance on probable health risks, in reality, people consume diets in which individual compounds interact, influencing unknown biological pathways. Fourth, several studies that did not specifically report on the proportion and type of NPBF in (self-)reported PB diets had to be excluded. For those studies, it was impossible, therefore, to assess the effect on health and environment of NBPFs alone vs all PB foods together (ie, whole foods, NPBFs, other PB foods such as tofu and tempeh) and complicated any efforts to calculate dietary shifts. Finally, most studies did not report the precision of measures of effect (n = 68), making it difficult to pool and synthesize results across the 3 themes assessed in this review.

Food systems and diets need to change to meet environmental and health targets. This comprehensive systematic review presents a holistic approach to summarize the evidence on the nutrient, health, and environmental impacts of NPBF consumption. Although PB whole foods remain the preferred option on health grounds, some NPBFs have potential for being a useful steppingstone in the process of food system and dietary transformation, functioning as a healthy and environmentally friendly alternative to ABFs, if carefully selected. Reformulation and fortification could further enhance NPBFs as a viable and effective food group that could accelerate the dietary transition toward sustainable and healthy diets. However, given the great variability in nutritional composition of individual NPBFs, widespread promotion of such products should be introduced and addressed with caution. Given that NPBFs are already important in the food system and consumption is expected to continue to increase, a few steps are urgently required to guide consumers and enable them to make informed decisions regarding their diets. These include a further subdivision or categorization of NPBFs, which currently fall mainly in the ultraprocessed (hence, “unhealthy”) food category. Furthermore, standardized and verifiable environmental assessments of NPBFs are needed to compare foods with regard to their environmental footprints. Finally, more research on the short- and longer-term health effects of NPBFs is urgently required to facilitate informed decision-making on the inclusion of NPBFs as part of a wider net-zero and health strategy.

Gratitude is extended to the authors who responded to inquiries and generously shared their individual data. Additionally, sincere appreciation is expressed to the FoodDB team for sharing their time-stamped data set of observations from UK supermarkets to estimate the total fruit, vegetable, legume, and nut content of foods.

Author contributions. S.N.E. contributed to conceptualization of the study, methodology, investigation, data curation, formal analysis, coding and analysis of machine learning, and writing the original draft of the article. G.H. contributed to the formal analysis (screening process), data validation, and review and editing of the manuscript. A.J.S. led the coding and analysis of machine learning, and reviewed and edited the manuscript. C.A.-C. and G.T. contributed to the literature screening process, and reviewed and edited the manuscript. R.G. reviewed and edited the manuscript. S.P. and R.P. contributed to data validation, and reviewed and edited the manuscript. P.S. contributed to conceptualization of the study, methodology, reviewed and edited the manuscript, supervised the work, acquired funding, and contributed to project administration.

Funding. This work was supported by a research grant from the National Institute for Health Research, Health Protection Research Unit PhD Studentship in Environmental Change and Health (grant NIHR200909) and the af Jochnick Foundation.

The funders had no role in the conception, design, performance, and approval of this work.

Declaration of interest. The authors have no relevant interests to declare.

Data availability. Source code for this work is available online (DOI: 10.5281/zenodo.7116157).

The following supporting information is available through the online version of this article at the publisher’s website:

Supplementary file 1 .

Supplementary file 2 .

Supplementary file 3 .

Box 1 Key Definitions

Novel plant-based foods (NPBFs): Acknowledging differences in terminology for NPBFs, for the purpose of this review, the term novel plant-based foods is used to describe plant-based (PB) drinks and PB meat, cheese, eggs, and yogurt alternatives that are of plant or fungal origin and designed to directly replace or mimic animal-based foods. This definition includes fungi-based foods (ie, mycoprotein) that biologically do not belong to the plant kingdom but are typically “designed” similarly to NPBFs as a direct replacement for animal-based foods. Here, the term excludes tofu, tempeh, and seitan because although these might be novel to some high-income settings, they have been part of traditional Asian diets for centuries and, hence, are not subject to the same challenges and evidence gap as NPBFs.

Ultraprocessed: Foods that have undergone a series of industrial techniques and processes

Minimally processed plant-based foods: Plant-based whole foods such as nuts, seeds, cereals, and legumes

Food and Agriculture Organization , UNICEF, World Food Program, World Health Organization . The state of food security and nutrition in the world 2021. Transforming food systems for food security, improved nutrition and affordable healthy diets for all. 2021 . Available at: https://www.fao.org/3/cb4474en/online/cb4474en.html . Accessed February 2024.

Willett W , Rockstrom J , Jonell M , et al.    Food in the Anthropocene: the EAT-Lancet Commission on healthy diets from sustainable food systems . Lancet . 2019 ; 393 : 447 – 492 . doi: 10.1016/S0140-6736%2818%2931788-4 .

Google Scholar

Food and Agriculture Organization , UNICEF, World Food Program, World Health Organization . The state of food security and nutrition in the world 2023. 2023 . Available at: https://www.fao.org/3/CC3017EN/online/CC3017EN.html . Accessed February 2024.

Alae-Carew C , Nicoleau S , Bird FA , et al.    The impact of environmental changes on the yield and nutritional quality of fruits, nuts and seeds: a systematic review . Environ Res Lett . 2020 ; 15 : 023002 . doi: 10.1088/1748-9326/ab5cc0 .

Parajuli R , Thoma G , Matlock MD.   Environmental sustainability of fruit and vegetable production supply chains in the face of climate change: a review . Sci Total Environ . 2019 ; 650 : 2863 – 2879 . doi: 10.1016/j.scitotenv.2018.10.019 .

Wang J , Vanga S , Saxena R , et al.    Effect of climate change on the yield of cereal crops: a review . Climate . 2018 ; 6 : 41 . doi: 10.3390/cli6020041 .

Bisbis MB , Gruda N , Blanke M.   Potential impacts of climate change on vegetable production and product quality—a review . J Clean Prod . 2018 ; 170 : 1602 – 1620 . doi: 10.1016/j.jclepro.2017.09.224 .

Scheelbeek P , Bird FA , Tuomisto HL , et al.    Effect of environmental changes on vegetable and legume yields and nutritional quality . Proc Natl Acad Sci USA . 2018 ; 115 : 6804 – 6809 . doi: 10.1073/pnas.1800442115 .

Intergovernmental Panel on Climate Change ( IPCC) . Climate change 2022: impacts, adaptation and vulnerability. IPCC sixth assessment report. 2022 . Available at: https://www.ipcc.ch/report/ar6/wg2/ . Accessed February 2024.

United Nations . Sustainable development goals. Available at: https://www.un.org/en/sustainable-development-goals . Accessed June 2022.

Mendenhall E , Singer M.   The global syndemic of obesity, undernutrition, and climate change . Lancet . 2019 ; 393 : 741 . doi: 10.1016/s0140-6736(19)30310-1 .

Aggarwal A , Rehm CD , Monsivais P , et al.    Importance of taste, nutrition, cost and convenience in relation to diet quality: evidence of nutrition resilience among US adults using National Health and Nutrition Examination Survey (NHANES) 2007-2010 . Prev Med . 2016 ; 90 : 184 – 192 . doi: 10.1016/j.ypmed.2016.06.030 .

Davis R , Campbell R , Hildon Z , et al.    Theories of behaviour and behaviour change across the social and behavioural sciences: a scoping review . Health Psychol Rev . 2015 ; 9 : 323 – 344 . doi: 10.1080/17437199.2014.941722 .

Markowski KL , Roxburgh S.   "If I became a vegan, my family and friends would hate me”: anticipating vegan stigma as a barrier to plant-based diets . Appetite . 2019 ; 135 : 1 – 9 . doi: 10.1016/j.appet.2018.12.040 .

Lea EJ , Crawford D , Worsley A.   Consumers' readiness to eat a plant-based diet . Eur J Clin Nutr . 2006 ; 60 : 342 – 351 . doi: 10.1038/sj.ejcn.1602320 .

Good Food Institute . U.S. retail market data for the plant-based industry. Available at: https://gfi.org/marketresearch/#comparison-to-animal-based-foods . Accessed June 2022.

Spalding N. The APAC Alternative Protein Industry Report 2021. 2021 . Available at: https://www.greenqueen.com.hk/apac-alt-protein-report-2021/ . Accessed February 2024.

Gaan K. Plant-based meat, eggs and dairy. 2020 The State of the Industry Report. 2020 . Available at: https://gfi.org/resource/plant-based-meat-eggs-and-dairy-state-of-the-industry-report/ . Accessed February 2024.

Smart Protein Project . Plant-based foods in Europe: how big is the market ? Smart Protein Plant-Based Food Sector Report. 2021 . Available at: https://smartproteinproject.eu/plant-based-food-sector-report . Accessed February 2024.

Devlin E. This raises £15m in series B funding round to fuel ongoing rapid growth. Available at: https://www.thegrocer.co.uk/fundraising/this-raises-15m-in-series-b-funding-round-to-fuel-ongoing-rapid-growth/676873.article . Accessed April 2023.

Ettinger J. Heura saw a 260% increase in international vegan meat sales in 2022. Green Queen. https://www.greenqueen.com.hk/heura-2022-international-vegan-meat-sales/ . Accessed March 2023.

Datassential Research . 2023 Food trends. FOODBYTES: trend report. 2022 . Available at: https://offers.datassential.com/2023-trends . Accessed February 2024.

Foster-Collins L. The death of vegan brands and the unlikely saviour of plant-based diets. Available at: https://www.thegrocer.co.uk/plant-based/the-death-of-vegan-brands-and-saviour-of-plant-based-diets/680057.article?utm_source=Weekly%20Shop%20(The%20Grocer)&utm_medium=email&utm_campaign=2023-06-12&c=&cid=DM1080641&bid=132435027 . Accessed June 2023.

Tatum M. Is Beyond Meat beyond hope? And what its slump says about the other plant-based players. The Grocer . July 9, 2022. Available at: https://www.thegrocer.co.uk/plant-based/is-beyond-meat-beyond-hope-and-what-its-slump-says-about-the-other-plant-based-players/669272.article . Accessed March 2023.

Good Food Institute . 2022 State of the Industry Report: plant-based meat, seafood, eggs, and dairy. 2022 . Available at: https://gfi.org/resource/plant-based-meat-eggs-and-dairy-state-of-the-industry-report/ . Accessed February 2024.

EAT, GlobeScan . Grains of Truth 2: a consumer report on healthy and sustainable food system. 2022 . Available at: https://eatforum.org/content/uploads/2022/11/EAT-Forum-Report.pdf . Accessed February 2024.

Good Food Institute . U.S. retail market insights for the plant-based industry. Available at: https://gfi.org/marketresearch/#purchase-dynamics . Accessed October 2023.

Alae-Carew C , Green R , Stewart C , et al.    The role of plant-based alternative foods in sustainable and healthy food systems: consumption trends in the UK . Sci Total Environ . 2021 ; 807 : 151041 . doi: 10.1016/j.scitotenv.2021.151041 .

Frontier F , Lichtenstein T. Plant-based meat: a healthier choice? 2020 . A comprehensive health and nutrition analysis of plant-based meat products in the Australian and New Zealand markets. Available at: https://www.foodfrontier.org/wp-content/uploads/2022/10/Plant-Based_Meat_A_Healthier_Choice-1.pdf . Accessed February 2024.

World Economic Forum . Meat: The Future series. Alternative Proteins. 2019 . Available at: https://www3.weforum.org/docs/WEF_White_Paper_Alternative_Proteins.pdf . Accessed February 2024.

Sahakian M , Fuchs D , Lorek S , et al.    Advancing the concept of consumption corridors and exploring its implications . Sustainability . 2021 ; 17 : 305 – 315 . doi: 10.1080/15487733.2021.1919437 .

Noguerol AT , Pagan MJ , Garcia-Segovia P , et al.    Green or clean? Perception of clean label plant-based products by omnivorous, vegan, vegetarian and flexitarian consumers . Food Res Int . 2021 ; 149 : 110652 . doi: 10.1016/j.foodres.2021.110652 .

Mertens E , Biesbroek S , Dofkova M , et al.    Potential impact of meat replacers on nutrient quality and greenhouse gas emissions of diets in four European countries . Sustainability . 2020 ; 12 : 6838 .

Edge MS , Garrett JL.   The nutrition limitations of mimicking meat . Cereal Foods World . 2020 ; 65 : 3 . doi: 10.1094/CFW-65-4-0045 .

Paul AA , Kumar S , Kumar V , et al.    Milk analog: plant based alternatives to conventional milk, production, potential and health concerns . Crit Rev Food Sci Nutr . 2020 ; 60 : 3005 – 3023 .

Verduci E , D'Elios S , Cerrato L , et al.    Cow's milk substitutes for children: nutritional aspects of milk from different mammalian species, special formula and plant-based beverages . Nutrients   2019 ; 11 : 1739 .

Chalupa-Krebzdak S , Long CJ , Bohrer BM.   Nutrient density and nutritional value of milk and plant-based milk alternatives . Int Dairy J . 2018 ; 87 : 84 – 92 .

Kazir M , Livney YD.   Plant-based seafood analogs . Molecules . 2021 ; 26 : 1559 .

Silva ARA , Silva MMN , Ribeiro BD.   Health issues and technological aspects of plant-based alternative milk . Food Res Int . 2020 ; 131 : 108972 .

McClements DJ , Newman E , McClements IF.   Plant-based milks: a review of the science underpinning their design, fabrication, and performance . Compr Rev Food Sci Food Saf . 2019 ; 18 : 2047 – 2067 .

Hadi J , Brightwell G.   Safety of alternative proteins: technological, environmental and regulatory aspects of cultured meat, plant-based meat, insect protein and single-cell protein . Foods . 2021 ; 10 : 1226 .

Liu S , Sha L.   Nutrition and safety assessment of plant protein-based meat alternative products . Food Ferment Industr . 2021 ; 47 : 297 – 303 .

Kołodziejczak K , Onopiuk A , Szpicer A , et al.    Meat analogues in the perspective of recent scientific research: a review . Foods . 2021 ; 11 : 105 . doi: 10.3390/foods11010105 .

Lonkila A , Kaljonen M.   Promises of meat and milk alternatives: an integrative literature review on emergent research themes . Agric Hum Values . 2021 ; 38 : 625 – 639 .

Boukid F.   Plant-based meat analogues: from niche to mainstream . Eur Food Res Technol . 2020 ; 247 : 297 – 308 .

Good Food Institute . Plant-Based Meat for a Growing World. 2019 . Available at: https://gfi.org/wp-content/uploads/2021/02/GFI-Plant-Based-Meat-Fact-Sheet_Environmental-Comparison.pdf . Accessed February 2023.

Nawaz MA , Tan MV , Oiseth S , et al.    An emerging segment of functional legume-based beverages: a review . Food Rev Int . 2020 ; 38 : 1064 – 1102 .

Boukid F , Rosell CM , Rosene S , et al.    Non-animal proteins as cutting-edge ingredients to reformulate animal-free foodstuffs: present status and future perspectives . Crit Rev Food Sci Nutr . 2022 ; 62 : 6390 – 6420 .

Vatansever S , Tulbek MC , Riaz MN.   Low- and high-moisture extrusion of pulse proteins as plant-based meat ingredients: a review . Cereal Foods World . 2020 ; 65 : 12 – 14 .

Chalvon-Demersay T , Azzout-Marniche D , Arfsten J , et al.    A systematic review of the effects of plant compared with animal protein sources on features of metabolic syndrome . J Nutr . 2017 2017; 147 : 281 – 292 .

Gibbs J , Leung G-K.   The effect of plant-based and mycoprotein-based meat substitute consumption on cardiometabolic risk factors: a systematic review and meta-analysis of controlled intervention trials . Dietetics . 2023 ; 2 : 104 – 122 . doi: 10.3390/dietetics2010009 .

Silva BQ , Smetana S.   Review on milk substitutes from an environmental and nutritional point of view . Appl Food Res . 2022 ; 2 : 100105 . doi: 10.1016/j.afres.2022.100105 .

Bryant CJ.   Plant-based animal product alternatives are healthier and more environmentally sustainable than animal products . Future Foods . 2022 ; 6 : 100174 . doi: 10.1016/j.fufo.2022.100174 .

Nájera Espinosa S , Pastorino S , Scheelbeek P. Mapping the evidence of plant-based alternative foods: a systematic literature review protocol on nutritional, health, and environmental impacts. 2022 . Available at: https://datacompass.lshtm.ac.uk/id/eprint/3015/1/Lit_Review_Protocol.pdf . Accessed December 2024.

Page MJ , Moher D , Bossuyt PM , et al.    PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews . BMJ . 2021 ; 372 : n160 . doi: 10.1136/bmj.n160 .

Chang C-C , Lin C-J.   LIBSVM: a library for support vector machines . ACM Trans Intell Syst Technol . 2011 ; 2 : 1 – 27 . doi: 10.1145/1961189.1961199 .

Pedregosa F , Varoquaux G , Gramfort A , et al.    Scikit-learn: machine learning in Python . J Mach Learn Res . 2011 ; 12 : 2825 – 2830 .

Berrang-Ford L , Sietsma AJ , Callaghan M , et al.    Systematic mapping of global research on climate and health: a machine learning review . Lancet Planet Health . 2021 ; 5 : e514 – 25 . doi: 10.1016/s2542-5196(21)00179-0 .

Renotte N. nicknochnack/Longform-Summarization-with-Hugging-Face. Available at: https://github.com/nicknochnack/Longform-Summarization-with-Hugging-Face/blob/main/LongSummarization.ipynb . Accessed September 2022.

Pistilli M. Plant-based food stocks: 10 biggest companies in 2023. Available at: https://investingnews.com/daily/life-science-investing/top-plant-based-food-stocks/ . Accessed May 2023.

Team V . The big list of vegan food companies. Available at: https://www.veganrecipeclub.org.uk/articles/big-list-vegan-food-companies/ . Accessed May 2023.

Meticulous Research . Top 10 companies in plant based food market. Available at: https://meticulousblog.org/top-10-companies-in-plant-based-food-market/ . Accessed May 2023.

Enerva K. Top plant-based food companies in the UK. Available at: https://viable.earth/plant-based-food/top-plant-based-food-companies-in-the-uk/ . Accessed May 2023.

Critical Appraisal Skills Programme . CASP checklist: CASP randomised controlled trial checklist. Available at: https://casp-uk.net/casp-tools-checklists/ . Accessed March 2023.

Harrington RA , Adhikari V , Rayner M , et al.    Nutrient composition databases in the age of big data: FoodDB, a comprehensive, real-time database infrastructure . BMJ Open . 2019 ; 9 : e026652 . doi: 10.1136/bmjopen-2018-026652 .

Reyes-Jurado F , Soto-Reyes N , Davila-Rodriguez M , et al.    Plant-based milk alternatives: types, processes, benefits, and characteristics . Food Rev Int . 2021 ; 39 : 2320 – 2351 .

Tangyu M , Muller J , Bolten CJ , et al.    Fermentation of plant-based milk alternatives for improved flavour and nutritional value . Appl Microbiol Biotechnol . 2019 ; 103 : 9263 – 9275 .

World Health Organization . Guideline: sugars intake for adults and children. World Health Organization;   2015 . Available at: https://www.who.int/publications/i/item/9789241549028 . Accessed February 2024.

World Health Organization . Salt intake. World Health Organization. Available at: https://www.who.int/data/gho/indicator-metadata-registry/imr-details/3082 . Accessed October 2022.

Sumner O , Burbridge L.   Plant-based milks: the dental perspective . Br Dent J . 2020 ; 8 : 16 – 23 .

Goudie S , Hughes I. The Broken Plate Report 2022. 2022 . Available at: https://foodfoundation.org.uk/publication/broken-plate-2022 . Accessed February 2024.

Collard KM , McCormick DP.   A nutritional comparison of cow's milk and alternative milk products . Acad Pediatr . 2020 ; 21 : 1067 – 1069 .

Marques M , Correig E , Capdevila E , et al.    Essential and non-essential trace elements in milks and plant-based drinks . Biol Trace Elem Res . 2021 ; 200 : 4524 – 4533 . doi: 10.1007/s12011-021-03021-5 .

Kazer J , Orfanos G , Gallop C. Quorn Footprint Comparison Report. Carbon Trust; 2022 . Available at: https://www.quorn.co.uk/assets/files/content/Carbon-Trust-Comparison-Report-2022.pdf .

De Marchi M , Costa A , Pozza M , et al.    Detailed characterization of plant-based burgers . Sci Rep . 2021 ; 11 : 2049 .

Zhang YY , Hughes J , Grafenauer S.   Got Mylk? The emerging role of Australian plant-based milk alternatives as a cow's milk substitute . Nutrients . 2020 ; 12 : 1254 .

Isidro V.   The nutritional limitations of plant-based beverages in infancy and childhood . Nutr Hosp . 2017 ; 34 : 1205 – 1214 .

Farsi DN , Uthumange D , Munoz Munoz J , et al.    The nutritional impact of replacing dietary meat with meat alternatives in the UK: a modelling analysis using nationally representative data . Br J Nutr . 2021 ; 127 : 1 – 31 .

Bryngelsson S , Moshtaghian H , Bianchi M , et al.    Nutritional assessment of plant-based meat analogues on the Swedish market . Int J Food Sci Nutr . 2022 ; 73 : 889 – 901 . doi: 10.1080/09637486.2022.2078286 .

Seves SM , Verkaik-Kloosterman J , Biesbroek S , et al.    Are more environmentally sustainable diets with less meat and dairy nutritionally adequate?   Public Health Nutr . 2017 ; 20 : 2050 – 2062 .

Goldstein B , Moses R , Sammons N , et al.    Potential to curb the environmental burdens of American beef consumption using a novel plant-based beef substitute . PLoS One . 2017 ; 12 : E 0189029 .

Kazer J , Orfanos G , Gallop C. Quorn Footprint Comparison Report. Carbon Trust; 2021 . Available at: https://www.quorn.co.uk/assets/files/content/Carbon-Trust-Comparison-Report-2021.pdf .

Allen LH , Carriquiry AL , Murphy SP.   Perspective: proposed harmonized nutrient reference values for populations . Adv Nutr . 2020 ; 11 : 469 – 483 . doi: 10.1093/advances/nmz096 .

Cherta-Murillo A , Frost GS.   The association of mycoprotein-based food consumption with diet quality, energy intake and non-communicable diseases' risk in the UK adult population using the National Diet and Nutrition Survey (NDNS) years 2008/2009-2016/2017: a cross-sectional study . Br J Nutr . 2021 ; 127 : 1685 – 1694 .

Bottin JH , Swann JR , Cropp E , et al.    Mycoprotein reduces energy intake and postprandial insulin release without altering glucagon-like peptide-1 and peptide tyrosine-tyrosine concentrations in healthy overweight and obese adults: a randomised-controlled trial . Br J Nutr . 2016 ; 116 : 360 – 374 .

Coelho MOC , Monteyne AJ , Dirks ML , et al.    Daily mycoprotein consumption for 1 week does not affect insulin sensitivity or glycaemic control but modulates the plasma lipidome in healthy adults: a randomised controlled trial . Br J Nutr . 2021 ; 125 : 147 – 160 .

Crimarco A , Springfield S , Petlura C , et al.    A randomized crossover trial on the effect of plant-based compared with animal-based meat on trimethylamine- N -oxide and cardiovascular disease risk factors in generally healthy adults: Study With Appetizing Plantfood-Meat Eating Alternative Trial (SWAP-MEAT) . Am J Clin Nutr . 2020 ; 112 : 1188 – 1199 .

Bianchi F , Stewart C , Astbury NM , et al.    Replacing meat with alternative plant-based products (RE-MAP): a randomized controlled trial of a multicomponent behavioral intervention to reduce meat consumption . Am J Clin Nutr . 2021 ; 115 : 1357 – 1366 . doi: 10.1093/ajcn/nqab414 .

Kouw IWK , Pinckaers PJM , Le Bourgot C , et al.    Ingestion of an ample amount of meat substitute based on a lysine-enriched, plant-based protein blend stimulates postprandial muscle protein synthesis to a similar extent as an isonitrogenous amount of chicken in healthy, young men . Br J Nutr . 2021 ; 128 : 1 – 1965 . doi: 10.1017/S0007114521004906 .

Toribio-Mateas MA , Bester A , Klimenko N.   Impact of plant-based meat alternatives on the gut microbiota of consumers: a real-world study . Foods . 2021 ; 10 : 2040 . doi: 10.3390/foods10092040 .

Sun LJ , Tan KWJ , Siow PC , et al.    Soya milk exerts different effects on plasma amino acid responses and incretin hormone secretion compared with cows' milk in healthy, young men . Br J Nutr . 2016 ; 116 : 1216 – 1221 .

Dineva M , Rayman MP , Bath SC.   Iodine status of consumers of milk-alternative drinks v. cows' milk: data from the UK National Diet and Nutrition Survey . Br J Nutr . 2021 ; 126 : 28 – 36 .

Shen P , Walker GD , Yuan Y , et al.    Effects of soy and bovine milk beverages on enamel mineral content in a randomized, double-blind in situ clinical study . J Dent . 2019 ; 88 : 103160 . doi: 10.1016/j.jdent.2019.06.007 .

Farsi DN , Gallegos JL , Koutsidis G , et al.    Substituting meat for mycoprotein reduces genotoxicity and increases the abundance of beneficial microbes in the gut: Mycomeat, a randomised crossover control trial . Eur J Nutr . 2023 ; 62 : 1479 – 1492 . doi: 10.1007/s00394-023-03088-x .

Grant CA , Hicks AL.   Comparative life cycle assessment of milk and plant-based alternatives . Environ Eng Sci . 2018 ; 35 : 1235 – 1247 .

Ritchie H. Dairy vs. plant-based milk: what are the environmental impacts? Available at: https://ourworldindata.org/environmental-impact-milks . Accessed May 2022.

Saget S , Costa M , Santos CS , et al.    Substitution of beef with pea protein reduces the environmental footprint of meat balls whilst supporting health and climate stabilisation goals . J Clean Prod . 2021 ; 297 : 126447 .

Rubio NR , Xiang N , Kaplan DL.   Plant-based and cell-based approaches to meat production . Nat Commun . 2020 ; 11 : 6276 .

Julia C , Fialon M , Galan P , et al.    Are foods ‘healthy’ or ‘healthier’? Front-of-pack labelling and the concept of healthiness applied to foods . Br J Nutr . 2022 ; 127 : 948 – 952 . doi: 10.1017/S0007114521001458 .

Boeck T , Zannini E , Sahin AW , et al.    Nutritional and rheological features of lentil protein isolate for yoghurt-like application . Foods . 2021 ; 10 : 1692 .

Han H , Choi JK , Park J , et al.    Recent innovations in processing technologies for improvement of nutritional quality of soymilk . CyTA-J Food . 2021 ; 19 : 287 – 303 .

Kumari M , Kokkiligadda A , Dasriya V , et al.    Functional relevance and health benefits of soymilk fermented by lactic acid bacteria . J Appl Microbiol . 2021 ; 133 : 104 – 119 . doi: 10.1111/jam.15342 .

Jeske S , Zannini E , Lynch KM , et al.    Polyol-producing lactic acid bacteria isolated from sourdough and their application to reduce sugar in a quinoa-based milk substitute . Int J Food Microbiol . 2018 ; 286 : 31 – 36 .

Dhankhar J.   Perspective on the pros and cons, manufacturing aspects, and recent advances in non-dairy milk alternatives . J Microb Biotech Food Sci . 2023 ; 12 : e9543 . doi: 10.55251/jmbfs.9543 .

Boeck T , Sahin AW , Zannini E , et al.    Nutritional properties and health aspects of pulses and their use in plant-based yogurt alternatives . Compr Rev Food Sci Food Saf . 2021 ; 20 : 3858 – 3880 .

Aydar EF , Tutuncu S , Ozcelik B.   Plant-based milk substitutes: bioactive compounds, conventional and novel processes, bioavailability studies, and health effects . J Funct Foods . 2020 ; 70 : 103975 .

Dahdouh S , Grande F , Espinosa SN , et al.    Development of the FAO/INFOODS/IZINCG global food composition database for phytate . J Food Compost Anal . 2019 ; 78 : 42 – 48 . doi: 10.1016/j.jfca.2019.01.023 .

van Vliet S , Kronberg SL , Provenza FD.   Plant-based meats, human health, and climate change . Front Sustain Food Syst . 2020 ; 4 : 128 .

WHO European Office for the Prevention and Control of Noncommunicable Diseases . Plant-based diets and their impact on health, sustainability and the environment: a review of the evidence. WHO Regional Office for Europe ; 2021 . Available at: https://www.who.int/europe/publications/i/item/WHO-EURO-2021-4007-43766-61591 . Accessed February 2024.

van Vliet S , Bain JR , Muehlbauer MJ , et al.    A metabolomics comparison of plant-based meat and grass-fed meat indicates large nutritional differences despite comparable Nutrition Facts panels . Sci Rep . 2021 ; 11 : 13828 .

Neuhofer ZT , Lusk JL.   Most plant-based meat alternative buyers also buy meat: an analysis of household demographics, habit formation, and buying behavior among meat alternative buyers . Sci Rep . 2022 ; 12 : 13062 . doi: 10.1038/s41598-022-16996-5 .

Coelho MOC , Monteyne AJ , Dunlop M , et al.    Mycoprotein as a possible alternative source of dietary protein to support muscle and metabolic health . Nutr Rev . 2020 ; 78 : 486 – 497 .

Derbyshire E , Ayoob KT.   Mycoprotein: nutritional and health properties . Nutr Today . 2019 ; 54 : 7 – 15 .

Souza Filho PF , Andersson D , Ferreira JA , et al.    Mycoprotein: environmental impact and health aspects . World J Microbiol Biotechnol . 2019 ; 35 : 147 .

Dunlop MV , Kilroe SP , Bowtell JL , et al.    Mycoprotein represents a bioavailable and insulinotropic non-animal-derived dietary protein source: a dose-response study . Br J Nutr . 2017 ; 118 : 673 – 685 .

Otis BO , Hu FB , McCarthy G.   Can plant-based meat alternatives be part of a healthy and sustainable diet?   J Am Med Assoc . 2019 ; 322 : 1547 – 1548 .

Monteiro CA , Cannon G , Levy RB , et al.    Ultra-processed foods: what they are and how to identify them . Public Health Nutr . 2019 ; 22 : 936 – 941 . doi: 10.1017/S1368980018003762 .

Pagliai G , Dinu M , Madarena MP , et al.    Consumption of ultra-processed foods and health status: a systematic review and meta-analysis . Br J Nutr . 2021 ; 125 : 308 – 318 . doi: 10.1017/S0007114520002688 .

Carlsson Kanyama A , Hedin B , Katzeff C.   Differences in environmental impact between plant-based alternatives to dairy and dairy products: a systematic literature review . Sustainability . 2021 ; 13 : 12599 . doi: 10.3390/su132212599 .

Li Y.   Feeding the future: plant-based meat for global food security and environmental sustainability . Cereal Foods World . 2020 ; 65 : 8 – 11 .

Santo RE , Kim BF , Goldman SE , et al.    Considering plant-based meat substitutes and cell-based meats: a public health and food systems perspective . Front Sustain Food Syst . 2020 ; 4 : 134 .

Jeske S , Zannini E , Arendt EK.   Evaluation of physicochemical and glycaemic properties of commercial plant-based milk substitutes . Plant Foods Hum Nutr . 2017 ; 72 : 26 – 33 .

Jaaskelainen T , Itkonen ST , Lundqvist A , et al.    The positive impact of general vitamin D food fortification policy on vitamin D status in a representative adult Finnish population: evidence from an 11-y follow-up based on standardized 25-hydroxyvitamin D data . Am J Clin Nutr . 2017 ; 105 : 1512 – 1520 . doi: 10.3945/ajcn.116.151415 .

• alcoholic beverages
• micronutrients
• meat substitutes

Supplementary data

Email alerts, citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1753-4887
• Print ISSN 0029-6643
• Copyright © 2024 International Life Sciences Institute
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Efficacy of psilocybin...

Efficacy of psilocybin for treating symptoms of depression: systematic review and meta-analysis

Linked editorial.

Psilocybin for depression

• Related content
• Peer review

This article has a correction. Please see:

• EXPRESSION OF CONCERN: Efficacy of psilocybin for treating symptoms of depression: systematic review and meta-analysis - May 04, 2024
• Athina-Marina Metaxa , masters graduate researcher 1 ,
• Mike Clarke , professor 2
• 1 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK
• 2 Northern Ireland Methodology Hub, Centre for Public Health, ICS-A Royal Hospitals, Belfast, Ireland, UK
• Correspondence to: A-M Metaxa athina.metaxa{at}hmc.ox.ac.uk (or @Athina_Metaxa12 on X)
• Accepted 6 March 2024

Objective To determine the efficacy of psilocybin as an antidepressant compared with placebo or non-psychoactive drugs.

Design Systematic review and meta-analysis.

Data sources Five electronic databases of published literature (Cochrane Central Register of Controlled Trials, Medline, Embase, Science Citation Index and Conference Proceedings Citation Index, and PsycInfo) and four databases of unpublished and international literature (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform, ProQuest Dissertations and Theses Global, and PsycEXTRA), and handsearching of reference lists, conference proceedings, and abstracts.

Data synthesis and study quality Information on potential treatment effect moderators was extracted, including depression type (primary or secondary), previous use of psychedelics, psilocybin dosage, type of outcome measure (clinician rated or self-reported), and personal characteristics (eg, age, sex). Data were synthesised using a random effects meta-analysis model, and observed heterogeneity and the effect of covariates were investigated with subgroup analyses and metaregression. Hedges’ g was used as a measure of treatment effect size, to account for small sample effects and substantial differences between the included studies’ sample sizes. Study quality was appraised using Cochrane’s Risk of Bias 2 tool, and the quality of the aggregated evidence was evaluated using GRADE guidelines.

Eligibility criteria Randomised trials in which psilocybin was administered as a standalone treatment for adults with clinically significant symptoms of depression and change in symptoms was measured using a validated clinician rated or self-report scale. Studies with directive psychotherapy were included if the psychotherapeutic component was present in both experimental and control conditions. Participants with depression regardless of comorbidities (eg, cancer) were eligible.

Results Meta-analysis on 436 participants (228 female participants), average age 36-60 years, from seven of the nine included studies showed a significant benefit of psilocybin (Hedges’ g=1.64, 95% confidence interval (CI) 0.55 to 2.73, P<0.001) on change in depression scores compared with comparator treatment. Subgroup analyses and metaregressions indicated that having secondary depression (Hedges’ g=3.25, 95% CI 0.97 to 5.53), being assessed with self-report depression scales such as the Beck depression inventory (3.25, 0.97 to 5.53), and older age and previous use of psychedelics (metaregression coefficient 0.16, 95% CI 0.08 to 0.24 and 4.2, 1.5 to 6.9, respectively) were correlated with greater improvements in symptoms. All studies had a low risk of bias, but the change from baseline metric was associated with high heterogeneity and a statistically significant risk of small study bias, resulting in a low certainty of evidence rating.

Conclusion Treatment effects of psilocybin were significantly larger among patients with secondary depression, when self-report scales were used to measure symptoms of depression, and when participants had previously used psychedelics. Further research is thus required to delineate the influence of expectancy effects, moderating factors, and treatment delivery on the efficacy of psilocybin as an antidepressant.

Systematic review registration PROSPERO CRD42023388065.

• Download figure
• Open in new tab
• Download powerpoint

Introduction

Depression affects an estimated 300 million people around the world, an increase of nearly 20% over the past decade. 1 Worldwide, depression is also the leading cause of disability. 2

Drugs for depression are widely available but these seem to have limited efficacy, can have serious adverse effects, and are associated with low patient adherence. 3 4 Importantly, the treatment effects of antidepressant drugs do not appear until 4-7 weeks after the start of treatment, and remission of symptoms can take months. 4 5 Additionally, the likelihood of relapse is high, with 40-60% of people with depression experiencing a further depressive episode, and the chance of relapse increasing with each subsequent episode. 6 7

Since the early 2000s, the naturally occurring serotonergic hallucinogen psilocybin, found in several species of mushrooms, has been widely discussed as a potential treatment for depression. 8 9 Psilocybin’s mechanism of action differs from that of classic selective serotonin reuptake inhibitors (SSRIs) and might improve the treatment response rate, decrease time to improvement of symptoms, and prevent relapse post-remission. Moreover, more recent assessments of harm have consistently reported that psilocybin generally has low addictive potential and toxicity and that it can be administered safely under clinical supervision. 10

The renewed interest in psilocybin’s antidepressive effects led to several clinical trials on treatment resistant depression, 11 12 major depressive disorder, 13 and depression related to physical illness. 14 15 16 17 These trials mostly reported positive efficacy findings, showing reductions in symptoms of depression within a few hours to a few days after one dose or two doses of psilocybin. 11 12 13 16 17 18 These studies reported only minimal adverse effects, however, and drug harm assessments in healthy volunteers indicated that psilocybin does not induce physiological toxicity, is not addictive, and does not lead to withdrawal. 19 20 Nevertheless, these findings should be interpreted with caution owing to the small sample sizes and open label design of some of these studies. 11 21

Several systematic reviews and meta-analyses since the early 2000s have investigated the use of psilocybin to treat symptoms of depression. Most found encouraging results, but as well as people with depression some included healthy volunteers, 22 and most combined data from studies of multiple serotonergic psychedelics, 23 24 25 even though each compound has unique neurobiological effects and mechanisms of action. 26 27 28 Furthermore, many systematic reviews included non-randomised studies and studies in which psilocybin was tested in conjunction with psychotherapeutic interventions, 25 29 30 31 32 which made it difficult to distinguish psilocybin’s treatment effects. Most systematic reviews and meta-analyses did not consider the impact of factors that could act as moderators to psilocybin’s effects, such as type of depression (primary or secondary), previous use of psychedelics, psilocybin dosage, type of outcome measure (clinician rated or self-reported), and personal characteristics (eg, age, sex). 25 26 29 30 31 32 Lastly, systematic reviews did not consider grey literature, 33 34 which might have led to a substantial overestimation of psilocybin’s efficacy as a treatment for depression. In this review we focused on randomised trials that contained an unconfounded evaluation of psilocybin in adults with symptoms of depression, regardless of country and language of publication.

In this systematic review and meta-analysis of indexed and non-indexed randomised trials we investigated the efficacy of psilocybin to treat symptoms of depression compared with placebo or non-psychoactive drugs. The protocol was registered in the International Prospective Register of Systematic Reviews (see supplementary Appendix A). The study overall did not deviate from the pre-registered protocol; one clarification was made to highlight that any non-psychedelic comparator was eligible for inclusion, including placebo, niacin, micro doses of psychedelics, and drugs that are considered the standard of care in depression (eg, SSRIs).

Inclusion and exclusion criteria

Double blind and open label randomised trials with a crossover or parallel design were eligible for inclusion. We considered only studies in humans and with a control condition, which could include any type of non -active comparator, such as placebo, niacin, or micro doses of psychedelics.

Eligible studies were those that included adults (≥18 years) with clinically significant symptoms of depression, evaluated using a clinically validated tool for depression and mood disorder outcomes. Such tools included the Beck depression inventory, Hamilton depression rating scale, Montgomery-Åsberg depression rating scale, profile of mood states, and quick inventory of depressive symptomatology. Studies of participants with symptoms of depression and comorbidities (eg, cancer) were also eligible. We excluded studies of healthy participants (without depressive symptomatology).

Eligible studies investigated the effect of psilocybin as a standalone treatment on symptoms of depression. Studies with an active psilocybin condition that involved micro dosing (ie, psilocybin <100 μg/kg, according to the commonly accepted convention 22 35 ) were excluded. We included studies with directive psychotherapy if the psychotherapeutic component was present in both the experimental and the control conditions, so that the effects of psilocybin could be distinguished from those of psychotherapy. Studies involving group therapy were also excluded. Any non-psychedelic comparator was eligible for inclusion, including placebo, niacin, and micro doses of psychedelics.

Changes in symptoms, measured by validated clinician rated or self-report scales, such as the Beck depression inventory, Hamilton depression rating scale, Montgomery-Åsberg depression rating scale, profile of mood states, and quick inventory of depressive symptomatology were considered. We excluded outcomes that were measured less than three hours after psilocybin had been administered because any reported changes could be attributed to the transient cognitive and affective effects of the substance being administered. Aside from this, outcomes were included irrespective of the time point at which measurements were taken.

Search strategy

We searched major electronic databases and trial registries of psychological and medical research, with no limits on the publication date. Databases were the Cochrane Central Register of Controlled Trials via the Cochrane Library, Embase via Ovid, Medline via Ovid, Science Citation Index and Conference Proceedings Citation Index-Science via Web of Science, and PsycInfo via Ovid. A search through multiple databases was necessary because each database includes unique journals. Supplementary Appendix B shows the search syntax used for the Cochrane Central Register of Controlled Trials, which was slightly modified to comply with the syntactic rules of the other databases.

Unpublished and grey literature were sought through registries of past and ongoing trials, databases of conference proceedings, government reports, theses, dissertations, and grant registries (eg, ClinicalTrials.gov, WHO International Clinical Trials Registry Platform, ProQuest Dissertations and Theses Global, and PsycEXTRA). The references and bibliographies of eligible studies were checked for relevant publications. The original search was done in January 2023 and updated search was performed on 10 August 2023.

Data collection, extraction, and management

The results of the literature search were imported to the Endnote X9 reference management software, and the references were imported to the Covidence platform after removal of duplicates. Two reviewers (AM and DT) independently screened the title and abstract of each reference and then screened the full text of potentially eligible references. Any disagreements about eligibility were resolved through discussion. If information was insufficient to determine eligibility, the study’s authors were contacted. The reviewers were not blinded to the studies’ authors, institutions, or journal of publication.

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram shows the study selection process and reasons for excluding studies that were considered eligible for full text screening. 36

Critical appraisal of individual studies and of aggregated evidence

The methodological quality of eligible studies was assessed using the Cochrane Risk of Bias 2 tool (RoB 2) for assessing risk of bias in randomised trials. 37 In addition to the criteria specified by RoB 2, we considered the potential impact of industry funding and conflicts of interest. The overall methodological quality of the aggregated evidence was evaluated using GRADE (Grading of Recommendations, Assessment, Development and Evaluation). 38

If we found evidence of heterogeneity among the trials, then small study biases, such as publication bias, were assessed using a funnel plot and asymmetry tests (eg, Egger’s test). 39

We used a template for data extraction (see supplementary Appendix C) and summarised the extracted data in tabular form, outlining personal characteristics (age, sex, previous use of psychedelics), methodology (study design, dosage), and outcome related characteristics (mean change from baseline score on a depression questionnaire, response rates, and remission rates) of the included studies. Response conventionally refers to a 50% decrease in symptom severity based on scores on a depression rating scale, whereas remission scores are specific to a questionnaire (eg, score of ≤5 on the quick inventory of depressive symptomatology, score of ≤10 on the Montgomery-Åsberg depression rating scale, 50% or greater reduction in symptoms, score of ≤7 on the Hamilton depression rating scale, or score of ≤12 on the Beck depression inventory). Across depression scales, higher scores signify more severe symptoms of depression.

Continuous data synthesis

From each study we extracted the baseline and post-intervention means and standard deviations (SDs) of the scores between comparison groups for the depression questionnaires and calculated the mean differences and SDs of change. If means and SDs were not available for the included studies, we extracted the values from available graphs and charts using the Web Plot Digitizer application ( https://automeris.io/WebPlotDigitizer/ ). If it was not possible to calculate SDs from the graphs or charts, we generated values by converting standard errors (SEs) or confidence intervals (CIs), depending on availability, using formulas in the Cochrane Handbook (section 7.7.3.2). 40

Standardised mean differences were calculated for each study. We chose these rather than weighted mean differences because, although all the studies measured depression as the primary outcome, they did so with different questionnaires that score depression based on slightly different items. 41 If we had used weighted mean differences, any variability among studies would be assumed to reflect actual methodological or population differences and not differences in how the outcome was measured, which could be misleading. 40

The Hedges’ g effect size estimate was used because it tends to produce less biased results for studies with smaller samples (<20 participants) and when sample sizes differ substantially between studies, in contrast with Cohen’s d. 42 According to the Cochrane Handbook, the Hedges’ g effect size measure is synonymous with the standardised mean difference, 40 and the terms may be used interchangeably. Thus, a Hedges’ g of 0.2, 0.5, 0.8, or 1.2 corresponds to a small, medium, large, or very large effect, respectively. 40

Owing to variation in the participants’ personal characteristics, psilocybin dosage, type of depression investigated (primary or secondary), and type of comparators, we used a random effects model with a Hartung-Knapp-Sidik-Jonkman modification. 43 This model also allowed for heterogeneity and within study variability to be incorporated into the weighting of the results of the included studies. 44 Lastly, this model could help to generalise the findings beyond the studies and patient populations included, making the meta-analysis more clinically useful. 45 We chose the Hartung-Knapp-Sidik-Jonkman adjustment in favour of more widely used random effects models (eg, DerSimonian and Laird) because it allows for better control of type 1 errors, especially for studies with smaller samples, and provides a better estimation of between study variance by accounting for small sample sizes. 46 47

For studies in which multiple treatment groups were compared with a single placebo group, we split the placebo group to avoid multiplicity. 48 Similarly, if studies included multiple primary outcomes (eg, change in depression at three weeks and at six weeks), we split the treatment groups to account for overlapping participants. 40

Prediction intervals (PIs) were calculated and reported to show the expected effect range of a similar future study, in a different setting. In a random effects model, within study measures of variability, such as CIs, can only show the range in which the average effect size could lie, but they are not informative about the range of potential treatment effects given the heterogeneity between studies. 49 Thus, we used PIs as an indication of variation between studies.

Heterogeneity and sensitivity analysis

Statistical heterogeneity was tested using the χ 2 test (significance level P<0.1) and I 2 statistic, and heterogeneity among included studies was evaluated visually and displayed graphically using a forest plot. If substantial or considerable heterogeneity was found (I 2 ≥50% or P<0.1), 50 we considered the study design and characteristics of the included studies. Sources of heterogeneity were explored by subgroup analysis, and the potential effects on the results are discussed.

Planned sensitivity analyses to assess the effect of unpublished studies and studies at high risk of bias were not done because all included studies had been published and none were assessed as high risk of bias. Exclusion sensitivity plots were used to display graphically the impact of individual studies and to determine which studies had a particularly large influence on the results of the meta-analysis. All sensitivity analyses were carried out with Stata 16 software.

Subgroup analysis

To reduce the risk of errors caused by multiplicity and to avoid data fishing, we planned subgroup analyses a priori and limited to: (1) patient characteristics, including age and sex; (2) comorbidities, such as a serious physical condition (previous research indicates that the effects of psilocybin may be less strong for such participants, compared with participants with no comorbidities) 33 ; (3) number of doses and amount of psilocybin administered, because some previous meta-analyses found that a higher number of doses and a higher dose of psilocybin both predicted a greater reduction in symptoms of depression, 34 whereas others reported the opposite 33 ; (4) psilocybin administered alongside psychotherapeutic guidance or as a standalone treatment; (5) severity of depressive symptoms (clinical v subclinical symptomatology); (6) clinician versus patient rated scales; and (7) high versus low quality studies, as determined by RoB 2 assessment scores.

Metaregression

Given that enough studies were identified (≥10 distinct observations according to the Cochrane Handbook’s suggestion 40 ), we performed metaregression to investigate whether covariates, or potential effect modifiers, explained any of the statistical heterogeneity. The metaregression analysis was carried out using Stata 16 software.

Random effects metaregression analyses were used to determine whether continuous variables such as participants’ age, percentage of female participants, and percentage of participants who had previously used psychedelics modified the effect estimate, all of which have been implicated in differentially affecting the efficacy of psychedelics in modifying mood. 51 We chose this approach in favour of converting these continuous variables into categorical variables and conducting subgroup analyses for two primary reasons; firstly, the loss of any data and subsequent loss of statistical power would increase the risk of spurious significant associations, 51 and, secondly, no cut-offs have been agreed for these factors in literature on psychedelic interventions for mood disorders, 52 making any such divisions arbitrary and difficult to reconcile with the findings of other studies. The analyses were based on within study averages, in the absence of individual data points for each participant, with the potential for the results to be affected by aggregate bias, compromising their validity and generalisability. 53 Furthermore, a group level analysis may not be able to detect distinct interactions between the effect modifiers and participant subgroups, resulting in ecological bias. 54 As a result, this analysis should be considered exploratory.

Sensitivity analysis

A sensitivity analysis was performed to determine if choice of analysis method affected the primary findings of meta-analysis. Specifically, we reanalysed the data on change in depression score using a random effects Dersimonian and Laird model without the Hartung-Knapp-Sidik-Jonkman modification and compared the results with those of the originally used model. This comparison is particularly important in the presence of substantial heterogeneity and the potential of small study effects to influence the intervention effect estimate. 55

Patient and public involvement

Research on novel depression treatments is of great interest to both patients and the public. Although patients and members of the public were not directly involved in the planning or writing of this manuscript owing to a lack of available funding for recruitment and researcher training, patients and members of the public read the manuscript after submission.

Figure 1 presents the flow of studies through the systematic review and meta-analysis. 56 A total of 4884 titles were retrieved from the five databases of published literature, and a further 368 titles were identified from the databases of unpublished and international literature in February 2023. After the removal of duplicate records, we screened the abstracts and titles of 875 reports. A further 12 studies were added after handsearching of reference lists and conference proceedings and abstracts. Overall, nine studies totalling 436 participants were eligible. The average age of the participants ranged from 36-60 years. During an updated search on 10 August 2023, no further studies were identified.

Flow of studies in systematic review and meta-analysis

After screening of the title and abstract, 61 titles remained for full text review. Native speakers helped to translate papers in languages other than English. The most common reasons for exclusion were the inclusion of healthy volunteers, absence of control groups, and use of a survey based design rather than an experimental design. After full text screening, nine studies were eligible for inclusion, and 15 clinical trials prospectively registered or underway as of August 2023 were noted for potential future inclusion in an update of this review (see supplementary Appendix D).

We sent requests for further information to the authors of studies by Griffiths et al, 57 Barrett, 58 and Benville et al, 59 because these studies appeared to meet the inclusion criteria but were only provided as summary abstracts online. A potentially eligible poster presentation from the 58th annual meeting of the American College of Neuropsychopharmacology was identified but the lead author (Griffiths) clarified that all information from the presentation was included in the studies by Davis et al 13 and Gukasyan et al 60 ; both of which we had already deemed ineligible.

Barrett 58 reported the effects of psilocybin on the cognitive flexibility and verbal reasoning of a subset of patients with major depressive disorder from Griffith et al’s trial, 61 compared with a waitlist group, but when contacted, Barrett explained that the results were published in the study by Doss et al, 62 which we had already screened and judged ineligible (see supplementary Appendix E). Benville et al’s study 59 presented a follow-up of Ross et al’s study 17 on a subset of patients with cancer and high suicidal ideation and desire for hastened death at baseline. Measures of antidepressant effects of psilocybin treatment compared with niacin were taken before and after treatment crossover, but detailed results are not reported. Table 1 describes the characteristics of the included studies and table 2 lists the main findings of the studies.

Characteristics of included studies

• View inline

Main findings of included studies

Side effects and adverse events

Side effects reported in the included studies were minor and transient (eg, short term increases in blood pressure, headache, and anxiety), and none were coded as serious. Cahart-Harris et al noted one instance of abnormal dreams and insomnia. 63 This side effect profile is consistent with findings from other meta-analyses. 30 68 Owing to the different scales and methods used to catalogue side effects and adverse events across trials, it was not possible to combine these data quantitatively (see supplementary Appendix F).

Risk of bias

The Cochrane RoB 2 tools were used to evaluate the included studies ( table 3 ). RoB 2 for randomised trials was used for the five reports of parallel randomised trials (Carhart-Harris et al 63 and its secondary analysis Barba et al, 64 Goodwin et al 18 and its secondary analysis Goodwin et al, 65 and von Rotz et al 66 ) and RoB 2 for crossover trials was used for the four reports of crossover randomised trials (Griffiths et al, 14 Grob et al, 15 and Ross et al 17 and its follow-up Ross et al 67 ). Supplementary Appendix G provides a detailed explanation of the assessment of the included studies.

Summary risk of bias assessment of included studies, based on domains in Cochrane Risk of Bias 2 tool

Quality of included studies

Confidence in the quality of the evidence for the meta-analysis was assessed using GRADE, 38 through the GRADEpro GDT software program. Figure 2 shows the results of this assessment, along with our summary of findings.

GRADE assessment outputs for outcomes investigated in meta-analysis (change in depression scores and response and remission rates). The risk in the intervention group (and its 95% CI) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). BDI=Beck depression inventory; CI=confidence interval; GRADE=Grading of Recommendations, Assessment, Development and Evaluation; HADS-D=hospital anxiety and depression scale; HAM-D=Hamilton depression rating scale; MADRS=Montgomery-Åsberg depression rating scale; QIDS=quick inventory of depressive symptomatology; RCT=randomised controlled trial; SD=standard deviation

Meta-analyses

Continuous data, change in depression scores —Using a Hartung-Knapp-Sidik-Jonkman modified random effects meta-analysis, change in depression scores was significantly greater after treatment with psilocybin compared with active placebo. The overall Hedges’ g (1.64, 95% CI 0.55 to 2.73) indicated a large effect size favouring psilocybin ( fig 3 ). PIs were, however, wide and crossed the line of no difference (95% CI −1.72 to 5.03), indicating that there could be settings or populations in which psilocybin intervention would be less efficacious.

Forest plot for overall change in depression scores from before to after treatment. CI=confidence interval; DL=DerSimonian and Laird; HKSJ=Hartung-Knapp-Sidik-Jonkman

Exploring publication bias in continuous data —We used Egger’s test and a funnel plot to examine the possibility of small study biases, such as publication bias. Statistical significance of Egger’s test for small study effects, along with the asymmetry in the funnel plot ( fig 4 ), indicates the presence of bias against smaller studies with non-significant results, suggesting that the pooled intervention effect estimate is likely to be overestimated. 69 An alternative explanation, however, is that smaller studies conducted at the early stages of a new psychotherapeutic intervention tend to include more high risk or responsive participants, and psychotherapeutic interventions tend to be delivered more effectively in smaller trials; both of these factors can exaggerate treatment effects, resulting in funnel plot asymmetry. 70 Also, because of the relatively small number of included studies and the considerable heterogeneity observed, test power may be insufficient to distinguish real asymmetry from chance. 71 Thus, this analysis should be considered exploratory.

Funnel plot assessing publication bias among studies measuring change in depression scores from before to after treatment. CI=confidence interval; θ IV =estimated effect size under inverse variance random effects model

Dichotomous data

We extracted response and remission rates for each group when reported directly, or imputed information when presented graphically. Two studies did not measure response or remission and thus did not contribute data for this part of the analysis. 15 18 The random effects model with a Hartung-Knapp-Sidik-Jonkman modification was used to allow for heterogeneity to be incorporated into the weighting of the included studies’ results, and to provide a better estimation of between study variance accounting for small sample sizes.

Response rate —Overall, the likelihood of psilocybin intervention leading to treatment response was about two times greater (risk ratio 2.02, 95% CI 1.33 to 3.07) than with placebo. Despite the use of different scales to measure response, the heterogeneity between studies was not significant (I 2 =25.7%, P=0.23). PIs were, however, wide and crossed the line of no difference (−0.94 to 3.88), indicating that there could be settings or populations in which psilocybin intervention would be less efficacious.

Remission rate —Overall, the likelihood of psilocybin intervention leading to remission of depression was nearly three times greater than with placebo (risk ratio 2.71, 95% CI 1.75 to 4.20). Despite the use of different scales to measure response, no statistical heterogeneity was found between studies (I 2 =0.0%, P=0.53). PIs were, however, wide and crossed the line of no difference (0.87 to 2.32), indicating that there could be settings or populations in which psilocybin intervention would be less efficacious.

Exploring publication bias in response and remission rates data —We used Egger’s test and a funnel plot to examine whether response and remission estimates were affected by small study biases. The result for Egger’s test was non-significant (P>0.05) for both response and remission estimates, and no substantial asymmetry was observed in the funnel plots, providing no indication for the presence of bias against smaller studies with non-significant results.

Heterogeneity: subgroup analyses and metaregression

Heterogeneity was considerable across studies exploring changes in depression scores (I 2 =89.7%, P<0.005), triggering subgroup analyses to explore contributory factors. Table 4 and table 5 present the results of the heterogeneity analyses (subgroup analyses and metaregression, respectively). Also see supplementary Appendix H for a more detailed description and graphical representation of these results.

Subgroup analyses to explore potential causes of heterogeneity among included studies

Metaregression analyses to explore potential causes of heterogeneity among included studies

Cumulative meta-analyses

We used cumulative meta-analyses to investigate how the overall estimates of the outcomes of interest changed as each study was added in chronological order 72 ; change in depression scores and likelihood of treatment response both increased as the percentage of participants with past use of psychedelics increased across studies, as expected based on the metaregression analysis (see supplementary Appendix I). No other significant time related patterns were found.

We reanalysed the data for change in depression scores using a random effects Dersimonian and Laird model without the Hartung-Knapp-Sidik-Jonkman modification and compared the results with those of the original model. All comparisons found to be significant using the Dersimonian and Laird model with the Hartung-Knapp-Sidik-Jonkman adjustment were also significant without the Hartung-Knapp-Sidik-Jonkman adjustment, and confidence intervals were only slightly narrower. Thus, small study effects do not appear to have played a major role in the treatment effect estimate.

Additionally, to estimate the accuracy and robustness of the estimated treatment effect, we excluded studies from the meta-analysis one by one; no important differences in the treatment effect, significance, and heterogeneity levels were observed after the exclusion of any study (see supplementary Appendix J).

In our meta-analysis we found that psilocybin use showed a significant benefit on change in depression scores compared with placebo. This is consistent with other recent meta-analyses and trials of psilocybin as a standalone treatment for depression 73 74 or in combination with psychological support. 24 25 29 30 31 32 68 75 This review adds to those finding by exploring the considerable heterogeneity across the studies, with subsequent subgroup analyses showing that the type of depression (primary or secondary) and the depression scale used (Montgomery-Åsberg depression rating scale, quick inventory of depressive symptomatology, or Beck depression inventory) had a significant differential effect on the outcome. High between study heterogeneity has been identified by some other meta-analyses of psilocybin (eg, Goldberg et al 29 ), with a higher treatment effect in studies with patients with comorbid life threatening conditions compared with patients with primary depression. 22 Although possible explanations, including personal factors (eg, patients with life threatening conditions being older) or depression related factors (eg, secondary depression being more severe than primary depression) could be considered, these hypotheses are not supported by baseline data (ie, patients with secondary depression do not differ substantially in age or symptom severity from patients with primary depression). The differential effects from assessment scales used have not been examined in other meta-analyses of psilocybin, but this review’s finding that studies using the Beck depression inventory showed a higher treatment effect than those using the Montgomery-Åsberg depression rating scale and quick inventory of depressive symptomatology is consistent with studies in the psychological literature that have shown larger treatment effects when self-report scales are used (eg, Beck depression inventory). 76 77 This finding may be because clinicians tend to overestimate the severity of depression symptoms at baseline assessments, leading to less pronounced differences between before and after treatment identified in clinician assessed scales (eg, Montgomery-Åsberg depression rating scale, quick inventory of depressive symptomatology). 78

Metaregression analyses further showed that a higher average age and a higher percentage of participants with past use of psychedelics both correlated with a greater improvement in depression scores with psilocybin use and explained a substantial amount of between study variability. However, the cumulative meta-analysis showed that the effects of age might be largely an artefact of the inclusion of one specific study, and alternative explanations are worth considering. For instance, Studerus et al 79 identified participants’ age as the only personal variable significantly associated with psilocybin response, with older participants reporting a higher “blissful state” experience. This might be because of older people’s increased experience in managing negative emotions and the decrease in 5-hydroxytryptamine type 2A receptor density associated with older age. 80 Furthermore, Rootman et al 81 reported that the cognitive performance of older participants (>55 years) improved significantly more than that of younger participants after micro dosing with psilocybin. Therefore, the higher decrease in depressive symptoms associated with older age could be attributed to a decrease in cognitive difficulties experienced by older participants.

Interestingly, a clear pattern emerged for past use of psychedelics—the higher the proportion of study participants who had used psychedelics in the past, the higher the post-psilocybin treatment effect observed. Past use of psychedelics has been proposed to create an expectancy bias among participants and amplify the positive effects of psilocybin 82 83 84 ; however, this important finding has not been examined in other meta-analyses and may highlight the role of expectancy in psilocybin research.

Limitations of this study

Generalisability of the findings of this meta-analysis was limited by the lack of racial and ethnic diversity in the included studies—more than 90% of participants were white across all included trials, resulting in a homogeneous sample that is not representative of the general population. Moreover, it was not possible to distinguish between subgroups of participants who had never used psilocybin and those who had taken psilocybin more than a year before the start of the trial, as these data were not provided in the included studies. Such a distinction would be important, as the effects of psilocybin on mood may wane within a year after being administered. 21 85 Also, how psychological support was conceptualised was inconsistent within studies of psilocybin interventions; many studies failed to clearly describe the type of psychological support participants received, and others used methods ranging from directive guidance throughout the treatment session to passive encouragement or reassurance (eg, Griffiths et al, 14 Carhart-Harris et al 63 ). The included studies also did not gather evidence on participants’ previous experiences with treatment approaches, which could influence their response to the trials’ intervention. Thus, differences between participant subgroups related to past use of psilocybin or psychotherapy may be substantial and could help interpret this study’s findings more accurately. Lastly, the use of graphical extraction software to estimate the findings of studies where exact numerical data were not available (eg, Goodwin et al, 18 Grob et al 15 ), may have affected the robustness of the analyses.

A common limitation in studies of psilocybin is the likelihood of expectancy effects augmenting the treatment effect observed. Although some studies used low dose psychedelics as comparators to deal with this problem (eg, Carhart-Harris et al, 63 Goodwin et al, 18 Griffiths et al 14 ) or used a niacin placebo that can induce effects similar to those of psilocybin (eg, Grob et al, 15 Ross et al 17 ), the extent to which these methods were effective in blinding participants is not known. Other studies have, however, reported that participants can accurately identify the study groups to which they had been assigned 70-85% of the time, 84 86 indicating a high likelihood of insufficient blinding. This is especially likely for studies in which a high proportion of participants had previously used psilocybin and other hallucinogens, making the identification of the drug’s acute effects easier (eg, Griffiths et al, 14 Grob et al, 15 Ross et al 17 ). Patients also have expectations related to the outcome of their treatment, expecting psilocybin to improve their symptoms of depression, and these positive expectancies are strong predictors of actual treatment effects. 87 88 Importantly, the effect of outcome expectations on treatment effect is particularly strong when patient reported measures are used as primary outcomes, 89 which was the case in several of the included studies (eg, Griffiths et al, 14 Grob et al, 15 Ross et al 17 ). Unfortunately, none of the included studies recorded expectations before treatment, so it is not possible to determine the extent to which this factor affected the findings.

Implications for clinical practice

Although this review’s findings are encouraging for psilocybin’s potential as an effective antidepressant, a few areas about its applicability in clinical practice remain unexplored. Firstly, it is unclear whether the protocols for psilocybin interventions in clinical trials can be reliably and safely implemented in clinical practice. In clinical trials, patients receive psilocybin in a non-traditional medical setting, such as a specially designed living room, while they may be listening to curated calming music and are isolated from most external stimuli by wearing eyeshades and external noise-cancelling earphones. A trained therapist closely supervises these sessions, and the patient usually receives one or more preparatory sessions before the treatment commences. Standardising an intervention setting with so many variables is unlikely to be achievable in routine practice, and consensus is considerably lacking on the psychotherapeutic training and accreditations needed for a therapist to deliver such treatment. 90 The combination of these elements makes this a relatively complex and expensive intervention, which could make it challenging to gain approval from regulatory agencies and to gain reimbursement from insurance companies and others. Within publicly funded healthcare systems, the high cost of treatment may make psilocybin treatment inaccessible. The high cost associated with the intervention also increases the risk that unregulated clinics may attempt to cut costs by making alterations to the protocol and the therapeutic process, 91 92 which could have detrimental effects for patients. 92 93 94 Thus, avoiding the conflation of medical and commercial interests is a primary concern that needs to be dealt with before psilocybin enters mainstream practice.

Implications for future research

More large scale randomised trials with long follow-up are needed to fully understand psilocybin’s treatment potential, and future studies should aim to recruit a more diverse population. Another factor that would make clinical trials more representative of routine practice would be to recruit patients who are currently using or have used commonly prescribed serotonergic antidepressants. Clinical trials tend to exclude such participants because many antidepressants that act on the serotonin system modulate the 5-hydroxytryptamine type 2A receptor that psilocybin primarily acts upon, with prolonged use of tricyclic antidepressants associated with more intense psychedelic experiences and use of monoamine oxidase inhibitors or SSRIs inducing weaker responses to psychedelics. 95 96 97 Investigating psilocybin in such patients would, however, provide valuable insight on how psilocybin interacts with commonly prescribed drugs for depression and would help inform clinical practice.

Minimising the influence of expectancy effects is another core problem for future studies. One strategy would be to include expectancy measures and explore the level of expectancy as a covariate in statistical analysis. Researchers should also test the effectiveness of condition masking. Another proposed solution would be to adopt a 2×2 balanced placebo design, where both the drug (psilocybin or placebo) and the instructions given to participants (told they have received psilocybin or told they have received placebo) are crossed. 98 Alternatively, clinical trials could adopt a three arm design that includes both an inactive placebo (eg, saline) and active placebo (eg, niacin, lower psylocibin dose), 98 allowing for the effects of psilocybin to be separated from those of the placebo.

Overall, future studies should explore psilocybin’s exact mechanism of treatment effectiveness and outline how its physiological effects, mystical experiences, dosage, treatment setting, psychological support, and relationship with the therapist all interact to produce a synergistic antidepressant effect. Although this may be difficult to achieve using an explanatory randomised trial design, pragmatic clinical trial designs may be better suited to psilocybin research, as their primary objective is to achieve high external validity and generalisability. Such studies may include multiple alternative treatments rather than simply an active and placebo treatment comparison (eg, psilocybin v SSRI v serotonin-noradrenaline reuptake inhibitor), and participants would be recruited from broader clinical populations. 99 100 Although such studies are usually conducted after a drug’s launch, 100 earlier use of such designs could help assess the clinical effectiveness of psilocybin more robustly and broaden patient access to a novel type of antidepressant treatment.

Conclusions

This review’s findings on psilocybin’s efficacy in reducing symptoms of depression are encouraging for its use in clinical practice as a drug intervention for patients with primary or secondary depression, particularly when combined with psychological support and administered in a supervised clinical environment. However, the highly standardised treatment setting, high cost, and lack of regulatory guidelines and legal safeguards associated with psilocybin treatment need to be dealt with before it can be established in clinical practice.

What is already known on this topic

Recent research on treatments for depression has focused on psychedelic agents that could have strong antidepressant effects without the drawbacks of classic antidepressants; psilocybin being one such substance

Over the past decade, several clinical trials, meta-analyses, and systematic reviews have investigated the use of psilocybin for symptoms of depression, and most have found that psilocybin can have antidepressant effects

Studies published to date have not investigated factors that may moderate psilocybin’s effects, including type of depression, past use of psychedelics, dosage, outcome measures, and publication biases

What this study adds

This review showed a significantly greater efficacy of psilocybin among patients with secondary depression, patients with past use of psychedelics, older patients, and studies using self-report measures for symptoms of depression

Efficacy did not appear to be homogeneous across patient types—for example, those with depression and a life threatening illness appeared to benefit more from treatment

Further research is needed to clarify the factors that maximise psilocybin’s treatment potential for symptoms of depression

Ethics statements

Ethical approval.

This study was approved by the ethics committee of the University of Oxford Nuffield Department of Medicine, which waived the need for ethical approval and the need to obtain consent for the collection, analysis, and publication of the retrospectively obtained anonymised data for this non-interventional study.

Data availability statement

The relevant aggregated data and statistical code will be made available on reasonable request to the corresponding author.

Acknowledgments

We thank DT who acted as an independent secondary reviewer during the study selection and data review process.

Contributors: AMM contributed to the design and implementation of the research, analysis of the results, and writing of the manuscript. MC was involved in planning and supervising the work and contributed to the writing of the manuscript. AMM and MC are the guarantors. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: None received.

Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; AMM is employed by IDEA Pharma, which does consultancy work for pharmaceutical companies developing drugs for physical and mental health conditions; MC was the supervisor for AMM’s University of Oxford MSc dissertation, which forms the basis for this paper; no other relationships or activities that could appear to have influenced the submitted work.

Transparency: The corresponding author (AMM) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as registered have been explained.

Dissemination to participants and related patient and public communities: To disseminate our findings and increase the impact of our research, we plan on writing several social media posts and blog posts outlining the main conclusions of our paper. These will include blog posts on the websites of the University of Oxford’s Department of Primary Care Health Sciences and Department for Continuing Education, as well as print publications, which are likely to reach a wider audience. Furthermore, we plan to present our findings and discuss them with the public in local mental health related events and conferences, which are routinely attended by patient groups and advocacy organisations.

Provenance and peer review: Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• ↵ World Health Organization. Depressive Disorder (Depression); 2023. https://www.who.int/news-room/fact-sheets/detail/depression .
• GBD 2017 Disease and Injury Incidence and Prevalence Collaborators
• Cipriani A ,
• Furukawa TA ,
• Salanti G ,
• Trivedi MH ,
• Wisniewski SR ,
• Mitchell AJ
• Bockting CL ,
• Hollon SD ,
• Jarrett RB ,
• Nierenberg AA ,
• Petersen TJ ,
• Páleníček T ,
• Carbonaro TM ,
• Bradstreet MP ,
• Barrett FS ,
• Carhart-Harris RL ,
• Bolstridge M ,
• Griffiths RR ,
• Johnson MW ,
• Carducci MA ,
• Danforth AL ,
• Chopra GS ,
• Kraehenmann R ,
• Preller KH ,
• Scheidegger M ,
• Goodwin GM ,
• Aaronson ST ,
• Alvarez O ,
• Bogenschutz MP ,
• Podrebarac SK ,
• Roseman L ,
• Galvão-Coelho NL ,
• Gonzalez M ,
• Dos Santos RG ,
• Osório FL ,
• Crippa JA ,
• Zuardi AW ,
• Cleare AJ ,
• Martelli C ,
• Benyamina A
• Vollenweider FX ,
• Demetriou L ,
• Carhart-Harris RL
• Timmermann C ,
• Giribaldi B ,
• Goldberg SB ,
• Nicholas CR ,
• Raison CL ,
• Irizarry R ,
• Winczura A ,
• Dimassi O ,
• Dhillon N ,
• Griffiths RR
• Castro Santos H ,
• Gama Marques J
• Moreno FA ,
• Wiegand CB ,
• Taitano EK ,
• Liberati A ,
• Tetzlaff J ,
• Altman DG ,
• PRISMA Group
• Sterne JAC ,
• Savović J ,
• Guyatt GH ,
• Schünemann HJ ,
• Tugwell P ,
• Knottnerus A
• Sterne JA ,
• Sutton AJ ,
• Ioannidis JP ,
• Higgins JPT ,
• Chandler J ,
• Borenstein M ,
• Hedges LV ,
• Higgins JP ,
• Rothstein HR
• DerSimonian R ,
• ↵ Borenstein M, Hedges L, Rothstein H. Meta-analysis: Fixed effect vs. random effects. Meta-analysis. com. 2007;1-62.
• IntHout J ,
• Rovers MM ,
• Gøtzsche PC
• Spineli LM ,
• ↵ Higgins JP, Green S. Identifying and measuring heterogeneity. Cochrane handbook for systematic reviews of interventions. 2011;5(0).
• Austin PC ,
• O’Donnell KC ,
• Mennenga SE ,
• Bogenschutz MP
• Sander SD ,
• Berlin JA ,
• Santanna J ,
• Schmid CH ,
• Szczech LA ,
• Feldman HI ,
• Anti-Lymphocyte Antibody Induction Therapy Study Group
• ↵ Iyengar S, Greenhouse J. Sensitivity analysis and diagnostics. Handbook of research synthesis and meta-analysis. Russell Sage Foundation, 2009:417-33.
• McKenzie JE ,
• Bossuyt PM ,
• ↵ Griffiths R, Barrett F, Johnson M, Mary C, Patrick F, Alan D. Psilocybin-Assisted Treatment of Major Depressive Disorder: Results From a Randomized Trial. Proceedings of the ACNP 58th Annual Meeting: Poster Session II. In Neuropsychopharmacology. 2019;44:230-384.
• ↵ Barrett F. ACNP 58th Annual Meeting: Panels, Mini-Panels and Study Groups. [Abstract.] Neuropsychopharmacology 2019;44:1-77. doi: 10.1038/s41386-019-0544-z . OpenUrl CrossRef
• Benville J ,
• Agin-Liebes G ,
• Roberts DE ,
• Gukasyan N ,
• Hurwitz ES ,
• Považan M ,
• Rosenberg MD ,
• Carhart-Harris R ,
• Buehler S ,
• Kettner H ,
• von Rotz R ,
• Schindowski EM ,
• Jungwirth J ,
• Vargas AS ,
• Barroso M ,
• Gallardo E ,
• Isojarvi J ,
• Lefebvre C ,
• Glanville J
• Sukpraprut-Braaten S ,
• Narlesky M ,
• Strayhan RC
• Prouzeau D ,
• Conejero I ,
• Voyvodic PL ,
• Becamel C ,
• Lopez-Castroman J
• Więckiewicz G ,
• Stokłosa I ,
• Gorczyca P ,
• John Mann J ,
• Currier D ,
• Zimmerman M ,
• Friedman M ,
• Boerescu DA ,
• Attiullah N
• Borgherini G ,
• Conforti D ,
• Studerus E ,
• Kometer M ,
• Vollenweider FX
• Pinborg LH ,
• Rootman JM ,
• Kryskow P ,
• Turner EH ,
• Rosenthal R
• Bershad AK ,
• Schepers ST ,
• Bremmer MP ,
• Sepeda ND ,
• Hurwitz E ,
• Horvath AO ,
• Del Re AC ,
• Flückiger C ,
• Rutherford BR ,
• Pearson C ,
• Husain SF ,
• Harris KM ,
• George JR ,
• Michaels TI ,
• Sevelius J ,
• Williams MT
• Collins A ,
• Bonson KR ,
• Buckholtz JW ,
• Yamauchi M ,
• Matsushima T ,
• Coleshill MJ ,
• Colloca L ,
• Zachariae R ,
• Colagiuri B
• Heifets BD ,
• Pratscher SD ,
• Bradley E ,
• Sugarman J ,

• Open access
• Published: 01 May 2024

Hospital performance evaluation indicators: a scoping review

• Shirin Alsadat Hadian   ORCID: orcid.org/0000-0002-1443-1990 1 ,
• Reza Rezayatmand   ORCID: orcid.org/0000-0002-9907-3597 2 ,
• Nasrin Shaarbafchizadeh   ORCID: orcid.org/0000-0001-7104-2214 3 ,
• Saeedeh Ketabi   ORCID: orcid.org/0000-0002-6778-5645 4 &
• Ahmad Reza Pourghaderi   ORCID: orcid.org/0000-0003-2682-2160 5  

BMC Health Services Research volume  24 , Article number:  561 ( 2024 ) Cite this article

264 Accesses

Metrics details

Hospitals are the biggest consumers of health system budgets and hence measuring hospital performance by quantitative or qualitative accessible and reliable indicators is crucial. This review aimed to categorize and present a set of indicators for evaluating overall hospital performance.

We conducted a literature search across three databases, i.e., PubMed, Scopus, and Web of Science, using possible keyword combinations. We included studies that explored hospital performance evaluation indicators from different dimensions.

We included 91 English language studies published in the past 10 years. In total, 1161 indicators were extracted from the included studies. We classified the extracted indicators into 3 categories, 14 subcategories, 21 performance dimensions, and 110 main indicators. Finally, we presented a comprehensive set of indicators with regard to different performance dimensions and classified them based on what they indicate in the production process, i.e., input, process, output, outcome and impact.

The findings provide a comprehensive set of indicators at different levels that can be used for hospital performance evaluation. Future studies can be conducted to validate and apply these indicators in different contexts. It seems that, depending on the specific conditions of each country, an appropriate set of indicators can be selected from this comprehensive list of indicators for use in the performance evaluation of hospitals in different settings.

Peer Review reports

Healthcare is complex [ 1 ] and a key sector [ 2 ] that is now globally faced with problems of rising costs, lack of service efficiency, competition, and equity as well as responsiveness to users [ 3 ]. One estimate by the WHO has shown a yearly waste of approximately 20–40% of total healthcare resources because of inefficiency [ 4 ]. European countries have spent on average 9.6% of their gross domestic product (GDP) on healthcare in 2017 and 9.92% in 2019. Germany, France, and Sweden reported the highest healthcare expenditures in Europe in 2018 (between 10.9% and 11.5% of GDP) [ 5 ]. In the U.S., healthcare spending consumes 18% of the GDP, which is likely to eclipse $6 trillion by 2027 [ 6 ].

Hospitals, as the biggest consumers of health system budgets [ 7 ], are the major part of the health system [ 8 ]. In many countries 50–80% of the health sector budget is dedicated to hospitals [ 8 , 9 ]. As a result, hospital performance analysis is becoming a routine task for every hospital manager. On the one hand, hospital managers worldwide are faced with difficult decisions regarding cost reduction, increasing service efficiency, and equity [ 10 ]. On the other hand, measuring hospital efficiency is an issue of interest among researchers because patients demand high-quality care at lower expenses [ 11 ].

To address the above mentioned need to measure hospital performance, implementing an appropriate hospital performance evaluation system is crucial in any hospital. In doing so, hospital administrators use various tools to analyse and monitor hospital activities [ 1 ], which need well-defined objectives, standards and quantitative indicators [ 12 ]. The latter are used to evaluate care provided to patients both quantitatively and qualitatively and are often related to input, output, processes, and outcomes. These indicators can be used for continuous quality improvement by monitoring, benchmarking, and prioritizing activities [ 13 ]. These parameters are developed to improve health outcomes and to provide comparative information for monitoring and managing and formulating policy objectives within and across health services [ 12 ]. Studies thus far have used their own set of indicators while evaluating hospital performance, which could be context dependent. In addition, those studies have mostly used a limited set of indicators that focus on few dimensions (2–6 dimensions) of hospital performance [ 14 , 15 , 16 , 17 , 18 ].

Therefore, comprehensive knowledge of potential indicators that can be used for hospital performance evaluation is necessary. It would help choose appropriate indicators when evaluating hospital performance in different contexts. It would also help researchers extend the range of analysis to evaluate performance from a wider perspective by considering more dimensions of performance. Although performance is a very commonly used term, it has several definitions [ 19 , 20 ], yet, it is often misunderstood [ 21 ]. Therefore, some researchers have expressed confusion about the related terms and considered them interchangeable. These terms are effectiveness, efficiency, productivity, quality, flexibility, creativity, sustainability, evaluation, and piloting [ 21 , 22 , 23 ]. Thus, this scoping review aimed to categorize and present a comprehensive set of indicators that can be used as a suitable set for hospital performance evaluation at any needed level of analysis, i.e., clinical, para-clinical, logistical, or departmental, and relate those indicators to the appropriate performance dimensions. The uniqueness of this paper is that it provides its readers with a comprehensive collection of indicators that have been used in different performance analysis studies.

Materials and methods

We conducted a scoping review of a body of literature. The scoping review can be of particular use when the topic has not yet been extensively reviewed or has a complex or heterogeneous nature. This type of review is commonly undertaken to examine the extent, range, and nature of research activity in a topic area; determine the value and potential scope and cost of undertaking a full systematic review; summarize and disseminate research findings; and identify research gaps in the existing literature. As a scoping review provides a rigorous and transparent method for mapping areas of research, it can be used as a standalone project or as a preliminary step to a systematic review [ 24 ]. While a systematic review (qualitative or quantitative) usually addresses a narrow topic/scope and is a method for integrating or comparing findings from previous studies [ 25 ].

In our study, we used the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist following the methods outlined by Arksey and O’Malley [ 26 ] and Tricco [ 27 ]. A systematic search for published and English-language literature on hospital performance evaluation models was conducted, using three databases, i.e., PubMed, Scopus, and Web of Science, from 2013 to January 2023. Initially, the identified keywords were refined and validated by a team of experts. Then, a combination of vocabularies was identified by the authors through a brainstorming process. The search strategy was formulated using Boolean operators. The title and abstract of the formulas were searched in the online databases. The search query for each database is presented in Table  1 .

In the screening process, relevant references related to hospital performance evaluation were screened and abstracted into researcher-developed Microsoft® Excel forms by dual independent reviewers and conflicting information was provided by other reviewers.

The inclusion criteria were as follows: focused only on the hospital setting, available full text and written in English. We excluded studies that focused on health organization indicators, not specifically on hospital indicators; articles without appropriate data (only focused on models and not indicators; or qualitative checklist questionnaires); and articles that focused only on clinical or disease-related indicators, not hospital performance dimensions, and provided very general items as indicators, not the domains of the indicators themselves. Then, a PRISMA-ScR Checklist was used to improve transparency in our review [ 28 ].

To extract the data, researcher-developed Microsoft® Excel forms (data tables) were designed. The following data were subsequently extracted into Microsoft®Excel for synthesis and evaluation: title, author, article year, country, indicator category, study environment (number of hospitals studied), study time frame, indicator name, number of indicators, indicator level (hospital level, department level), evaluation perspective (performance, productivity, efficiency, effectiveness, quality, cost, safety, satisfaction, etc. ) , study type (quantitative or qualitative), indicator subtype (input (structure), process, output (result), outcome and impact), and other explanations. To create a descriptive summary of the results that address the objectives of this scoping review, numerical summarization was also used.

The purpose of creating the main category and the evaluation perspective section was to develop them and create new categories, which focused on the type of indicators related to the performance term. For example, in the “Category” section, the names of the departments or wards of the hospital (such as hospital laboratories, pharmacies, clinical departments, and warehouses) and in the “Evaluation perspective” section, various terms related to the evaluation of hospital performance were extracted. These two types were used after extracting their information under the title “performance dimension”.

The indicators’ levels were collected to determine the level of performance evaluation with the relevant index. Some indicators were used to evaluate the performance of the entire hospital, some were used to evaluate the performance of hospital departments, and some were used to evaluate the performance at the level of a specific project. For example, several indicators (such as bed occupancy ratio, length of stay, and waiting time) were used to evaluate the performance of the entire hospital, and other indicators (such as laboratory department indicators, energy consumption indicators, and neonatal department indicators) were used only to measure the performance of specific departments. This sections were used under the title “category”. The “category” and “indicator’s name” sections were defined according to the results of the “subcategory” section.

The subtypes of indicators (input (structure), process, output(result), outcome and impact) were defined based on the chain model, and each of the selected indicators was linked to it (Appendix 1 ). As a result of the chain model, inputs were used to carry out activities, activities led to the delivery of services or products (outputs). The outputs started to bring about change (outcomes), and eventually, this (hopefully) contributed to the impact [ 29 ]. The classification of the set of input, process, output, outcome and impact indicators was such that readers could access these categories if necessary according to their chosen evaluation models. The term was used under the title “Indicators by types”.

The type of study was considered quantitative or qualitative for determining whether an indicator was able to perform calculations. In this way, readers can choose articles that use quantitative or qualitative indicators to evaluate hospital performance.

We included 91 full-text studies (out of 7475) in English published between 2013 and January 2023 (Fig.  1 ), approximately 40% of which were published between 2020 and 2023. More than 20% of the retrieved studies were conducted in Iran and USA.

Study selection and data abstraction

Study characteristic

As shown in Table  2 , in 85% of the reviewed studies, a number of hospitals (1 to 3828 hospitals, 13,221 hospitals in total) were evaluated. More than 90% of the studies used a quantitative approach. In more than 70% of the studies, hospital evaluation occurred at the department level, which can also be divided into three levels: administrative, clinical ward, and paramedical department. In addition, the administrative departments consist of 13 departments, including financial management [ 48 , 55 , 61 , 67 , 68 , 80 , 83 , 109 , 113 ], supply chain management and warehouse [ 15 , 43 , 84 ], value-based purchasing [ 33 , 85 ], human resource management [ 97 , 101 ], medical equipment [ 32 , 87 ], health information management department [ 90 ], information systems [ 106 ], nutritional assessment [ 93 ], energy management [ 30 , 45 , 92 ], facility management [ 52 , 53 ], building sustainability and resilience [ 35 ], research activities [ 44 ], and education [ 107 ].

The clinical wards consisted of 8 wards, namely, emergency departments (EDs) [ 16 , 39 , 56 , 57 , 69 , 70 , 89 ], surgery departments [ 58 , 62 , 63 , 91 , 102 ], intensive care units (ICUs) [ 47 , 64 , 65 ], operating rooms (ORs) [ 38 , 88 , 108 ], surgical intensive care units (SICUs) [ 111 ], obstetrics and gynecology department [ 59 ], neonatal intensive care units (NICUs) [ 74 , 103 ] and quality of care [ 18 , 31 , 40 , 50 , 72 , 92 , 95 , 112 ] indicators. The paramedical departments consisted of 3 departments, pharmacy [ 60 , 76 , 98 ], laboratory and blood bank [ 37 , 42 , 43 , 49 ], and outpatient assessment [ 86 ] indicators.

With regard to data categorization, firstly, a total of 1204 indicators in 91 studies were extracted and after detailed examination, 43 indices (such as hospital ownership, level of care, admission process, and personal discipline) were removed due to their generality and impossibility of calculation in the hospital environment. Then, 1161 performance indicators were entered in this research and were categorized based on the performance criteria (more details about the indicators can be found in Appendix 1 ). Secondly, 145 functional dimensions, including divisions based on different departments and units of the hospital, were defined according to several focus group discussions with 5 health experts. Then, re-categorization and functional summarization were performed, after which 21 performance dimensions were finalized.

As shown in Table  4 , the 21 performance dimensions were divided into three parts: category, subcategory, and related indicators. Additionally, according to the hospital levels, there were three categories: ‘organizational management’, ‘clinical management’, and ‘administrative management’. Then, according to the type of indicators, fifteen subcategories were defined for the 110 selected main indicators.

Performance dimensions

The ‘productivity’ dimension focuses on indicators reflecting the macro-performance of the hospital, considering that this index is more effective and efficient. The ‘efficiency’ dimension focuses on general performance indicators for the optimal use of resources to create optimal output in the hospital. The ‘effectiveness’ dimension is a general performance indicator with an outcome view. The ‘speed’ dimension focuses on the indicators that show attention to the service delivery time and the speed of the procedures. The ‘development’ dimension focuses on matters related to employees’ and students’ training and related training courses. In terms of ‘safety’ dimension, there were issues related to patient safety, unwanted and harmful events, and hospital infections.

The “quality of work life” dimension emphasizes matters related to personnel volume and work conditions. The ‘quality’ dimension is related to the quality of service provided in different parts of the hospital and possible complications in improving the quality of services. The ‘satisfaction’ dimension focuses on the satisfaction of patients, employees, and their complaints. The ‘innovation’ dimension relates to the research process and its output. The ‘appropriateness’ dimension involves proper service from clinical departments, pharmaceutical services, and patient treatment. The ‘evaluation’ dimension focuses on the indicators related to the assessment scores of the para-clinical departments of the hospital.

The ‘profitability’ dimension focuses on the overall output indicators for income and profitability. The ‘cost’ dimension focuses on indicators related to general expenditures and the average cost per bed and patient and budgeting. The ‘economy’ dimension is related to financial rates and their indicators. The ‘coherence’ dimension emphasizes the indicators related to the continuity of the service delivery process. The ‘patient-centeredness’ dimension focuses on the indicators related to the patient’s experience of the facility, environment, treatment processes, communications, and relevant support for the patient. The ‘equity’ dimension studies indicators related to social and financial justice and life expectancy. The ‘relationship’ dimension evaluates the process of consultations and discussions required during the patients’ care provided by the treatment team. The ‘sustainability’ dimension focuses on indicators related to energy standards. The ‘flexibility’ dimension focuses on the hospital’s response to the crisis.

According to Table  4 , most studies focused on ‘efficiency’, ‘productivity’, ‘safety’ and ‘effectiveness’ as performance dimensions in 54, 53, 38 and 37 studies, respectively (40–70% of studies). In the ‘efficiency’ subcategory, resource management, supportive unit assessment, and human resource management indicators were the first to third most common indicators used in 26, 23 and 22 studies, respectively (approximately 25% of the studies).

In addition, for the ‘efficiency’ dimension, ‘medical staff numbers’, ‘emergency department bed numbers’, and ‘nonmedical staff numbers’ were reported in 16, 13, and 11 studies, respectively (between 20 and 30% of the studies). For the ‘productivity’ subcategory, ‘bed utilization rate’ and ‘service delivery and treatment’ were reported in 50% and 20% of the studies, respectively (46 and 19 out of 91).

Additionally, for the ‘productivity’ dimension, the ‘length of stay’ indicator was used more than others and reported in approximately 80% of the studies (43 out of 53), followed by the ‘bed occupancy rate’ in approximately 40% of the studies (21 out of 53). The ‘bed turnover ratio’ and ‘hospitalization rate’ were also reported in 12 studies. Furthermore, for ‘safety’ dimensions, all indicators were in the ‘patient safety’ subcategory, which has been reported in 38 studies, and ‘complications’, ‘accidents or adverse events’, and ‘incidents or errors rates’ were the most concentrated indicators by researchers in 13, 12, and 11 studies, respectively. The performance dimension of ‘effectiveness’ was presented in 37 studies (40%), with only two indicators, ‘mortality rate’ in 29 studies and ‘readmission rate’ in 23 studies.

Performance categories

Considering the three categories shown in Table  4 , ‘organizational management’ indicators were more commonly used among the other two categories (‘clinical’ and ‘administrative’) and were present in more than 85% of the studies (78 out of 91). Two categories, ‘clinical management’ and ‘administrative management’, were reported in 62 and 51 studies, respectively.

Performance subcategories

Considering the 14 subcategories shown in Table  4 , both the ‘bed utilization rate’ and ‘patient safety’ indicators were mentioned in 46 studies and were more common among the other subcategories. The second most common indicator of the ‘financial management’ subcategory was reported in 38 studies. At the third level, both the ‘human resource management’ and ‘time management’ indicators were presented in 31 studies. The ‘paramedical’ subcategory indicators were presented in less than 10% of the studies [ 60 , 96 , 97 , 98 , 106 , 113 ].

Performance indicators

According to the indicator columns in Table  3 , the most used indicators in reviewed studies were the length of stay, mortality rate, and readmission rate in 47%, 32%, and 25% of studies, respectively. Bed occupancy rate and non-personnel costs were reported in 23% of studies. Additionally, among the 110 indicators, 16 indicators, namely, the lab cancellation rate, exam-physician ratios, number of coded diagnoses, number of medical records, laboratory sample/report intervals, medical information request time, safety standards in the archives, nutritional risk screening, imaging quality control failures, errors in medical reports, average impact factor, nutritional measures, laboratory scoring, imaging inspection, discharge process and emergency response rate, were reported in less than 1% of the studies.

The classification of the indicators in Table  4 was performed based on the chain model, which included the input, process, output, outcome and impact. The assignment of the indicators to each category was performed according to the experts’ opinions. For instance, the number of publications by academic member of an academic hospital and the average impact factor of those publications were considered outcome indicators. As depicted in the Table  4 , most studies (80%) focused more on output indicators. Additionally, fifteen studies focused on introducing and extracting some of the input, process, output, outcome and impact indicators; among those, only one study [ 96 ] has examined the input, process, output and impact indicators simultaneously.

Additionally, in approximately 42% (36 out of 91) of the studies, the indicators’ definitions, formulas, or descriptions have been illustrated, while less than 10% of the studies have defined measuring units, standard or benchmark units for all studied indicators [ 15 , 43 , 45 , 51 , 52 , 57 , 67 ].

Overall, nine studies related to hospital performance evaluation were conducted using systematic review methodologies (five systematic reviews [ 16 , 29 , 30 , 56 , 113 ], two literature reviews [ 79 , 80 ], one narrative review [ 98 ] and one brief review [ 92 ]). Most of these studies focused on extracting performance indicators from one or more hospital departments (e.g., the emergency department) [ 16 , 56 ], hospital laboratory and radiology information systems [ 106 ], supply chain performance [ 29 ], resources and financial results and activity [ 113 ], hospital water consumption [ 30 ], and the pharmaceutical sector [ 98 ]. Other reviews included a three-step process to review, evaluate and rank these hospital indicators in a systematic approach [ 16 ], or to evaluate performance indicator models to create an interactive network and visualize the causal relationships between performance indicators [ 79 ]; moreover, some have focused on the importance of indicators to ensure adequate coverage of the relevant areas of health care services to be evaluated [ 92 ].

Only one scoping review aimed to identify current assessments of hospital performance and compared quality measures from each method in the context of the six qualitative domains of STEEEP (safety, timeliness, effectiveness, efficiency, equity, and patient-centeredness) of the Institute of Medicine (IOM) in accordance with Donabedian’s framework and formulating policy recommendations [ 115 ].

In addition, 21 studies divided performance indicators into 2 to 6 dimensions of performance. Also, the reviewed studies included 2–40 indicators in zero [ 29 , 30 , 98 ] to 6 domains [ 34 ]. Moreover, none of the studies have tried to comprehensively summarize and categorize the performance indicators in several categories, focusing on all the indicators reflecting the performance of the entire hospital organization, or the indicators of administrative units or clinical departments.

In this scoping review, a unique set of hospital performance evaluation indicators related to the various performance dimensions was categorized from 91 studies over the past ten years.

Similarly, in a study, 19 performance dimensions, 32 sub-dimensions, and 138 indicators were extracted from only six studies. Those dimensions were described by all studies included in the review, but only three studies specified the relevant indicators, and the list provided for all possible indicators was not comprehensive. Also, despite current review, there was no classification of indicators based on the hospital levels: managerial, clinical, or organizational levels [ 116 ]. Another study has similarly investigated the performance evaluation indicators of the hospital in such a way that among 42 studies, 111 indicators were presented in the four categories: input, output, outcome, and impact. But, there was no classification of indicators based on performance dimensions and hospital levels [ 117 ].

In this study, the importance of categorized indicators, for the first time to our knowledge, was determined based on their frequency of use in the published literature (Appendix 2 ). The ‘Organizational management’ indicators were the most common compared with the other two categories (‘clinical’ and ‘administrative’). It could be because of the fact that the indicators such as ‘bed occupancy rate’, ‘average length of stay’, ‘mortality rate’, ‘hospital infection rate’, and ‘patient safety’ are easier to be registered in hospital software compared to other indicators, and also they better reflect the overall performance of hospital. Thus, researchers are more interested in using these indicators.

Considering 14 subcategories, indicators related to three subcategories i.e. bed utilization, patient safety and financial management are the most frequent used indicators for hospital performance evaluation. It reflects the need of hospital managers to increase the profitability of hospital in one hand, and to control cost on the other hand. As a results, researchers have paid special attention to ‘cost income’, ‘profitability’, ‘economic’, etc., as indicators for evaluating hospital performance.

When considering indicators by type, more studies have focused on output indicators, while input indicators were the least common used. This might be because of the fact that at hospital level, it is difficult for managers to change those inputs such as ‘beds’, ‘human resources’, ‘equipment and facilities’. In addition, due to the complexity of interdepartmental relationships in hospitals, process indicators seemed to provide more variety for analysis than input indicators, so they were more often used. As mentioned above, output indicators were the most used indicators for hospital performance evaluation due to their ease of calculation and interpretation.

The main purpose of this paper was to identify a comprehensive set of indicators that can be used to evaluate hospital performance in various hospital settings by being distilled into a smaller and more related set of indicators for every hospital or department setting. future studies could be designed to validate each set of indicators in any specific context. In addition, they could investigate the relationship between the indicators and their outcomes of interest and the performance dimension each could address. This will enable hospital managers to build their own set of indicators for performance evaluation both at organization or at department level. Also it should be mentioned that.

Although some previous studies have provided definitions for each indicator and determined the standard criteria for them, this was not done in this study because the focus of this study was to provide a collection of all the indicators used in hospital performance evaluation, which resulted in the identification of more than a thousand indicators without limiting to specific country or context. So while preparing a smaller set of indicators, specific conditions of each country, such as the type of health system and its policy, the type of financing system, and the structure of services, should be taken into account to select appropriate indicators.

In addition, although it is important to examine the scope of each article to compare the list of indicators and the relationships between the dimensions of the hospital in terms of size and type and between the number and type of selected indicators, this was considered beyond the scope of this review due to the high number of indicators, which made the abovementioned investigations impossible. Future studies could do that while working with a smaller set of indicators.

This review aimed to categorize and present a comprehensive set of indicators for evaluating overall hospital performance in a systematic way. 1161 hospital performance indicators were drawn from 91 studies over the past ten years. They then were summarized into 110 main indicators, and categorized into three categories: 14 subcategories, and 21 performance dimensions This scoping review also highlighted the most frequent used indicators in performance evaluation studies which could reflect their importance for that purpose. The results of this review help hospital managers to build their own set of indicators for performance evaluation both at organization or at department level with regard to various performance dimensions.

As the results of this review was not limited to any specific country or context, specific conditions of each country, such as the type of health system and its policy, the type of financing system, and the structure of services, should be taken into account while selecting appropriate indicators as a smaller set of indicators for hospital performance evaluation in specific context.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Gross domestic product

Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews

Emergency departments

Intensive care unit

Operating room

Surgical intensive care unit

Neonatal intensive care unit

Readmission rate

Quality Control

Medication use evaluation

safety, timeliness, effectiveness, efficiency, equity, and patient-centeredness

Institute of Medicine

Abdullah A, Ahmad S, Athar MA, Rajpoot N, Talib F. Healthcare performance management using integrated FUCOM-MARCOS approach: the case of India. Int J Health Plann Manage. 2022;37(5):2635–68.

Article   PubMed   Google Scholar  

Pestana M, Pereira R, Moro S. Improving health care management in hospitals through a productivity dashboard. J Med Syst. 2020;44(4):87.

Amos D. A practical framework for performance measurement of facilities management services in developing countries’ public hospitals. J Facil Manag. 2022;20(5):713–31.

Article   Google Scholar  

Ahmed S, Hasan MZ, MacLennan M, Dorin F, Ahmed MW, Hasan MM, et al. Measuring the efficiency of health systems in Asia: a data envelopment analysis. BMJ Open. 2019;9(3):e022155.

Article   PubMed   PubMed Central   Google Scholar  

Mitkova Z, Doneva M, Gerasimov N, Tachkov K, Dimitrova M, Kamusheva M, et al. Analysis of healthcare expenditures in Bulgaria. Healthc. 2022;10(2):274.

Patrinely JR, Walker SH, Glassman GE, Davis MJ, Abu-Ghname A, Khan U, et al. The importance of financial metrics in physician funding and performance evaluation. Plast Reconstr Surg. 2021;147:1213–8.

Article   CAS   PubMed   Google Scholar  

Buathong S, Bangchokdee S. The use of the performance measures in Thai public hospitals. ASIAN Rev Acc. 2017;25(4):472–85.

Google Scholar  

Imani A, Alibabayee R, Golestani M, Dalal K. Key indicators affecting hospital efficiency: a systematic review. Front Public Heal. 2022;10:830102.

Mahdiyan S, Dehghani A, Tafti AD, Pakdaman M, Askari R. Hospitals’ efficiency in Iran: a systematic review and meta-analysis. J Educ Health Promot. 2019;8(1):126.

PubMed   PubMed Central   Google Scholar  

Amos D, Musa ZN, Au-Yong CP. Performance measurement of facilities management services in Ghana’s public hospitals. Build Res Inf. 2020;48(2):218–38.

Feibert DC, Andersen B, Jacobsen P. Benchmarking healthcare logistics processes–a comparative case study of Danish and US hospitals. Total Qual Manag Bus Excell. 2019;30(1–2):108–34.

Gün I, Yilmaz F, Şenel IK. Efficiency analysis of health systems in world bank countries. Arch Heal Sci Res. 2021;8(2):147–52.

Breyer JZ, Giacomazzi J, Kuhmmer R, Lima KM, Hammes LS, Ribeiro RA, et al. Hospital quality indicators: a systematic review. Int J Health Care Qual Assur. 2019;32(2):474–87.

Regragui H, Sefiani N, Azzouzi H. Improving performance through measurement: the application of BSC and AHP in healthcare organization. In: Equipe De Recherche, Ingénierie, Innovation Et Management Des Systèmes Industriels, Université Abdelmalek Saadi. Tanger, Morocco: Institute of Electrical and Electronics Engineers Inc; 2018. p. 51–6.

Ghozali MT, Latifah DN, Darayani A. Analysis of Drug Supply Management of the Pharmacy Warehouse of Prof. Dr. Soerojo Mental Health Hospital, Magelang, Indonesia. Clin Schizophr Relat Psychoses. 2021;15:1–6.

Etu EE, Monplaisir L, Aguwa C, Arslanturk S, Masoud S, Markevych I, et al. Identifying indicators influencing emergency department performance during a medical surge: a consensus-based modified fuzzy Delphi approach. PLoS ONE. 2022;17(4 April):e0265101.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Lin C-Y, Shih F-C, Ho Y-H. Applying the balanced scorecard to build service performance measurements of medical institutions: An AHP-DEMATEL approach. Int J Environ Res Public Health. 2023;20(2):1022.

Backman C, Vanderloo S, Forster AJ. Measuring and improving quality in university hospitals in Canada: the collaborative for excellence in healthcare quality. Health Policy (New York). 2016;120(9):982–6.

Ghalem Â, Okar C, Chroqui R, Semma E. Performance: A concept to define. In: Performance: A concept to define. LOGISTIQUA 2016; 2016. p. 1–13.

Sonnentag S, Frese M. Performance Concepts and Performance Theory. In 2005. p. 1–25.

Tangen S. Demystifying productivity and performance. Int J Prod Perform Manag. 2005;54:34–46.

Elena-Iuliana I, Maria C. Organizational Performance – A Concept That Self-Seeks To Find Itself. Ann - Econ Ser Constantin Brancusi Univ Fac Econ. 2016;4(4):179–83.

Riratanaphong C, Van der Voordt T, Sarasoja A. Performance Measurement in the context of CREM and FM. In: Per Anker Jensen, Theo Van der Voordt CC, editor. The added value of facilities management: concepts, findings and perspectives. Lyngby Denmark: Polyteknisk Forlag; 2012. p. 1–21.

Pham M, Rajić A, Greig J, Sargeant J, Papadopoulos A, Mcewen S. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85.

Chaney M. So you want to write a narrative review article? J Cardiothorac Vasc Anesth. 2021;35:3045–9.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Tricco A, Lillie E, Zarin W, O’Brien K, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Dolatabad AH, Mahdiraji HA, Babgohari AZ, Garza-Reyes JA, Ai A. Analyzing the key performance indicators of circular supply chains by hybrid fuzzy cognitive mapping and Fuzzy DEMATEL: evidence from healthcare sector. Environ Dev Sustain. 2022;1–27.

Batista KJM, da Silva SR, Rabbani ERK, Zlatar T. Systematic review of indicators for the assessment of water consumption rates at hospitals. Water Supply. 2020;20(2):373–82.

Beta G, Role D, Berloviene D, Balkena Z. PATIENTS’ SATISFICATION AS THE QUALITY INDICATOR OF NURSING. In: Lubkina V, Kaupuzs A, Znotina D, editors. SOCIETY INTEGRATION EDUCATION, VOL VI: PUBLIC HEALTH AND SPORT, RESEARCHES IN ECONOMICS AND MANAGEMENT FOR SUSTAINABLE EDUCATION. 2020. p. 79–88.

Bhardwaj P, Joshi NK, Singh P, Suthar P, Joshi V, Jain YK, et al. Competence-based assessment of biomedical equipment management and maintenance system (e-Upkaran) using benefit evaluation framework. CUREUS J Med Sci. 2022;14(10):e30579.

Cheon O, Song M, Mccrea AM, Meier KJ. Health care in America: the relationship between subjective and objective assessments of hospitals. Int PUBLIC Manag J. 2021;24(5):596–622.

Craig KJT, McKillop MM, Huang HT, George J, Punwani ES, Rhee KB. US hospital performance methodologies: a scoping review to identify opportunities for crossing the quality chasm. BMC Health Serv Res. 2020;20(1):640.

Cristiano S, Ulgiati S, Gonella F. Systemic sustainability and resilience assessment of health systems, addressing global societal priorities: Learnings from a top nonprofit hospital in a bioclimatic building in Africa. Renew Sustain ENERGY Rev. 2021;141:110765.

Dadi D, Introna V, Santolamazza A, Salvio M, Martini C, Pastura T, et al. Private Hospital Energy Performance Benchmarking Using Energy Audit Data: An Italian Case Study. Energies. 2022;15(3):1–16.

Dawande PP, Wankhade RS, Akhtar FI, Noman O. Turnaround time: an efficacy measure for medical laboratories. CUREUS J Med Sci. 2022;14(9):e28824.

De Sousa LR, Mazzo A, De Almeida ACF, Tonello C, Lourençone LFM. Evaluation of quality indicators in the management of an operating room at a tertiary-level hospital. Med. 2022;55(1):1–8.

Drynda S, Schindler W, Slagman A, Pollmanns J, Horenkamp-Sonntag D, Schirrmeister W, et al. Evaluation of outcome relevance of quality indicators in the emergency department (ENQuIRE): study protocol for a prospective multicentre cohort study. BMJ Open. 2020;10(9):e038776.

Fekri O, Manukyan E, Klazinga N. Appropriateness, effectiveness and safety of care delivered in Canadian hospitals: a longitudinal assessment on the utility of publicly reported performance trend data between 2012–2013 and 2016–2017. BMJ Open. 2020;10(6):e035447.

Galloa AJO, Ramírez CA. Evaluating Colombian public hospitals productivity during 2004–2015. A luenberger-indicator approach. Rev Gerenc Y Polit Salud. 2020;19:1–23.

Gebreyes M, Sisay A, Tegen D, Asnake A, Wolde M. Evaluation of laboratory performance, associated factors and staff awareness towards achieving turnaround time in tertiary hospitals, Ethiopia. Ethiop J Health Sci. 2020;30(5):767–76.

Gnanaraj J, Kulkarni RG, Sahoo D, Abhishekh B. Assessment of the Key Performance Indicator Proposed by NABH in the Blood Centre of a Tertiary Health Care Hospital in Southern India. Indian J Hematol Blood Transfus. 2022;39:308–16.

Horenberg F, Lungu DA, Nuti S. Measuring research in the big data era: the evolution of performance measurement systems in the Italian teaching hospitals. Health Policy (New York). 2020;124(12):1387–94.

Hwang DK, Cho J, Moon J. Feasibility study on energy audit and data driven analysis procedure for building energy efficiency: bench-marking in Korean hospital buildings. Energies. 2019;14(15):3006.

Jaskova D. Efficiency of management, processes in a private hospital. Entrep Sustain Issues. 2021;9(1):436–46.

Jebraeily M, Valizadeh MA, Rahimi B, Saeidi S. The development of a clinical dashboard for monitoring of key performance indicators in ICU. J Iran Med Counc. 2022;5(2):308–17.

Kang Y, Kim M, Jung K. The equity of health care spending in South Korea: testing the impact of publicness. Int J Environ Res Public Health. 2020;17(5):1775.

Abou Tarieh RR, Zayyat R, Naoufal RN, Samaha HR. A case study exploring the impact of JCI standards implementation on staff productivity and motivation at the laboratory and blood bank. Heal Sci Rep. 2022;5(1):e497.

Kadoic N, Simic D, Mesaric J, Redep NB. Measuring quality of public hospitals in croatia using a multi-criteria Approach. Int J Environ Res Public Health. 2021;18:19.

Khalilabad T, Amir N, Asl P, Raeissi Shali M, Niknam N. Assessment of clinical and paraclinical departments of military hospitals based on the Pabon Lasso Model. J Educ Health Promot. 2020;9:1–6.

Lai JHK, Hou H, Edwards DJ, Yuen PL. An analytic network process model for hospital facilities management performance evaluation. Facilities. 2022;40(5–6):333–52.

Lai J, Yuen PL. Identification, classification and shortlisting of performance indicators for hospital facilities management. Facilities. 2021;39(1–2):4–18.

Lin CS, Chiu CM, Huang YC, Lang HC, Chen MS. Evaluating the operational efficiency and quality of Tertiary hospitals in Taiwan: the application of the EBITDA Indicator to the DEA Method and TOBIT Regression. Healthcare. 2022;10(1):58.

Matos R, Ferreira D, Pedro MI. Economic analysis of portuguese public hospitals through the construction of quality, efficiency, access, and financial related composite indicators. Soc Indic Res. 2021;157(1):361–92.

Morisod K, Luta X, Marti J, Spycher J, Malebranche M, Bodenmann P. Measuring health equity in emergency care using routinely collected data: a systematic review. Heal Equity. 2021;5(1):801–17.

Nik Hisamuddin R, Tuan Hairulnizam TK. Developing key performance indicators for emergency department of teaching hospitals: a mixed fuzzy Delphi and nominal group technique approach. Malays J Med Sci. 2022;29(2):114–25.

Ramírez Calazans A, Paredes Esteban RM, Grijalva Estrada OB, Ibarra Rodríguez MR. Assessment of quality indicators in pediatric major outpatient surgery. Influence of the COVID-19 pandemic. Cir Pediatr. 2023;36(1):17–21.

PubMed   Google Scholar  

Shaqura II, Gholami M, Akbari Sari A. Assessment of public hospitals performance in Gaza governorates using the Pabón Lasso Model. Int J Health Plann Manage. 2021;36(4):1223–35.

Al-Jazairi AS, Alnakhli AO. Quantifying clinical pharmacist activities in a tertiary care hospital using key performance indicators. Hosp Pharm. 2021;56(4):321–7.

Aloh HE, Onwujekwe OE, Aloh OG, Nweke CJ. Is bed turnover rate a good metric for hospital scale efficiency? A measure of resource utilization rate for hospitals in Southeast Nigeria. Cost Eff Resour Alloc. 2020;18(1):1–8.

Bari S, Incorvia J, Ahearn O, Dara L, Sharma S, Varallo J, et al. Building safe surgery knowledge and capacity in Cambodia: a mixed-methods evaluation of an innovative training and mentorship intervention. Glob Health Action. 2021;14(1):1998996.

Bari S, Incorvia J, Iverson KR, Bekele A, Garringer K, Ahearn O, et al. Surgical data strengthening in Ethiopia: results of a Kirkpatrick framework evaluation of a data quality intervention. Glob Health Action. 2021;14(1):1–11.

Bastos LSL, Hamacher S, Zampieri FG, Cavalcanti AB, Salluh JIF, Bozza FA. Structure and process associated with the efficiency of intensive care units in low-resource settings: an analysis of the CHECKLIST-ICU trial database. J Crit Care. 2020;59:118–23.

Bastos LSL, Wortel SA, de Keizer NF, Bakhshi-Raiez F, Salluh JIF, Dongelmans DA, et al. Comparing continuous versus categorical measures to assess and benchmark intensive care unit performance. J Crit Care. 2022;70:154063.

Kocisova K, Hass-Symotiuk M, Kludacz-Alessandri M. Use of the dea method to verify the performance model for hospitals. E M Ekon A Manag. 2018;21(4):125–40.

Lee D, Yu S, Yoon SN. Analysis of hospital management based on the characteristics of hospitals: focusing on financial indicators. Glob Bus Financ Rev. 2019;24(3):1–13.

Mirzaei A, Tabibi SJ, Nasiripour AA, Riahi L. Evaluating the feasibility of financial variables of health: A hospital administrator’s viewpoint. Galen Med J. 2016;5(1):25–30.

Middleton S, Gardner G, Gardner A, Considine J, FitzGerald G, Christofis L, et al. Are service and patient indicators different in the presence or absence of nurse practitioners? The EDPRAC cohort study of Australian emergency departments. BMJ Open. 2019;9(7):e024529.

Nobakht S, Jahangiri K, Hajinabi K. Correlation of performance indicators and productivity: A cross sectional study of emergency departments in Tehran, Iran during year 2016. Trauma Mon. 2018;23(5):1–6.

Nuti S, Grillo Ruggieri T, Podetti S. Do university hospitals perform better than general hospitals? A comparative analysis among Italian regions. BMJ Open. 2016;6(8):e011426.

Petrovic GM, Vukovic M, Vranes AJ. The impact of accreditation on health care quality in hospitals. Vojnosanit Pregl. 2018;75(8):803–8.

Pirani N, Zahiri M, Engali KA, Torabipour A. Hospital efficiency measurement before and after health sector evolution plan in Southwest of Iran: a DEA-panel data study. Acta Inf Med. 2018;26(2):106–10.

Profit J, Gould JB, Bennett M, Goldstein BA, Draper D, Phibbs CS, et al. The association of level of care with NICU quality. Pediatrics. 2016;137(3):44–51.

Rahimi H, Bahmaei J, Shojaei P, Kavosi Z, Khavasi M. Developing a strategy map to improve public hospitals performance with balanced scorecard and dematel approach. Shiraz E Med J. 2018;19(7):1–12.

Ahmed S, Hasan MZ, Laokri S, Jannat Z, Ahmed MW, Dorin F, et al. Technical efficiency of public district hospitals in Bangladesh: a data envelopment analysis. COST Eff Resour Alloc. 2019;17:17.

Rahman MH, Tumpa TJ, Ali SM, Paul SK. A grey approach to predicting healthcare performance. Meas J Int Meas Confed. 2019;134:307–25.

Sajadi HS, Sajadi ZS, Sajadi FA, Hadi M, Zahmatkesh M. The comparison of hospitals’ performance indicators before and after the Iran’s hospital care transformations plan. J Educ Health Promot. 2017;6:89.

Si S-L, You X-Y, Liu H-C, Huang J. Identifying key performance indicators for holistic hospital management with a modified DEMATEL approach. Int J Environ Res Public Health. 2017;14(8): 934.

Váchová L, Hajdíková T. Evaluation of Czech hospitals performance using MCDM methods. In: A SI, G WS, C D, editors. Department of exact methods, faculty of management, university of economics, Prague, Jarošovská 1117, Jindřichuv Hradec, vol. 37701. Czech Republic: Newswood Limited; 2017. p. 732–5.

Xenos P, Yfantopoulos J, Nektarios M, Polyzos N, Tinios P, Constantopoulos A. Efficiency and productivity assessment of public hospitals in Greece during the crisis period 2009–2012. Cost Eff Resour Alloc. 2017;15(1):6.

Zhang L, Liu R, Jiang S, Luo G, Liu H-C. Identification of key performance indicators for hospital management using an extended hesitant linguistic DEMATEL Approach. Healthc (Basel Switzerland). 2019;8(1):7.

Aksezer CS. A nonparametric approach for optimal reliability allocation in health services. Int J Qual Reliab Manag. 2016;33(2):284–94.

Cagliano AC, Grimaldi S, Rafele C. Assessing warehouse centralization and outsourcing in the healthcare sector: an Italian case study. In: Department of Management and Production Engineering, Politecnico Di Torino, Corso Duca Degli Abruzzi 24, Torino, 10129. Italy: AIDI - Italian Association of Industrial Operations Professors; 2017. p. 244–50.

Cefalu MS, Elliott MN, Setodji CM, Cleary PD, Hays RD. Hospital quality indicators are not unidimensional: a reanalysis of Lieberthal and Comer. Health Serv Res. 2019;54(2):502–8.

Gao H, Chen H, Feng J, Qin X, Wang X, Liang S, et al. Balanced scorecard-based performance evaluation of Chinese county hospitals in underdeveloped areas. J Int Med Res. 2018;46(5):1947–62.

Gonnelli V, Satta F, Frosini F, Iadanza E. Evidence-based approach to medical equipment maintenance monitoring. In: V HEO, V J, editors. University of Florence, Dept. of Information Engineering. Florence, Italy: Springer; 2017. p. 258–61.

Helkio P, Aantaa R, Virolainen P, Tuominen R. Productivity benchmarks for operative service units. ACTA Anaesthesiol Scand. 2016;60(4):450–6.

Khalifa M, Zabani I. Developing emergency room key performance indicators: What to measure and why should we measure it? J. M, A. H, P. G, A. K, M.S. H, editors. Vol. 226. King Faisal Specialist Hospital and Research Center, Jeddah, Saudi Arabia: IOS Press BV; 2016. p. 179–182.

Ajami S, Ebadsichani A, Tofighi S, Tavakoli N. Medical records department and balanced scorecard approach. J Educ Health Promot. 2013;2:7.

Bosse G, Mtatifikolo F, Abels W, Strosing C, Breuer J-P, Spies C. Immediate outcome indicators in perioperative care: a controlled intervention study on quality improvement in hospitals in Tanzania. PLoS One. 2013;8(6):e65428.

Hung K-Y, Jerng J-S. Time to have a paradigm shift in health care quality measurement. J Formos Med Assoc. 2014;113(10):673–9.

Jeejeebhoy KN, Keller H, Gramlich L, Allard JP, Laporte M, Duerksen DR, et al. Nutritional assessment: comparison of clinical assessment and objective variables for the prediction of length of hospital stay and readmission. Am J Clin Nutr. 2015;101(5):956–65.

Kittelsen SAC, Anthun KS, Goude F, Huitfeldt IMS, Häkkinen U, Kruse M, et al. Costs and quality at the hospital level in the nordic countries. Heal Econ (United Kingdom). 2015;24:140–63.

Koné Péfoyo AJ, Wodchis WP. Organizational performance impacting patient satisfaction in Ontario hospitals: a multilevel analysis. BMC Res Notes. 2013;6: 509.

Li CH, Yu CH. Performance evaluation of public non-profit hospitals using a BP Artificial neural network: the case of Hubei Province in China. Int J Environ Res Public Health. 2013;10(8):3619–33.

Liu K, Jain S, Shi J. Physician performance assessment using a composite quality index. Stat Med. 2013;32(15):2661–80.

Lloyd GF, Bajorek B, Barclay P, Goh S. Narrative review: Status of key performance indicators in contemporary hospital pharmacy practice. J Pharm Pract Res. 2015;45(4):396–403.

Mehrtak M, Yusefzadeh H, Jaafaripooyan E. Pabon Lasso and data envelopment analysis: a complementary approach to hospital performance measurement. Glob J Health Sci. 2014;6(4):107–16.

Mohammadi M, Ziapoor A, Mahboubi M, Faroukhi A, Amani N, Pour FH, et al. Performance evaluation of hospitals under supervision of Kermanshah medical sciences using pabonlasoty diagram of a five-year period (2008–2012). Life Sci J. 2014;11:77–81 ( 1 SPECL. ISSUE) ).

Niaksu O, Zaptorius J. Applying operational research and data mining to performance based medical personnel motivation system. In: Vilnius University, Institute of Mathematics and Informatics. Lithuania: IOS; 2014. p. 63–70.

Córdoba S, Caballero I, Navalón R, Martínez-Sánchez D, Martínez-Morán C, Borbujo J. Analysis of the surgical activity in the dermatology department of Fuenlabrada University Hospital, Madrid, Spain, between 2005 and 2010: determination of the standard operating times. Actas Dermosifiliogr. 2013;104(2):141–7.

Profit J, Kowalkowski MA, Zupancic JAF, Pietz K, Richardson P, Draper D, et al. Baby-MONITOR: a composite indicator of NICU Quality. Pediatrics. 2014;134(1):74–82.

Rabar D, Pap N. Evaluation of crotia’s regional hospital effiency: an application of data envelopment analysis . Bacher U, Barkovic D, Dernoscheg KH, LamzaMaronic M, Matic B, Runzheimer B, editors. Interdisciplinary Management Research IX. 2013;9:649–59.

Ramos MCA, da Cruz LP, Kishima VC, Pollara WM, de Lira ACO, Couttolenc BF. Performance evaluation of hospitals that provide care in the public health system, Brazil. Rev Saude Publica. 2015;49:1–9.

Schuers M, Joulakian MB, Griffon N, Pachéco J, Périgard C, Lepage E, et al. In: S IN, de PM AM, editors. Quality indicators from laboratory and radiology information systems. A. G. Volume 216. France: IOS; 2015. pp. 212–6. Department of Biomedical Informatics, Rouen University Hospital, Rouen Cedex, 76031,.

Tabrizi JS, Saadati M, Sadeghi-Bazargani H, Ebadi A, Golzari SEJ. Developing indicators to improve educational governance in hospitals. Clin Gov. 2014;19(2):117–25.

Costa A Jr, aS., Leão LE, Novais MA, Zucchi P. An assessment of the quality indicators of operative and non-operative times in a public university hospital. Einstein (Sao Paulo). 2015;13(4):594–9.

Coyne JS, Helton J. How prepared are US hospitals for the affordable care act? A financial condition analysis of US hospitals in 2011. J Health Care Finance. 2015;41(3).

Davis P, Milne B, Parker K, Hider P, Lay-Yee R, Cumming J, et al. Efficiency, effectiveness, equity (E-3). Evaluating hospital performance in three dimensions. Health Policy (New York). 2013;112(1–2):19–27.

Flatow VH, Ibragimova N, Divino CM, Eshak DSA, Twohig BC, Bassily-Marcus AM, et al. Quality outcomes in the surgical intensive care unit after electronic health record implementation. Appl Clin Inf. 2015;6(4):611–8.

Article   CAS   Google Scholar  

Fonseca JRS, Ramos RMP, Santos AMP, Fonseca APSS. Policy effects on the quality of public health care: evaluating Portuguese public hospitals’ quality through customers’ views. Cent Eur J Public Policy. 2015;9(2):122–40.

Hadji B, Meyer R, Melikeche S, Escalon S, Degoulet P. Assessing the Relationships Between Hospital Resources and Activities: A Systematic Review. J Med Syst. 2014;38(10):1–21.

Hajduová Z, Herbrik G, Beslerová S. Application of DEA in the environment of Slovak hospitals. Invest Manag Financ Innov. 2015;12(4):148–53.

Thomas Craig KJ, McKillop MM, Huang HT, George J, Punwani ES, Rhee KB. U.S. hospital performance methodologies: a scoping review to identify opportunities for crossing the quality chasm. BMC Health Serv Res. 2020;20(1):640.

Carini E, Gabutti I, Frisicale EM, Di Pilla A, Pezzullo AM, de Waure C, et al. Assessing hospital performance indicators. What dimensions? Evidence from an umbrella review. BMC Health Serv Res. 2020;20(1):1038.

Rasi V, Delgoshaee B, Maleki M. Identification of common indicators of hospital performance evaluation models: a scoping review. J Educ Health Promot. 2020;9(1):63.

Xenos P, Yfantopoulos J, Nektarios M, Polyzos N, Tinios P, Constantopoulos A. Efficiency and productivity assessment of public hospitals in Greece during the crisis period 2009–2012. COST Eff Resour Alloc. 2017;15:15.

Shaqura II, Gholami M, Sari AA. Evaluation of performance at Palestinian public hospitals using Pabon Lasso model. Int J Health Plann Manage. 2021;36(3):896–910.

Li J, Seale H, Ray P, Wang Q, Yang P, Li S, et al. E-Health preparedness assessment in the context of an influenza pandemic: a qualitative study in China. BMJ Open. 2013;3(3):e002293.

Huang C-Y, Lai C-H. Effects of internal branding management in a hospital context. Serv Ind J. 2021;41(15–16):985–1006.

Download references

Acknowledgements

The authors are grateful for the support of the Vice Chancellor for Research of Isfahan University of Medical Sciences.

The present article is part of the result of a doctoral thesis approved by Isfahan University of Medical Sciences with code 55657 (IR.MUI.NUREMA.REC.1401.005), without financial source.

Author information

Authors and affiliations.

Student Research Committee, School of Management and Medical Information Sciences, Isfahan University of Medical Sciences, Isfahan, Iran

Shirin Alsadat Hadian

Health Management and Economics Research Center, Isfahan University of Medical Sciences, Isfahan, Iran

Reza Rezayatmand

Hospital Management Research Center, Health Management Research Institute, Iran University of Medical Sciences, Tehran, Iran

Nasrin Shaarbafchizadeh

Department of Management, Faculty of Administrative Sciences and Economics, University of Isfahan, Isfahan, Iran

Saeedeh Ketabi

School of Public Health and Preventive Medicine, Monash University, Victoria, Australia

Ahmad Reza Pourghaderi

You can also search for this author in PubMed   Google Scholar

Contributions

Shirin Alsadat Hadian and Reza Rezayatmans and Saeedeh Ketabi: Study conceptualization and design. Acquisition of data: Shirin Alsadat Hadian, Reza Rezayatmand. Analysis and interpretation of data: Shirin Alsadat Hadian, Reza Rezayatmand, Nasrin Shaarbafchizadeh, Saeedeh Ketabi. Drafting of the manuscript: Shirin Alsadat Hadian, Reza Rezayatmand. Critical revision of the manuscript for important intellectual content: Reza Rezayatmand, Nasrin Shaarbafchizadeh, Saeedeh Ketabi, Ahmad Reza Pourghaderi.

Corresponding author

Correspondence to Reza Rezayatmand .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hadian, S.A., Rezayatmand, R., Shaarbafchizadeh, N. et al. Hospital performance evaluation indicators: a scoping review. BMC Health Serv Res 24 , 561 (2024). https://doi.org/10.1186/s12913-024-10940-1

Download citation

Received : 03 January 2024

Accepted : 02 April 2024

Published : 01 May 2024

DOI : https://doi.org/10.1186/s12913-024-10940-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Performance evaluation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 8.5.2024 in Vol 10 (2024)

The Scope of Virtual Reality Simulators in Radiology Education: Systematic Literature Review

Authors of this article:

• Shishir Shetty 1 , PhD ; 
• Supriya Bhat 2 , MDS ; 
• Saad Al Bayatti 1 , MSc ; 
• Sausan Al Kawas 1 , PhD ; 
• Wael Talaat 1 , PhD ; 
• Mohamed El-Kishawi 3 , PhD ; 
• Natheer Al Rawi 1 , PhD ; 
• Sangeetha Narasimhan 1 , PhD ; 
• Hiba Al-Daghestani 1 , MSc ; 
• Medhini Madi 4 , MDS ; 
• Raghavendra Shetty 5 , PhD

1 Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, , Sharjah, , United Arab Emirates

2 Department of Oral Medicine and Radiology, AB Shetty Memorial Institute of Dental Sciences, Nitte (Deemed to be University), , Mangalore, , India

3 Department of Preventive and Restorative Dentistry, College of Dental Medicine, University of Sharjah, , Sharjah, , United Arab Emirates

4 Department of Oral Medicine and Radiology, Manipal College of Dental Sciences, Manipal Academy of Higher Education, , Manipal, , India

5 Department of Clinical Sciences, College of Dentistry, Ajman University, , Ajman, , United Arab Emirates

Corresponding Author:

Supriya Bhat, MDS

Background: In recent years, virtual reality (VR) has gained significant importance in medical education. Radiology education also has seen the induction of VR technology. However, there is no comprehensive review in this specific area. This review aims to fill this knowledge gap.

Objective: This systematic literature review aims to explore the scope of VR use in radiology education.

Methods: A literature search was carried out using PubMed, Scopus, ScienceDirect, and Google Scholar for articles relating to the use of VR in radiology education, published from database inception to September 1, 2023. The identified articles were then subjected to a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)–defined study selection process.

Results: The database search identified 2503 nonduplicate articles. After PRISMA screening, 17 were included in the review for analysis, of which 3 (18%) were randomized controlled trials, 7 (41%) were randomized experimental trials, and 7 (41%) were cross-sectional studies. Of the 10 randomized trials, 3 (30%) had a low risk of bias, 5 (50%) showed some concerns, and 2 (20%) had a high risk of bias. Among the 7 cross-sectional studies, 2 (29%) scored “good” in the overall quality and the remaining 5 (71%) scored “fair.” VR was found to be significantly more effective than traditional methods of teaching in improving the radiographic and radiologic skills of students. The use of VR systems was found to improve the students’ skills in overall proficiency, patient positioning, equipment knowledge, equipment handling, and radiographic techniques. Student feedback was also reported in the included studies. The students generally provided positive feedback about the utility, ease of use, and satisfaction of VR systems, as well as their perceived positive impact on skill and knowledge acquisition.

Conclusions: The evidence from this review shows that the use of VR had significant benefit for students in various aspects of radiology education. However, the variable nature of the studies included in the review reduces the scope for a comprehensive recommendation of VR use in radiology education.

Introduction

The use of technology in education helps students achieve improved acquisition of professional knowledge and practical skills [ 1 - 3 ]. Virtual reality (VR) is a modern technology that simulates experience by producing 3D interactive situations and presenting objects in a virtual world with spatial dimensions [ 4 , 5 ]. VR technology can be classified as nonimmersive or immersive [ 6 ]. In a nonimmersive VR, the simulated 3D environment is experienced through a computer monitor [ 6 ]. On the other hand, an immersive VR provides a sense of presence in a computer-generated environment, created by producing realistic sights, sounds, and other sensations that replicate a user’s physical presence in a virtual environment [ 6 , 7 ]. Using VR technology, a person can look about the artificial world, navigate around in it, and interact with simulated objects or items [ 5 , 8 ]. Due to the broad nature of VR technology, it has many applications, some of which are in the field of medicine [ 9 , 10 ].

The use of VR in medicine started in the 1990s when medical researchers were trying to create 3D models of patients’ internal organs [ 11 - 13 ]. Since then, VR use in the field of medicine and general health care has increased substantially to cover many areas including medical education. Radiology education has also come to see the use of VR technology in the recent past [ 14 ]. The use of VR in radiology education enables students to practice radiography in a virtual environment, which is radiation free [ 15 ]. Additionally, the use of VR enables effective and repeatable training. This allows trainees to recognize and correct errors as they occur [ 16 , 17 ]. The aim of this review is to explore the scope of VR in radiology education.

This systematic review has been performed using the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) guidelines [ 18 ] [ Checklist 1 ]).

Information Sources and Study Selection

The bibliographic databases used were PubMed, Scopus, ScienceDirect, and Google Scholar. A systematic literature search was conducted for articles published from database inception to September 1, 2023. Topic keywords were used to generate search strings. The search strings that were used are provided in Table 1 . Only the first 10 pages of Google Scholar results were exported. The identified studies were then subjected to a study selection process. The search string for ScienceDirect was shorter because the database only allows a maximum of 8 Boolean operators, hence the sting had to be shortened. The search in PubMed was limited to the title and abstract. The searches in Scopus and ScienceDirect were limited to title, abstract, and keywords.

Inclusion and Exclusion Criteria

Original research articles written in the English language were included in the review. Studies conducted on medical, dental, and allied health sciences students (undergraduate and postgraduate) from any part of the world were included in the review. Studies exploring the use of VR learning in radiology education were included.

Narrative reviews, scoping reviews, systematic reviews, meta-analyses, editorials, and commentaries were excluded. Studies that did not align with the required study objective were excluded.

Method of Quality Assessment

Randomized controlled trials (RCTs) and randomized experimental studies were appraised using the RoB 2 tool from the Cochrane Collaboration [ 19 ]. A visualization of the risk-of-bias assessment was done using the web-based robvis tool [ 20 ]. Cross-sectional studies were appraised using the appraisal checklist for analytical cross-sectional studies from the Joanna Briggs Institute [ 21 ].

Data Extraction

Each article included in the review was summarized in a table, including basic study characteristics. The extracted attributes were study author(s), publication year, study design, type and number of participants, type of radiology education under study, and the outcome being assessed. The extracted data are provided in Table 2 .

a RCT: randomized controlled trial.

b CT: computed tomography.

Search Results

The database search identified a total of 2877 studies; 374 (13%) studies were from PubMed, 2169 (75.4%) were from Scopus, 234 (8.1%) were from ScienceDirect, and 100 (3.5%) were from Google Scholar. Before the screening procedure, 37 duplicates were removed. During title and abstract screening, 2808 articles were excluded since they did not align with the eligibility criteria. The remaining 32 articles were then subjected to a full-text review, and 15 were excluded for reasons provided in Figure 1 , which shows the study selection process [ 38 ]. At the end of the process, 17 studies were found eligible for inclusion in the review.

Characteristics of Included Studies

Among the 17 studies, 3 (18%) RCTs, 7 (41%) randomized experimental trials, and 7 (41%) cross-sectional studies were included. The studies encompassed various aspects of radiology education, including dental radiology [ 28 , 29 ], diagnostic radiology [ 22 , 24 ], and interventional radiology [ 25 , 31 ].

Results of Quality Assessment

Among the 7 cross-sectional studies, 2 (29%) scored “good” in overall quality and the remaining 5 (71%) scored “fair.” The results for the quality appraisal of cross-sectional studies are shown in Table 3 . Studies were appraised using the checklist for analytical cross-sectional studies from the Joanna Briggs Institute [ 21 ].

Among the 10 randomized trials, 3 (30%) had a low risk of bias, 5 (50%) showed some concerns, and 2 (20%) had a high risk of bias. These results are shown in Table 4 . RCTs were appraised using the RoB 2 tool from the Cochrane Collaboration [ 19 ]. A risk-of-bias graph ( Figure 2 ) and a risk-of-bias summary ( Figure 3 ) are also provided.

a Item 1: were the criteria for inclusion in the sample clearly defined?

b Item 2: were the study subjects and the setting described in detail?

c Item 3: was the exposure measured in a valid and reliable way?

d Item 4: were objective, standard criteria used for measurement of the condition?

e Item 5: were confounding factors identified?

f Item 6: were strategies to deal with confounding factors stated?

g Item 7: were the outcomes measured in a valid and reliable way?

h Item 8: was appropriate statistical analysis used?

i N/A: not assessable.

a D1: risk of bias arising from the randomization process.

b D2: risk of bias due to deviations from the intended interventions (effect of assignment to intervention).

c D3: risk of bias due to missing outcome data.

d D4: risk of bias in measurement of the outcome.

e D5: risk of bias in selection of the reported result.

Type of VR Hardware and Software Used in the Studies

The studies used a wide range of VR software and hardware. Some of the studies used 3D simulation software packages displayed on 2D desktop computers [ 22 , 24 , 25 , 36 ], whereas others used headsets for an immersive VR environment [ 15 , 23 , 26 , 35 , 37 ]. The most used VR teaching software were the CETSOL VR Clinic software [ 33 , 35 ], Virtual Medical Coaching VR software [ 15 , 30 , 32 ], Projection VR (Shaderware) software [ 36 ], SieVRt VR system (Luxsonic Technologies) [ 37 ], medical imaging training immersive environment software [ 23 ], VR CT Sim software [ 25 ], VitaSim ApS software [ 26 ], VR X-Ray (Skilitics and Virtual Medical Coaching) software [ 27 ], and radiation dosimetry VR software (Virtual Medical Coaching Ltd) [ 31 ].

Effect of VR Teaching on Skill Acquisition

Ahlqvist et al [ 22 ] looked at how virtual simulation can be used as an effective tool to teach quality assessment of radiographic images. They also compared how it faired in comparison to traditional teaching. The study reported a statistically significant improvement in proficiency from before training to after training. Additionally, the study reported that the proficiency score improvement for the VR-trained students was higher than that for the students trained using conventional method.

In the study conducted by Sapkaroski et al [ 34 ], students in the VR group demonstrated significantly better patient positioning skills compared to those in the conventional role-play group. The positioning parameters that were assessed were digit separation and palm flatness (the VR group scored 11% better), central ray positioning onto the third metacarpophalangeal joint (the VR group scored 23% better), and a control position projection of an oblique hand. The results for the control position projection indicated no significant difference in positioning between the 2 groups [ 34 ].

Bridge et al [ 23 ] also performed a performance comparison between students trained by VR and traditional methods. They assessed skills about patient positioning, equipment positioning, and time taken to complete a performative role-play. Students in the VR group performed better than those in the control group, with 91% of them receiving an overall score of above average (>3). The difference in mean group performance was statistically significant ( P =.0366). Similarly, Gunn et al [ 24 ] reported improved and higher role-play skill scores for students trained using VR software simulation compared to those trained on traditional laboratory simulation. The mean role-play score for the VR group was 30.67 and that for the control group was 28.8 [ 24 ].

Another study reported that students trained using VR performed significantly better (ranked as “very good” or “excellent”) than the control group (conventional learning) in skills such as patient positioning, selecting exposure factors, centering and collimating the x-ray beam, placing the anatomical marker, appraisal of image quality, equipment positioning, and procedure explanation to the patient [ 30 ]. Another recently conducted study found that the VR-taught group achieved better test duration and fewer errors in moving equipment and positioning a patient. There was no significant difference in the frequency of errors in the radiographic exposure setting such as source-to-image distance between the VR and the physical simulation groups [ 32 ].

Nilsson et al [ 28 ] developed a test to evaluate the student’s ability to interpret 3D information in radiographs using parallax. This test was applied to students before and after training. There was a significantly larger ( P <.01) pre-post intervention mean score for the VR group (3.11 to 4.18) compared to the control group (3.24 to 3.72). A subgroup analysis was also performed, and students with low visuospatial ability in the VR group had a significantly higher improvement in the proficiency test compared to those in the control group. The same authors conducted another follow-up study to test skill retention [ 29 ]. Net skill improvement was calculated as the difference in test scores after 8 months. The results from the proficiency test showed that the ability to interpret spatial relations in radiographs 8 months after the completion of VR training was significantly better than before VR training. The students who trained conventionally showed almost the same positive trend in improvement. The group difference was smaller and not statistically significant. This meant that, 8 months after training, the VR group and the traditionally trained group had the same skill level [ 29 ].

Among the included studies, only 1 reported that the VR group had lower performance in proficiency tests and radiographic skill tests, compared to a conventionally trained group. The study, conducted in 2022, showed that the proficiency of the VR group was significantly lower than that of the conventional technique group in performing lateral elbow and posterior-anterior chest radiography [ 27 ]. An itemized rubric evaluation used in the study revealed that the VR group also had lower performance in most of the radiographic skills, such as locating and centering of the x-ray beam, side marker placement, positioning the x-ray image detector, patient interaction, and process control and safety [ 27 ]. The study concluded that VR simulation can be less effective than real-world training in radiographic techniques, which requires palpation and patient interaction. These results may be different from those of other studies due to different outcome evaluation methods and since they used head-mounted display VR coaching, whereas the other studies, except O’Connor et al [ 15 ], used VR on a PC monitor.

All of the studies except Kato et al [ 27 ] agreed that VR use was more effective for students in developing radiographic and radiologic skills. Despite this general agreement, there were slight in-study variations in learning outcomes, which made some of the studies look at factors that may influence skill and knowledge acquisition during VR use. In studies such as Bridge et al [ 23 ], it was noted that the arrangement of equipment had the greatest influence on the overall score. After performing a multivariable analysis, Gunn et al [ 24 ] reported that there was no effect of age, gender, and gaming skills or activity on the outcome of VR learning. In the study by Shanahan [ 36 ], a few students (19/84, 23%) had previously used VR simulation software. This had no bearing on the learning outcomes. Another observation in the same study was that student age was found to significantly affected the student’s confidence about skill acquisition after VR training [ 36 ].

Students’ Perception of VR Uses for Learning

The findings from the study by Gunn et al [ 25 ] revealed that 68% of students agreed or strongly agreed that VR simulation was significantly helpful in learning about computed tomography (CT) scanning. In another study by Jensen et al [ 26 ], 90% of the students strongly agreed that VR simulators could contribute to learning radiography, with 90% reporting that the x-ray equipment in the VR simulation was realistic. In the study by Wu et al [ 37 ], most of the students (55.6%) agreed or somewhat agreed that VR use was useful in radiology education. Similarly, 83% of the students in Shanahan’s [ 36 ] study regarded VR learning with an ease of use. In the same study, students also reported that one of the major benefits of VR learning include using the simulation to repeat activities until being satisfied with the results (95% of respondents). Students also stated that VR enabled them to quickly see images and understand if changes needed to be made (94%) [ 36 ]. In the study by Gunn et al [ 25 ], 75% of medical imaging students agreed on the ease of use and software enjoyment in VR simulated learning. In the same study, 57% of the students reported a positive perceived usefulness of VR. Most respondents (80%) in the study by Rainford et al [ 31 ] favored the in-person VR experience over web-based VR. Similarly, 58% of the respondents in the study conducted by O’Connor et al [ 15 ] reported enjoying learning using VR simulation. In the study by Wu et al [ 37 ], 83.3% of students agreed or strongly agreed that they enjoyed using VR for learning. Similarly, the studies by Rainford et al [ 31 ] and O’Connor et al [ 15 ] reported student recommendation of 87% and 94%, respectively, for VR as a learning tool.

Students’ Perceived Skill and Knowledge Acquisition

In the study by Bridge et al [ 23 ], students who trained using VR reported an increase in perceived skill acquisition and high levels of satisfaction. The study authors attributed this feedback to the availability of “gold standards” that showed correct positioning techniques, as well as instant feedback provided by the VR simulators. Gunn et al [ 25 ] examined students’ confidence in performing a CT scan in a real clinical environment after using VR simulations as a learning tool. The study reported an increase (from before to after training) in the students’ perceived confidence in performing diagnostic CT scans. Similarly, the study by Jensen et al [ 26 ] reported that the use of VR had influenced students’ self-perceived readiness to perform wrist x-ray radiographs. The study, however, found no significant difference in pre- and posttraining (perceived preparedness) scores. The pre- and posttraining scores were 75 (95% CI 54-96) and 77 (95% CI 59-95), respectively. The study by O’Connor et al [ 15 ] looked at the effect of VR on perceived skill adoption. Most of the students in the study reported high levels of perceived knowledge acquisition in the areas of beam collimation, anatomical marker placement, centering of the x-ray tube, image evaluation, anatomical knowledge, patient positioning, and exposure parameter selection to their VR practice. However, most students felt that VR did not contribute to their knowledge of patient dose tracking and radiation safety [ 15 ]. In the study by Rainford et al [ 31 ], 73% of radiography and medical students felt that VR learning increased their confidence across all relevant learning outcomes. The biggest increase in confidence level was regarding their understanding of radiation safety matters [ 31 ]. Sapkaroski et al [ 33 ] performed a self-perception test to see how students viewed their clinical and technical skills after using VR for learning. In their study, students reported a perceived improvement in their hand and patient positioning skills. Their study also compared 2 software, CETSOL VR Clinic and Shaderware. The cohort who used CETSOL VR Clinic had higher scores on perceived improvement [ 33 ]. Sapkaroski et al [ 35 ] compared the student’s perception scores on the educational enhancement of their radiographic hand positioning skills, after VR or clinical role-play scenario training. Although the VR group scored higher, there was no significant difference between the scores for the 2 groups [ 35 ]. In the study by Shanahan [ 36 ], when the perception of skill development was evaluated, most of the students reported that the simulation positively developed their technical (78%), radiographic image evaluation (85%), problem-solving (85%), and self-evaluation (88%) abilities. However, in the study by Kato et al [ 27 ], there was no difference in the perceived acquisition of knowledge among students using traditional teaching and VR-based teaching.

Principal Findings

The results presented in this review reveal strong evidence for the effectiveness of VR teaching in radiology education, particularly in the context of skill acquisition and development [ 22 , 24 , 27 , 30 , 32 , 34 ].

In this review, quality appraisal of the cross-sectional studies revealed that the strategies for deal with confounding factors was one of the factors directly affecting the reliability of the results. Similarly, the appraisal of the randomized trials revealed that the bias arising due to missing outcome data was one of the factors directly affecting the reliability of the results.

All the studies found that VR-based teaching had a positive impact on various areas of radiographic and radiologic skill development. In comparison to the traditional way of teaching, only 1 study by Kato et al [ 27 ] reported VR teaching as inferior to traditional teaching. The studies consistently reported better improvements in proficiency, patient positioning outcomes, equipment handling, and radiographic techniques among students trained using VR. According to Nilsson et al [ 29 ], O’Connor et al [ 15 ], and Wu et al [ 37 ], the improvements were due to the immersive and interactive nature of VR simulations, which allowed learners to engage with radiological scenarios in a dynamic and hands-on manner. The studies also revealed that VR learning has the ability to easily and effectively introduce students to new skills. It was also found that existing skills could be improved, mainly through simulation feedback that happens in real time during training [ 22 , 24 , 28 , 30 , 36 ].

The improvement of skills after VR training have been noted in different domains, including patient positioning, equipment positioning, equipment knowledge, assessment of radiographic image quality, and patient interaction. Improvement was also observed in other skills such as as central ray positioning, source-to-image distance, image receptor placement, and side marker placement [ 22 , 24 , 30 , 32 , 34 ]. Two studies, Nilsson et al [ 28 ] and Nilsson et al [ 29 ], looked at how VR affected the students’ ability to interpret 3D information in radiographs using parallax. They both reported a positive effect. Nilsson et al [ 29 ] also gave insights into the long-term benefits of VR training in radiology. Eight months after training, the control (traditionally taught) group in Nilsson et al [ 29 ] showed a slight increase in skills, but the VR-trained group still maintained a significantly higher skill level. This finding shows the enduring impact of VR-based education on skill acquisition in radiology. Although most studies supported the effectiveness of VR in radiology education, 1 study reported contrasting results [ 27 ]. VR-trained students were found to perform worse than traditionally trained students in conducting lateral elbow and posterior-anterior chest radiography in Kato et al [ 27 ]. This difference in results was, according to the authors, attributed to the use of a different rubric evaluation method and the use of a head-mounted display–based immersive VR system, which was not used in other studies. These 2 reasons may be the reason for the variation in study findings.

A wide range of VR software with different functions were used in the studies. In addition to acquiring radiographic images, the CETSOL VR Clinic software facilitated students to interact with their learning environment [ 33 , 35 ]. Students using the Virtual Medical Coaching VR software performed imaging exercise on a virtual patient with VR headsets and hand controllers [ 15 , 30 , 32 ]. The SieVRt VR system displayed Digital Imaging and Communications in Medicine format images in a virtual environment, thus facilitating teaching [ 37 ]. The medical imaging training immersive environment simulation software provided automated feedback to the learners including a rerun of procedures, thus highlighting procedural errors [ 23 ]. The VR CT Sim software allowed the student virtually to perform the complete CT workflow [ 25 ]. Students could manipulate patient positioning and get feedback from the VitaSim ApS software [ 26 ]. The VR X-Ray software allowed students to manipulate radiographic equipment and patient’s position with a high level of immersive experience [ 27 ]. Radiation dosimetry VR software facilitated virtual movement of the staff and equipment to radiation-free areas, thus optimizing radiation protection [ 31 ].

The included studies also looked at factors that could influence skill acquisition when VR is used in radiology education. Bridge et al [ 23 ], Gunn et al [ 24 ], Kato et al [ 27 ], and Shanahan [ 36 ] investigated factors such as age, gender, prior gaming experience, and familiarity with VR technology. However, these factors were shown to have no significant effect on VR learning outcomes. This shows that VR education can equally accommodate a wide range of learners, regardless of experience or existing attributes.

Across several studies, positive feedback emerged regarding the utility, ease of use, enjoyment, and perceived impact on skill and knowledge acquisition. The included studies consistently reported positive perceptions of VR use among students [ 25 , 26 , 37 ]. Gunn et al [ 25 ] reported that a significant proportion of medical imaging and radiation therapy students found the use of VR simulation to be significantly helpful in learning about CT scanning. Similarly, Jensen et al [ 26 ] and Wu et al [ 37 ] reported that a majority of students agreed on the usefulness of VR in radiology education. Another aspect that received positive feedback was the ease of use. Students liked the ability to repeat tasks until they were satisfied with the results and the ability to quickly visualize radiographs to determine the need for revisions [ 36 ]. Rainford et al [ 31 ] and O’Connor and Rainford [ 30 ] found that most students would recommend VR as a learning tool to other students.

Several studies investigated student’s perceptions of skill and knowledge acquisition when using VR for radiology education. Bridge et al [ 15 ] and O’Connor et al [ 23 ] discovered an increase in students’ perceived acquisition of radiographic skills. Gunn et al [ 25 ] reported an increase in students’ perceived confidence to perform CT scans after learning using VR simulations. According to Rainford et al [ 31 ], a large percentage of radiography and medical students felt that VR learning boosted their confidence across all relevant learning outcomes, with the highest levels of confidence recorded in radiation safety. Sapkaroski et al [ 33 ] discovered that after using VR for learning, students experienced an improvement in their hand and patient placement skills. In summary, the positive feedback from the students shows that VR use in radiology education is a useful, engaging, and effective teaching tool. This perceived acquisition of skills is backed by the results from the proficiency tests.

The VR modalities used in some of the studies allowed remote assistance from an external agent (teacher), as the VR training is conducted in front of a screen while being part of a team, with the teacher making constant corrections and indications [ 22 , 24 , 27 ]. However, researchers are looking into VR systems with artificial intelligence–supported tutoring, which includes the assessment of learners, generation of learning content, and automated feedback [ 39 ].

Findings from the included studies show that VR-based teaching offers substantial benefits in various aspects of radiographic and radiologic skill development. The studies consistently reported that students educated using VR systems improved significantly in overall proficiency, patient positioning, equipment knowledge, equipment handling, and radiographic techniques. However, the variable nature of the studies included in the review reduces the scope for a comprehensive recommendation of VR use in radiology education. A key contributing factor to relatively better learning outcomes was the immersive and interactive nature of VR systems, which provided real-time feedback and dynamic learning experiences to students. Factors such as age, gender, gaming experience, and familiarity with VR systems did not significantly influence learning outcomes. This shows that VR can be used for diverse groups of students when teaching radiology. Students generally provided positive feedback about the utility, ease of use, and satisfaction of VR, as well as its perceived impact on skill and knowledge acquisition. These students’ reports show the value of VR as an important, interesting, and effective tool in radiology education.

Conflicts of Interest

None declared.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

• Ding X, Li Z. A review of the application of virtual reality technology in higher education based on Web of Science literature data as an example. Front Educ. Nov 17, 2022;7:1048816. [ CrossRef ]
• Oyelere SS, Bouali N, Kaliisa R, Obaido G, Yunusa AA, Jimoh ER. Exploring the trends of educational virtual reality games: a systematic review of empirical studies. Smart Learning Environments. Oct 19, 2020;7:31. [ CrossRef ]
• Zamyatina NV, Ushakova IA, Mandrikov VB. Medical education in digital transformation. In: Proceedings of the International Scientific Conference “Digitalization of Education: History, Trends and Prospects” (DETP 2020). Atlantis Press; 2020;324-327. [ CrossRef ]
• Kardong-Edgren S, Farra SL, Alinier G, Young HM. A call to unify definitions of virtual reality. Clin Simul Nurs. Jun 2019;31:28-34. [ CrossRef ]
• Sousa Santos B, Dias P, Pimentel A, et al. Head-mounted display versus desktop for 3D navigation in virtual reality: a user study. Multimed Tools Appl. Jan 2009;41(1):161-181. [ CrossRef ]
• Hamad A, Jia B. How virtual reality technology has changed our lives: an overview of the current and potential applications and limitations. Int J Environ Res Public Health. Sep 8, 2022;19(18):11278. [ CrossRef ] [ Medline ]
• Kaplan-Rakowski R, Meseberg K. Immersive media and their future. In: Branch RM, Lee H, Tseng S, editors. Educational Media and Technology Yearbook. Vol 42. Springer; 2019;143-153. [ CrossRef ]
• Kuliga SF, Thrash T, Dalton RC, Hölscher C. Virtual reality as an empirical research tool — exploring user experience in a real building and a corresponding virtual model. Comput Environ Urban Syst. Nov 2015;54:363-375. [ CrossRef ]
• Guan H, Xu Y, Zhao D. Application of virtual reality technology in clinical practice, teaching, and research in complementary and alternative medicine. Evid Based Complement Alternat Med. Aug 11, 2022;2022:1373170. [ CrossRef ] [ Medline ]
• Pottle J. Virtual reality and the transformation of medical education. Future Healthc J. Oct 2019;6(3):181-185. [ CrossRef ] [ Medline ]
• Satava RM, Jones SB. Current and future applications of virtual reality for medicine. Proc IEEE. Mar 1998;86(3):484-489. [ CrossRef ]
• Satava RM. Medical virtual reality. the current status of the future. Stud Health Technol Iform. 1996;29:100-106. [ Medline ]
• Rosenberg LB, Stredney D. A haptic interface for virtual simulation of endoscopic surgery. Stud Health Technol Inform. 1996;29:371-387. [ Medline ]
• Chytas D, Salmas M, Demesticha T, et al. A review of the use of virtual reality for teaching radiology in conjunction with anatomy. Cureus. Dec 5, 2021;13(12):e20174. [ CrossRef ] [ Medline ]
• O’Connor M, Stowe J, Potocnik J, Giannotti N, Murphy S, Rainford L. 3D virtual reality simulation in radiography education: the students' experience. Radiography (Lond). Feb 2021;27(1):208-214. [ CrossRef ] [ Medline ]
• Hart R, Karthigasu K. The benefits of virtual reality simulator training for laparoscopic surgery. Curr Opin Obstet Gynecol. Aug 2007;19(4):297-302. [ CrossRef ] [ Medline ]
• Thomas DJ, Singh D. Letter to the editor: virtual reality in surgical training. Int J Surg. May 2021;89:105935. [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372(71):n71. [ CrossRef ] [ Medline ]
• Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. Aug 28, 2019;366:l4898. [ CrossRef ] [ Medline ]
• McGuinness LA, Higgins JPT. Risk-of-bias visualization (robvis): an R package and shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. Jan 2021;12(1):55-61. [ CrossRef ] [ Medline ]
• Critical appraisal tools. JBI. URL: https://jbi.global/critical-appraisal-tools [Accessed 2024-04-25]
• Ahlqvist JB, Nilsson TA, Hedman LR, et al. A randomized controlled trial on 2 simulation-based training methods in radiology: effects on radiologic technology student skill in assessing image quality. Simul Healthc. Dec 2013;8(6):382-387. [ CrossRef ] [ Medline ]
• Bridge P, Gunn T, Kastanis L, et al. The development and evaluation of a medical imaging training immersive environment. J Med Radiat Sci. Sep 2014;61(3):159-165. [ CrossRef ] [ Medline ]
• Gunn T, Jones L, Bridge P, Rowntree P, Nissen L. The use of virtual reality simulation to improve technical skill in the undergraduate medical imaging student. Interactive Learning Environments. 2018;26(5):613-620. [ CrossRef ]
• Gunn T, Rowntree P, Starkey D, Nissen L. The use of virtual reality computed tomography simulation within a medical imaging and a radiation therapy undergraduate programme. J Med Radiat Sci. Mar 2021;68(1):28-36. [ CrossRef ] [ Medline ]
• Jensen J, Graumann O, Jensen RO, et al. Using virtual reality simulation for training practical skills in musculoskeletal wrist x-ray - a pilot study. J Clin Imaging Sci. Jul 11, 2023;13:20. [ CrossRef ] [ Medline ]
• Kato K, Kon D, Ito T, Ichikawa S, Ueda K, Kuroda Y. Radiography education with VR using head mounted display: proficiency evaluation by rubric method. BMC Med Educ. Jul 28, 2022;22(1):579. [ CrossRef ] [ Medline ]
• Nilsson TA, Hedman LR, Ahlqvist JB. A randomized trial of simulation-based versus conventional training of dental student skill at interpreting spatial information in radiographs. Simul Healthc. 2007;2(3):164-169. [ CrossRef ] [ Medline ]
• Nilsson TA, Hedman LR, Ahlqvist JB. Dental student skill retention eight months after simulator-supported training in oral radiology. J Dent Educ. May 2011;75(5):679-684. [ Medline ]
• O’Connor M, Rainford L. The impact of 3D virtual reality radiography practice on student performance in clinical practice. Radiography (Lond). Jan 2023;29(1):159-164. [ CrossRef ] [ Medline ]
• Rainford L, Tcacenco A, Potocnik J, et al. Student perceptions of the use of three-dimensional (3-D) virtual reality (VR) simulation in the delivery of radiation protection training for radiography and medical students. Radiography (Lond). Jul 2023;29(4):777-785. [ CrossRef ] [ Medline ]
• Rowe D, Garcia A, Rossi B. Comparison of virtual reality and physical simulation training in first-year radiography students in South America. J Med Radiat Sci. Jun 2023;70(2):120-126. [ CrossRef ] [ Medline ]
• Sapkaroski D, Baird M, McInerney J, Dimmock MR. The implementation of a haptic feedback virtual reality simulation clinic with dynamic patient interaction and communication for medical imaging students. J Med Radiat Sci. Sep 2018;65(3):218-225. [ CrossRef ] [ Medline ]
• Sapkaroski D, Baird M, Mundy M, Dimmock MR. Quantification of student radiographic patient positioning using an immersive virtual reality simulation. Simul Healthc. Aug 2019;14(4):258-263. [ CrossRef ] [ Medline ]
• Sapkaroski D, Mundy M, Dimmock MR. Virtual reality versus conventional clinical role-play for radiographic positioning training: a students' perception study. Radiography (Lond). Feb 2020;26(1):57-62. [ CrossRef ] [ Medline ]
• Shanahan M. Student perspective on using a virtual radiography simulation. Radiography (Lond). Aug 2016;22(3):217-222. [ CrossRef ]
• Wu Y, Mondal P, Stewart M, Ngo R, Burbridge B. Bringing radiology education to a new reality: a pilot study of using virtual reality as a remote educational tool. Can Assoc Radiol J. May 2023;74(2):251-263. [ CrossRef ] [ Medline ]
• Haddaway NR, Page MJ, Pritchard CC, McGuinness LA. PRISMA2020: an R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and open synthesis. Campbell Syst Rev. Mar 27, 2022;18(2):e1230. [ CrossRef ] [ Medline ]
• King S, Boyer J, Bell T, Estapa A. An automated virtual reality training system for teacher-student interaction: a randomized controlled trial. JMIR Serious Games. Dec 8, 2022;10(4):e41097. [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Hasan Sapci, Taiane de Azevedo Cardoso; submitted 23.09.23; peer-reviewed by FernandezHerrero Jorge, Stacey Kassutto; final revised version received 01.02.24; accepted 31.03.24; published 08.05.24.

© Shishir Shetty, Supriya Bhat, Saad Al Bayatti, Sausan Al Kawas, Wael Talaat, Mohamed El-Kishawi, Natheer Al Rawi, Sangeetha Narasimhan, Hiba Al-Daghestani, Medhini Madi, Raghavendra Shetty. Originally published in JMIR Medical Education (https://mededu.jmir.org), 8.5.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/ , as well as this copyright and license information must be included.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• NATURE INDEX
• 01 May 2024

Plagiarism in peer-review reports could be the ‘tip of the iceberg’

• Jackson Ryan 0

Jackson Ryan is a freelance science journalist in Sydney, Australia.

You can also search for this author in PubMed   Google Scholar

Time pressures and a lack of confidence could be prompting reviewers to plagiarize text in their reports. Credit: Thomas Reimer/Zoonar via Alamy

Mikołaj Piniewski is a researcher to whom PhD students and collaborators turn when they need to revise or refine a manuscript. The hydrologist, at the Warsaw University of Life Sciences, has a keen eye for problems in text — a skill that came in handy last year when he encountered some suspicious writing in peer-review reports of his own paper.

Last May, when Piniewski was reading the peer-review feedback that he and his co-authors had received for a manuscript they’d submitted to an environmental-science journal, alarm bells started ringing in his head. Comments by two of the three reviewers were vague and lacked substance, so Piniewski decided to run a Google search, looking at specific phrases and quotes the reviewers had used.

To his surprise, he found the comments were identical to those that were already available on the Internet, in multiple open-access review reports from publishers such as MDPI and PLOS. “I was speechless,” says Piniewski. The revelation caused him to go back to another manuscript that he had submitted a few months earlier, and dig out the peer-review reports he received for that. He found more plagiarized text. After e-mailing several collaborators, he assembled a team to dig deeper.

Meet this super-spotter of duplicated images in science papers

The team published the results of its investigation in Scientometrics in February 1 , examining dozens of cases of apparent plagiarism in peer-review reports, identifying the use of identical phrases across reports prepared for 19 journals. The team discovered exact quotes duplicated across 50 publications, saying that the findings are just “the tip of the iceberg” when it comes to misconduct in the peer-review system.

Dorothy Bishop, a former neuroscientist at the University of Oxford, UK, who has turned her attention to investigating research misconduct, was “favourably impressed” by the team’s analysis. “I felt the way they approached it was quite useful and might be a guide for other people trying to pin this stuff down,” she says.

Peer review under review

Piniewski and his colleagues conducted three analyses. First, they uploaded five peer-review reports from the two manuscripts that his laboratory had submitted to a rudimentary online plagiarism-detection tool . The reports had 44–100% similarity to previously published online content. Links were provided to the sources in which duplications were found.

The researchers drilled down further. They broke one of the suspicious peer-review reports down to fragments of one to three sentences each and searched for them on Google. In seconds, the search engine returned a number of hits: the exact phrases appeared in 22 open peer-review reports, published between 2021 and 2023.

The final analysis provided the most worrying results. They took a single quote — 43 words long and featuring multiple language errors, including incorrect capitalization — and pasted it into Google. The search revealed that the quote, or variants of it, had been used in 50 peer-review reports.

Predominantly, these reports were from journals published by MDPI, PLOS and Elsevier, and the team found that the amount of duplication increased year-on-year between 2021 and 2023. Whether this is because of an increase in the number of open-access peer-review reports during this time or an indication of a growing problem is unclear — but Piniewski thinks that it could be a little bit of both.

Why would a peer reviewer use plagiarized text in their report? The team says that some might be attempting to save time , whereas others could be motivated by a lack of confidence in their writing ability, for example, if they aren’t fluent in English.

The team notes that there are instances that might not represent misconduct. “A tolerable rephrasing of your own words from a different review? I think that’s fine,” says Piniewski. “But I imagine that most of these cases we found are actually something else.”

The source of the problem

Duplication and manipulation of peer-review reports is not a new phenomenon. “I think it’s now increasingly recognized that the manipulation of the peer-review process, which was recognized around 2010, was probably an indication of paper mills operating at that point,” says Jennifer Byrne, director of biobanking at New South Wales Health in Sydney, Australia, who also studies research integrity in scientific literature.

Paper mills — organizations that churn out fake research papers and sell authorships to turn a profit — have been known to tamper with reviews to push manuscripts through to publication, says Byrne.

The fight against fake-paper factories that churn out sham science

However, when Bishop looked at Piniewski’s case, she could not find any overt evidence of paper-mill activity. Rather, she suspects that journal editors might be involved in cases of peer-review-report duplication and suggests studying the track records of those who’ve allowed inadequate or plagiarized reports to proliferate.

Piniewski’s team is also concerned about the rise of duplications as generative artificial intelligence (AI) becomes easier to access . Although his team didn’t look for signs of AI use, its ability to quickly ingest and rephrase large swathes of text is seen as an emerging issue.

A preprint posted in March 2 showed evidence of researchers using AI chatbots to assist with peer review, identifying specific adjectives that could be hallmarks of AI-written text in peer-review reports .

Bishop isn’t as concerned as Piniewski about AI-generated reports, saying that it’s easy to distinguish between AI-generated text and legitimate reviewer commentary. “The beautiful thing about peer review,” she says, is that it is “one thing you couldn’t do a credible job with AI”.

Preventing plagiarism

Publishers seem to be taking action. Bethany Baker, a media-relations manager at PLOS, who is based in Cambridge, UK, told Nature Index that the PLOS Publication Ethics team “is investigating the concerns raised in the Scientometrics article about potential plagiarism in peer reviews”.

How big is science’s fake-paper problem?

An Elsevier representative told Nature Index that the publisher “can confirm that this matter has been brought to our attention and we are conducting an investigation”.

In a statement, the MDPI Research Integrity and Publication Ethics Team said that it has been made aware of potential misconduct by reviewers in its journals and is “actively addressing and investigating this issue”. It did not confirm whether this was related to the Scientometrics article.

One proposed solution to the problem is ensuring that all submitted reviews are checked using plagiarism-detection software. In 2022, exploratory work by Adam Day, a data scientist at Sage Publications, based in Thousand Oaks, California, identified duplicated text in peer-review reports that might be suggestive of paper-mill activity. Day offered a similar solution of using anti-plagiarism software , such as Turnitin.

Piniewski expects the problem to get worse in the coming years, but he hasn’t received any unusual peer-review reports since those that originally sparked his research. Still, he says that he’s now even more vigilant. “If something unusual occurs, I will spot it.”

doi: https://doi.org/10.1038/d41586-024-01312-0

Piniewski, M., Jarić, I., Koutsoyiannis, D. & Kundzewicz, Z. W. Scientometrics https://doi.org/10.1007/s11192-024-04960-1 (2024).

Article   Google Scholar  

Liang, W. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2403.07183 (2024).

Download references

Related Articles

• Peer review
• Research management

Illuminating ‘the ugly side of science’: fresh incentives for reporting negative results

Career Feature 08 MAY 24

Algorithm ranks peer reviewers by reputation — but critics warn of bias

Nature Index 25 APR 24

Researchers want a ‘nutrition label’ for academic-paper facts

Nature Index 17 APR 24

Structure peer review to make it more robust

World View 16 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Mount Etna’s spectacular smoke rings and more — April’s best science images

News 03 MAY 24

How reliable is this research? Tool flags papers discussed on PubPeer

News 29 APR 24

W3-Professorship (with tenure) in Inorganic Chemistry

The Institute of Inorganic Chemistry in the Faculty of Mathematics and Natural Sciences at the University of Bonn invites applications for a W3-Pro...

53113, Zentrum (DE)

Rheinische Friedrich-Wilhelms-Universität

Principal Investigator Positions at the Chinese Institutes for Medical Research, Beijing

Studies of mechanisms of human diseases, drug discovery, biomedical engineering, public health and relevant interdisciplinary fields.

Beijing, China

The Chinese Institutes for Medical Research (CIMR), Beijing

Research Associate - Neural Development Disorders

Houston, Texas (US)

Baylor College of Medicine (BCM)

Staff Scientist - Mitochondria and Surgery

Recruitment of talent positions at shengjing hospital of china medical university.

Call for top experts and scholars in the field of science and technology.

Shenyang, Liaoning, China

Shengjing Hospital of China Medical University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies