Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

  • Search for relevant literature
  • Evaluate sources
  • Identify themes, debates, and gaps
  • Outline the structure
  • Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

upload-your-document-ai-proofreader

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

  • Quick Run-through
  • Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

  • Demonstrate your familiarity with the topic and its scholarly context
  • Develop a theoretical framework and methodology for your research
  • Position your work in relation to other researchers and theorists
  • Show how your research addresses a gap or contributes to a debate
  • Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Literature review guide

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

published literature review

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

  • Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
  • Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
  • Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
  • Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

  • Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
  • Body image, self-perception, self-esteem, mental health
  • Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

  • Your university’s library catalogue
  • Google Scholar
  • Project Muse (humanities and social sciences)
  • Medline (life sciences and biomedicine)
  • EconLit (economics)
  • Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

  • What question or problem is the author addressing?
  • What are the key concepts and how are they defined?
  • What are the key theories, models, and methods?
  • Does the research use established frameworks or take an innovative approach?
  • What are the results and conclusions of the study?
  • How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
  • What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

  • Academic style
  • Vague sentences
  • Style consistency

See an example

published literature review

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

  • Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
  • Themes: what questions or concepts recur across the literature?
  • Debates, conflicts and contradictions: where do sources disagree?
  • Pivotal publications: are there any influential theories or studies that changed the direction of the field?
  • Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

  • Most research has focused on young women.
  • There is an increasing interest in the visual aspects of social media.
  • But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

  • Look at what results have emerged in qualitative versus quantitative research
  • Discuss how the topic has been approached by empirical versus theoretical scholarship
  • Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

  • Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
  • Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
  • Critically evaluate: mention the strengths and weaknesses of your sources
  • Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

  • Sampling methods
  • Simple random sampling
  • Stratified sampling
  • Cluster sampling
  • Likert scales
  • Reproducibility

 Statistics

  • Null hypothesis
  • Statistical power
  • Probability distribution
  • Effect size
  • Poisson distribution

Research bias

  • Optimism bias
  • Cognitive bias
  • Implicit bias
  • Hawthorne effect
  • Anchoring bias
  • Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

  • To familiarize yourself with the current state of knowledge on your topic
  • To ensure that you’re not just repeating what others have already done
  • To identify gaps in knowledge and unresolved problems that your research can address
  • To develop your theoretical framework and methodology
  • To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved February 12, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

OWL logo

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

  • An introductory paragraph that explains what your working topic and thesis is
  • A forecast of key topics or texts that will appear in the review
  • Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
  • Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
  • Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
  • Critically Evaluate: Mention the strengths and weaknesses of your sources
  • Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

  • Summarize the key findings you have taken from the literature and emphasize their significance
  • Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

  • Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
  • Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
  • Qualitative versus quantitative research
  • Empirical versus theoretical scholarship
  • Divide the research by sociological, historical, or cultural sources
  • Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

  • It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
  • Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
  • Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
  • Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

The Writing Center • University of North Carolina at Chapel Hill

Literature Reviews

What this handout is about.

This handout will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.

Introduction

OK. You’ve got to write a literature review. You dust off a novel and a book of poetry, settle down in your chair, and get ready to issue a “thumbs up” or “thumbs down” as you leaf through the pages. “Literature review” done. Right?

Wrong! The “literature” of a literature review refers to any collection of materials on a topic, not necessarily the great literary texts of the world. “Literature” could be anything from a set of government pamphlets on British colonial methods in Africa to scholarly articles on the treatment of a torn ACL. And a review does not necessarily mean that your reader wants you to give your personal opinion on whether or not you liked these sources.

What is a literature review, then?

A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period.

A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information. It might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. And depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

But how is a literature review different from an academic research paper?

The main focus of an academic research paper is to develop a new argument, and a research paper is likely to contain a literature review as one of its parts. In a research paper, you use the literature as a foundation and as support for a new insight that you contribute. The focus of a literature review, however, is to summarize and synthesize the arguments and ideas of others without adding new contributions.

Why do we write literature reviews?

Literature reviews provide you with a handy guide to a particular topic. If you have limited time to conduct research, literature reviews can give you an overview or act as a stepping stone. For professionals, they are useful reports that keep them up to date with what is current in the field. For scholars, the depth and breadth of the literature review emphasizes the credibility of the writer in his or her field. Literature reviews also provide a solid background for a research paper’s investigation. Comprehensive knowledge of the literature of the field is essential to most research papers.

Who writes these things, anyway?

Literature reviews are written occasionally in the humanities, but mostly in the sciences and social sciences; in experiment and lab reports, they constitute a section of the paper. Sometimes a literature review is written as a paper in itself.

Let’s get to it! What should I do before writing the literature review?

If your assignment is not very specific, seek clarification from your instructor:

  • Roughly how many sources should you include?
  • What types of sources (books, journal articles, websites)?
  • Should you summarize, synthesize, or critique your sources by discussing a common theme or issue?
  • Should you evaluate your sources?
  • Should you provide subheadings and other background information, such as definitions and/or a history?

Find models

Look for other literature reviews in your area of interest or in the discipline and read them to get a sense of the types of themes you might want to look for in your own research or ways to organize your final review. You can simply put the word “review” in your search engine along with your other topic terms to find articles of this type on the Internet or in an electronic database. The bibliography or reference section of sources you’ve already read are also excellent entry points into your own research.

Narrow your topic

There are hundreds or even thousands of articles and books on most areas of study. The narrower your topic, the easier it will be to limit the number of sources you need to read in order to get a good survey of the material. Your instructor will probably not expect you to read everything that’s out there on the topic, but you’ll make your job easier if you first limit your scope.

Keep in mind that UNC Libraries have research guides and to databases relevant to many fields of study. You can reach out to the subject librarian for a consultation: https://library.unc.edu/support/consultations/ .

And don’t forget to tap into your professor’s (or other professors’) knowledge in the field. Ask your professor questions such as: “If you had to read only one book from the 90’s on topic X, what would it be?” Questions such as this help you to find and determine quickly the most seminal pieces in the field.

Consider whether your sources are current

Some disciplines require that you use information that is as current as possible. In the sciences, for instance, treatments for medical problems are constantly changing according to the latest studies. Information even two years old could be obsolete. However, if you are writing a review in the humanities, history, or social sciences, a survey of the history of the literature may be what is needed, because what is important is how perspectives have changed through the years or within a certain time period. Try sorting through some other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to consider what is currently of interest to scholars in this field and what is not.

Strategies for writing the literature review

Find a focus.

A literature review, like a term paper, is usually organized around ideas, not the sources themselves as an annotated bibliography would be organized. This means that you will not just simply list your sources and go into detail about each one of them, one at a time. No. As you read widely but selectively in your topic area, consider instead what themes or issues connect your sources together. Do they present one or different solutions? Is there an aspect of the field that is missing? How well do they present the material and do they portray it according to an appropriate theory? Do they reveal a trend in the field? A raging debate? Pick one of these themes to focus the organization of your review.

Convey it to your reader

A literature review may not have a traditional thesis statement (one that makes an argument), but you do need to tell readers what to expect. Try writing a simple statement that lets the reader know what is your main organizing principle. Here are a couple of examples:

The current trend in treatment for congestive heart failure combines surgery and medicine. More and more cultural studies scholars are accepting popular media as a subject worthy of academic consideration.

Consider organization

You’ve got a focus, and you’ve stated it clearly and directly. Now what is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? And in what order should you present them? Develop an organization for your review at both a global and local level:

First, cover the basic categories

Just like most academic papers, literature reviews also must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper. The following provides a brief description of the content of each:

  • Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern.
  • Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).
  • Conclusions/Recommendations: Discuss what you have drawn from reviewing literature so far. Where might the discussion proceed?

Organizing the body

Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further.

To help you come up with an overall organizational framework for your review, consider the following scenario:

You’ve decided to focus your literature review on materials dealing with sperm whales. This is because you’ve just finished reading Moby Dick, and you wonder if that whale’s portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 1980’s. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 1968 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick, so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel.

Now consider some typical ways of organizing the sources into a review:

  • Chronological: If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (1968), and finally the biology articles (1980s) and the recent articles on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus.
  • By publication: Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies.
  • By trend: A better way to organize the above sources chronologically is to examine the sources under another trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Under this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote a century apart.
  • Thematic: Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For instance, the sperm whale review could focus on the development of the harpoon for whale hunting. While the study focuses on one topic, harpoon technology, it will still be organized chronologically. The only difference here between a “chronological” and a “thematic” approach is what is emphasized the most: the development of the harpoon or the harpoon technology.But more authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as “evil” in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.
  • Methodological: A methodological approach differs from the two above in that the focusing factor usually does not have to do with the content of the material. Instead, it focuses on the “methods” of the researcher or writer. For the sperm whale project, one methodological approach would be to look at cultural differences between the portrayal of whales in American, British, and French art work. Or the review might focus on the economic impact of whaling on a community. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed. Once you’ve decided on the organizational method for the body of the review, the sections you need to include in the paper should be easy to figure out. They should arise out of your organizational strategy. In other words, a chronological review would have subsections for each vital time period. A thematic review would have subtopics based upon factors that relate to the theme or issue.

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

  • Current Situation: Information necessary to understand the topic or focus of the literature review.
  • History: The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
  • Methods and/or Standards: The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

Begin composing

Once you’ve settled on a general pattern of organization, you’re ready to write each section. There are a few guidelines you should follow during the writing stage as well. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as “writer,” “pedestrian,” and “persons.” The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine “generic” condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, “Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense,” Women and Language19:2).

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review’s focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though, if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton’s study, but then synthesize it by rephrasing the study’s significance and relating it to their own work.

Keep your own voice

While the literature review presents others’ ideas, your voice (the writer’s) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author’s information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil’s. For more information, please see our handout on plagiarism .

Revise, revise, revise

Draft in hand? Now you’re ready to revise. Spending a lot of time revising is a wise idea, because your main objective is to present the material, not the argument. So check over your review again to make sure it follows the assignment and/or your outline. Then, just as you would for most other academic forms of writing, rewrite or rework the language of your review so that you’ve presented your information in the most concise manner possible. Be sure to use terminology familiar to your audience; get rid of unnecessary jargon or slang. Finally, double check that you’ve documented your sources and formatted the review appropriately for your discipline. For tips on the revising and editing process, see our handout on revising drafts .

Works consulted

We consulted these works while writing this handout. This is not a comprehensive list of resources on the handout’s topic, and we encourage you to do your own research to find additional publications. Please do not use this list as a model for the format of your own reference list, as it may not match the citation style you are using. For guidance on formatting citations, please see the UNC Libraries citation tutorial . We revise these tips periodically and welcome feedback.

Anson, Chris M., and Robert A. Schwegler. 2010. The Longman Handbook for Writers and Readers , 6th ed. New York: Longman.

Jones, Robert, Patrick Bizzaro, and Cynthia Selfe. 1997. The Harcourt Brace Guide to Writing in the Disciplines . New York: Harcourt Brace.

Lamb, Sandra E. 1998. How to Write It: A Complete Guide to Everything You’ll Ever Write . Berkeley: Ten Speed Press.

Rosen, Leonard J., and Laurence Behrens. 2003. The Allyn & Bacon Handbook , 5th ed. New York: Longman.

Troyka, Lynn Quittman, and Doug Hesse. 2016. Simon and Schuster Handbook for Writers , 11th ed. London: Pearson.

You may reproduce it for non-commercial use if you use the entire handout and attribute the source: The Writing Center, University of North Carolina at Chapel Hill

Make a Gift

University of Texas

  • University of Texas Libraries

Literature Reviews

  • What is a literature review?
  • Steps in the Literature Review Process
  • Define your research question
  • Determine inclusion and exclusion criteria
  • Choose databases and search
  • Review Results
  • Synthesize Results
  • Analyze Results
  • Librarian Support

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

  • Provide a comprehensive and updated review of the literature;
  • Explain why this review has taken place;
  • Articulate a position or hypothesis;
  • Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

  • Demonstrate how a study fills a gap in research
  • Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

  • Organizes or describes a topic
  • Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

  • It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
  • It may be too focused. Some niche studies may miss the bigger picture.
  • It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
  • It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

  • October 26, 2022 recording
  • Last Updated: Oct 26, 2022 2:49 PM
  • URL: https://guides.lib.utexas.edu/literaturereviews

Creative Commons License

  • UConn Library
  • Literature Review: The What, Why and How-to Guide
  • Introduction

Literature Review: The What, Why and How-to Guide — Introduction

  • Getting Started
  • How to Pick a Topic
  • Strategies to Find Sources
  • Evaluating Sources & Lit. Reviews
  • Tips for Writing Literature Reviews
  • Writing Literature Review: Useful Sites
  • Citation Resources
  • Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

  • To develop a theory or evaluate an existing theory
  • To summarize the historical or existing state of a research topic
  • Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

  • A research paper assigned in a course
  • A thesis or dissertation
  • A grant proposal
  • An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

  • Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

  • Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

  • Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

  • Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

  • UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
  • << Previous: Getting Started
  • Next: How to Pick a Topic >>
  • Last Updated: Sep 21, 2022 2:16 PM
  • URL: https://guides.lib.uconn.edu/literaturereview

Creative Commons

Benedictine University Library

Literature Review: Examples of Published Literature Reviews

  • Sample Searches
  • Examples of Published Literature Reviews
  • Researching Your Topic
  • Subject Searching
  • Google Scholar
  • Track Your Work
  • Citation Managers This link opens in a new window
  • Citation Guides This link opens in a new window
  • Tips on Writing Your Literature Review This link opens in a new window
  • Research Help

Ask a Librarian

Chat with a Librarian

Lisle: (630) 829-6057 Mesa: (480) 878-7514 Toll Free: (877) 575-6050 Email: [email protected]

Book a Research Consultation Library Hours

Facebook

  • Example of a Literature Review in History
  • Comparing the Annotated Bibliography to the Literature Review
  • Literature Review Guidelines Literature Review (Historiographic Essay): Making sense of what has been written on your topic.

To find examples of published literature reviews in your field or niche, try searching ProQuest Dissertations and Theses by keyword, advisor, or subject. 

  • Dissertations & Theses Theses and dissertations from around the world covering all areas of study.
  • << Previous: Sample Searches
  • Next: Researching Your Topic >>
  • Last Updated: Feb 27, 2023 2:09 PM
  • URL: https://researchguides.ben.edu/lit-review

Kindlon Hall 5700 College Rd. Lisle, IL 60532 (630) 829-6050

Gillett Hall 225 E. Main St. Mesa, AZ 85201 (480) 878-7514

Instagram

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • CAREER FEATURE
  • 04 December 2020
  • Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

published literature review

  • Research management

A researcher-exchange programme made me a better doctor at home and abroad

A researcher-exchange programme made me a better doctor at home and abroad

Career Q&A 12 FEB 24

I took my case to Nepal’s highest court to improve conservation

I took my case to Nepal’s highest court to improve conservation

Postdocs celebrate 24% pay boost in one of the world’s most expensive cities

Postdocs celebrate 24% pay boost in one of the world’s most expensive cities

Career News 12 FEB 24

China conducts first nationwide review of retractions and research misconduct

China conducts first nationwide review of retractions and research misconduct

News 12 FEB 24

Could roving researchers help address the challenge of taking parental leave?

Could roving researchers help address the challenge of taking parental leave?

Career Feature 07 FEB 24

Best practice for LGBTQ+ data collection by STEM organizations

Correspondence 06 FEB 24

Open-access publishing: citation advantage is unproven

Correspondence 13 FEB 24

How journals are fighting back against a wave of questionable images

How journals are fighting back against a wave of questionable images

Postdoctoral Fellow (PhD)

Houston, Texas (US)

Baylor College of Medicine (BCM)

published literature review

Professional-Track Faculty Positions Available!

VGTI is seeking professional-track faculty candidates with demonstrated potential for creative collaborations in infectious disease.

Beaverton, Oregon

Vaccine & Gene Therapy Institute, Oregon Health & Science University

published literature review

Postdoctoral Fellow in Cancer Biology

Postdoctoral position in cancer biology is available to carry out projects focused on studying the effects of small molecules in cancer.

Ann Arbor, Michigan

Jolanta Grembecka, PhD, Professor

published literature review

Applied metabolomics/metabolomics informatics scientist.

The Department of Biomedical Informatics (BMI, https://bmi.osu.edu) and the Foods for Health Discovery Theme (FFH, http://foodsforhealth.osu.edu) a...

Columbus, Ohio

The Ohio State University

Tenure-track faculty in Bioinformatics

The Department of Biomedical Informatics (BMI, http://bmi.osu.edu) at The Ohio State University (OSU) currently has opportunities for tenure-track ...

Columbus, OH

published literature review

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

  • Knowledge Base
  • Dissertation
  • What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

  • Search for relevant literature
  • Evaluate sources
  • Identify themes, debates and gaps
  • Outline the structure
  • Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

upload-your-document-ai-proofreader

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

  • Quick Run-through
  • Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

  • Demonstrate your familiarity with the topic and scholarly context
  • Develop a theoretical framework and methodology for your research
  • Position yourself in relation to other researchers and theorists
  • Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

Literature review guide

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

published literature review

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

  • Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
  • Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
  • Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
  • Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

  • Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
  • Body image, self-perception, self-esteem, mental health
  • Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

  • Your university’s library catalogue
  • Google Scholar
  • Project Muse (humanities and social sciences)
  • Medline (life sciences and biomedicine)
  • EconLit (economics)
  • Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

  • What question or problem is the author addressing?
  • What are the key concepts and how are they defined?
  • What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
  • What are the results and conclusions of the study?
  • How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
  • How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
  • What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

  • Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
  • Themes: what questions or concepts recur across the literature?
  • Debates, conflicts and contradictions: where do sources disagree?
  • Pivotal publications: are there any influential theories or studies that changed the direction of the field?
  • Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

  • Most research has focused on young women.
  • There is an increasing interest in the visual aspects of social media.
  • But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

  • Look at what results have emerged in qualitative versus quantitative research
  • Discuss how the topic has been approached by empirical versus theoretical scholarship
  • Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

  • Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
  • Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
  • Critically evaluate: mention the strengths and weaknesses of your sources.
  • Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

  • To familiarise yourself with the current state of knowledge on your topic
  • To ensure that you’re not just repeating what others have already done
  • To identify gaps in knowledge and unresolved problems that your research can address
  • To develop your theoretical framework and methodology
  • To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 12 February 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Harvey Cushing/John Hay Whitney Medical Library

  • Collections
  • Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

  • Biomedical Databases
  • Global (Public Health) Databases
  • Soc. Sci., History, and Law Databases
  • Grey Literature
  • Trials Registers
  • Data and Statistics
  • Public Policy
  • Google Tips
  • Recommended Books
  • Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

  • Explains the background of research on a topic.
  • Demonstrates why a topic is significant to a subject area.
  • Discovers relationships between research studies/ideas.
  • Identifies major themes, concepts, and researchers on a topic.
  • Identifies critical gaps and points of disagreement.
  • Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

Cover Art

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

  • Make sure your research question is not too broad or too narrow.  Is it manageable?
  • Begin writing down terms that are related to your question. These will be useful for searches later.
  • If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

  • This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

  • use the tabs on this guide
  • Find other databases in the Nursing Information Resources web page
  • More on the Medical Library web page
  • ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

  • Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
  • Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
  • Review the abstracts of research studies carefully. This will save you time.
  • Use the bibliographies and references of research studies you find to locate others.
  • Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
  • Ask your librarian for help at any time.
  • Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

  • What was the research question of the study you are reviewing? What were the authors trying to discover?
  • Was the research funded by a source that could influence the findings?
  • What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
  • Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
  • If there are conflicting studies, why do you think that is?
  • How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

  • Review the abstracts carefully.  
  • Keep careful notes so that you may track your thought processes during the research process.
  • Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
  • << Previous: Recommended Books
  • Last Updated: Jan 4, 2024 10:52 AM
  • URL: https://guides.library.yale.edu/YSNDoctoral
  • USC Libraries
  • Research Guides

Organizing Your Social Sciences Research Paper

  • 5. The Literature Review
  • Purpose of Guide
  • Design Flaws to Avoid
  • Independent and Dependent Variables
  • Glossary of Research Terms
  • Reading Research Effectively
  • Narrowing a Topic Idea
  • Broadening a Topic Idea
  • Extending the Timeliness of a Topic Idea
  • Academic Writing Style
  • Choosing a Title
  • Making an Outline
  • Paragraph Development
  • Research Process Video Series
  • Executive Summary
  • The C.A.R.S. Model
  • Background Information
  • The Research Problem/Question
  • Theoretical Framework
  • Citation Tracking
  • Content Alert Services
  • Evaluating Sources
  • Primary Sources
  • Secondary Sources
  • Tiertiary Sources
  • Scholarly vs. Popular Publications
  • Qualitative Methods
  • Quantitative Methods
  • Insiderness
  • Using Non-Textual Elements
  • Limitations of the Study
  • Common Grammar Mistakes
  • Writing Concisely
  • Avoiding Plagiarism
  • Footnotes or Endnotes?
  • Further Readings
  • Generative AI and Writing
  • USC Libraries Tutorials and Other Guides
  • Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

  • Give a new interpretation of old material or combine new with old interpretations,
  • Trace the intellectual progression of the field, including major debates,
  • Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
  • Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

  • Place each work in the context of its contribution to understanding the research problem being studied.
  • Describe the relationship of each work to the others under consideration.
  • Identify new ways to interpret prior research.
  • Reveal any gaps that exist in the literature.
  • Resolve conflicts amongst seemingly contradictory previous studies.
  • Identify areas of prior scholarship to prevent duplication of effort.
  • Point the way in fulfilling a need for additional research.
  • Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

  • An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
  • Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
  • An explanation of how each work is similar to and how it varies from the others,
  • Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

  • Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
  • Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
  • Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
  • Persuasiveness -- which of the author's theses are most convincing or least convincing?
  • Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

  • Current Situation : Information necessary to understand the current topic or focus of the literature review.
  • Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
  • History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
  • Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
  • Standards : Description of the way in which you present your information.
  • Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

  • Sources in your literature review do not clearly relate to the research problem;
  • You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
  • Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
  • Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
  • Does not describe the search procedures that were used in identifying the literature to review;
  • Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
  • Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

  • How are they organizing their ideas?
  • What methods have they used to study the problem?
  • What theories have been used to explain, predict, or understand their research problem?
  • What sources have they cited to support their conclusions?
  • How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

  • Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
  • Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
  • Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

  • << Previous: Theoretical Framework
  • Next: Citation Tracking >>
  • Last Updated: Feb 8, 2024 1:57 PM
  • URL: https://libguides.usc.edu/writingguide

Literature Reviews

  • Tools & Visualizations
  • Literature Review Examples
  • Videos, Books & Links

Business & Econ Librarian

Profile Photo

Click to Chat with a Librarian

Text: (571) 248-7542

What is a literature review?

A literature review discusses published information in a particular subject area. Often part of the introduction to an essay, research report or thesis, the literature review is literally a "re" view or "look again" at what has already been written about the topic, wherein the author analyzes a segment of a published body of knowledge through summary, classification, and comparison of prior research studies, reviews of literature, and theoretical articles. Literature reviews provide the reader with a bibliographic history of the scholarly research in any given field of study. As such,  as new information becomes available, literature reviews grow in length or become focused on one specific aspect of the topic.

A literature review can be just a simple summary of the sources, but usually contains an organizational pattern and combines both summary and synthesis. A summary is a recap of the important information of the source, whereas a synthesis is a re-organization, or a reshuffling, of that information. The literature review might give a new interpretation of old material or combine new with old interpretations. Or it might trace the intellectual progression of the field, including major debates. Depending on the situation, the literature review may evaluate the sources and advise the reader on the most pertinent or relevant.

A literature review is NOT:

  • An annotated bibliography – a list of citations to books, articles and documents that includes a brief description and evaluation for each citation. The annotations inform the reader of the relevance, accuracy and quality of the sources cited.
  • A literary review – a critical discussion of the merits and weaknesses of a literary work.
  • A book review – a critical discussion of the merits and weaknesses of a particular book.
  • Teaching Information Literacy Reframed: 50+ Framework-Based Exercises for Creating Information-Literate Learners
  • The UNC Writing Center – Literature Reviews
  • The UW-Madison Writing Center: The Writer’s Handbook – Academic and Professional Writing – Learn How to Write a Literature Review

What is the difference between a literature review and a research paper?

The focus of a literature review is to summarize and synthesize the arguments and ideas of others without adding new contributions, whereas academic research papers present and develop new arguments that build upon the previously available body of literature.

How do I write a literature review?

There are many resources that offer step-by-step guidance for writing a literature review, and you can find some of them under Other Resources in the menu to the left. Writing the Literature Review: A Practical Guide suggests these steps:

  • Chose a review topic and develop a research question
  • Locate and organize research sources
  • Select, analyze and annotate sources
  • Evaluate research articles and other documents
  • Structure and organize the literature review
  • Develop arguments and supporting claims
  • Synthesize and interpret the literature
  • Put it all together

Cover Art

What is the purpose of writing a literature review?

Literature reviews serve as a guide to a particular topic: professionals can use literature reviews to keep current on their field; scholars can determine credibility of the writer in his or her field by analyzing the literature review.

As a writer, you will use the literature review to:

  • See what has, and what has not, been investigated about your topic
  • Identify data sources that other researches have used
  • Learn how others in the field have defined and measured key concepts
  • Establish context, or background, for the argument explored in the rest of a paper
  • Explain what the strengths and weaknesses of that knowledge and ideas might be
  • Contribute to the field by moving research forward
  • To keep the writer/reader up to date with current developments in a particular field of study
  • Develop alternative research projects
  • Put your work in perspective
  • Demonstrate your understanding and your ability to critically evaluate research in the field
  • Provide evidence that may support your own findings
  • Next: Tools & Visualizations >>
  • Last Updated: Sep 7, 2023 8:35 AM
  • URL: https://subjectguides.library.american.edu/literaturereview

The University of Edinburgh home

  • Schools & departments

Institute for Academic Development

Literature review

A general guide on how to conduct and write a literature review.

Please check course or programme information and materials provided by teaching staff , including your project supervisor, for subject-specific guidance.

What is a literature review?

A literature review is a piece of academic writing demonstrating knowledge and understanding of the academic literature on a specific topic placed in context.  A literature review also includes a critical evaluation of the material; this is why it is called a literature review rather than a literature report. It is a process of reviewing the literature, as well as a form of writing.

To illustrate the difference between reporting and reviewing, think about television or film review articles.  These articles include content such as a brief synopsis or the key points of the film or programme plus the critic’s own evaluation.  Similarly the two main objectives of a literature review are firstly the content covering existing research, theories and evidence, and secondly your own critical evaluation and discussion of this content. 

Usually a literature review forms a section or part of a dissertation, research project or long essay.  However, it can also be set and assessed as a standalone piece of work.

What is the purpose of a literature review?

…your task is to build an argument, not a library. Rudestam, K.E. and Newton, R.R. (1992) Surviving your dissertation: A comprehensive guide to content and process. California: Sage, p49.

In a larger piece of written work, such as a dissertation or project, a literature review is usually one of the first tasks carried out after deciding on a topic.  Reading combined with critical analysis can help to refine a topic and frame research questions.  Conducting a literature review establishes your familiarity with and understanding of current research in a particular field before carrying out a new investigation.  After doing a literature review, you should know what research has already been done and be able to identify what is unknown within your topic.

When doing and writing a literature review, it is good practice to:

  • summarise and analyse previous research and theories;
  • identify areas of controversy and contested claims;
  • highlight any gaps that may exist in research to date.

Conducting a literature review

Focusing on different aspects of your literature review can be useful to help plan, develop, refine and write it.  You can use and adapt the prompt questions in our worksheet below at different points in the process of researching and writing your review.  These are suggestions to get you thinking and writing.

Developing and refining your literature review (pdf)

Developing and refining your literature review (Word)

Developing and refining your literature review (Word rtf)

Writing a literature review has a lot in common with other assignment tasks.  There is advice on our other pages about thinking critically, reading strategies and academic writing.  Our literature review top tips suggest some specific things you can do to help you submit a successful review.

Literature review top tips (pdf)

Literature review top tips (Word rtf)

Our reading page includes strategies and advice on using books and articles and a notes record sheet grid you can use.

Reading at university

The Academic writing page suggests ways to organise and structure information from a range of sources and how you can develop your argument as you read and write.

Academic writing

The Critical thinking page has advice on how to be a more critical researcher and a form you can use to help you think and break down the stages of developing your argument.

Critical thinking

As with other forms of academic writing, your literature review needs to demonstrate good academic practice by following the Code of Student Conduct and acknowledging the work of others through citing and referencing your sources.  

Good academic practice

As with any writing task, you will need to review, edit and rewrite sections of your literature review.  The Editing and proofreading page includes tips on how to do this and strategies for standing back and thinking about your structure and checking the flow of your argument.

Editing and proofreading

Guidance on literature searching from the University Library

The Academic Support Librarians have developed LibSmart I and II, Learn courses to help you develop and enhance your digital research skills and capabilities; from getting started with the Library to managing data for your dissertation.

Searching using the library’s DiscoverEd tool: DiscoverEd

Finding resources in your subject: Subject guides

The Academic Support Librarians also provide one-to-one appointments to help you develop your research strategies.

1 to 1 support for literature searching and systematic reviews

Advice to help you optimise use of Google Scholar, Google Books and Google for your research and study: Using Google

Managing and curating your references

A referencing management tool can help you to collect and organise and your source material to produce a bibliography or reference list. 

Referencing and reference management

Information Services provide access to Cite them right online which is a guide to the main referencing systems and tells you how to reference just about any source (EASE log-in may be required).

Cite them right

Published study guides

There are a number of scholarship skills books and guides available which can help with writing a literature review.  Our Resource List of study skills guides includes sections on Referencing, Dissertation and project writing and Literature reviews.

Study skills guides

Purdue University

  • Ask a Librarian

Research: Overview & Approaches

  • Getting Started with Undergraduate Research
  • Planning & Getting Started
  • Building Your Knowledge Base
  • Locating Sources
  • Reading Scholarly Articles
  • Creating a Literature Review

Finding and Completing a Literature Review

Intro to creating a literature review.

  • Productivity & Organizing Research
  • Scholarly and Professional Relationships
  • Empirical Research
  • Interpretive Research
  • Action-Based Research
  • Creative & Experimental Approaches

  • Palgrave's Study Guide to Carrying Out a Literature Review Your research is seen as a contribution to knowledge in the field and it needs to indicate, therefore, that there is an awareness of what that knowledge comprises. Read this guide to getting started.
  • Purdue OWL's Guide to Writing a Literature Review A literature review requires the writer to perform extensive research on published work in one’s field in order to explain how one’s own work fits into the larger conversation regarding a particular topic. This task requires the writer to spend time reading, managing, and conveying information; the complexity of literature reviews can make this section one of the most challenging parts of writing about one’s research. This handout will provide some strategies for revising literature reviews.

Every time you conduct research, you will need to make it clear where you got your evidence from. This work of citing our sources is absolutely essential for a couple of reasons.

  • It demonstrates to the readers of our own research that we have evidence to back up our claims.
  • A complete and correct citation directs readers to the original source for them to verify our claims and learn more.
  • It gives credit to the researchers whose intellectual work helped form our own research.
  • << Previous: Reading Scholarly Articles
  • Next: Productivity & Organizing Research >>
  • Last Edited: Nov 10, 2023 3:32 PM
  • URL: https://guides.lib.purdue.edu/research_approaches

Want to publish a literature review? Think of it as an empirical paper

What to consider if you want to publish a literature review paper

Tatiana Andreeva - Fri 23 Apr 2021 07:50 (updated Fri 27 Oct 2023 17:26)

published literature review

[Guest post by CYGNA member Tatiana Andreeva ]

When you’ve been reading a lot on a particular topic – for example, reviewing the literature for your research project or for your PhD – at some point it looks like you have enough material and reflections to publish this piece of work as a separate paper. Recognize this? If you ever tried it, you might know that publishing a literature review paper in an academic journal is a tricky task. The literature review publications come in so many forms, and there is no single cheat-sheet or established format like for empirical papers that you could follow to ensure success in publication.

Through my own journey of trial-and-error on this path, as well as through reviewing for journals and for PhD students in my course, I came up with an idea that will help you to increase the chances of publishing a literature review: think of a literature review as simply another empirical research project. Think of it as an empirical study, in which your data comes not from your usual fieldwork but from the articles that you review.

Many literature reviews can be thought of as a qualitative empirical study, in which the papers included in the review substitute interviews or field observations that you would usually collect and code. Some literature reviews, e.g., meta-analyses, are more like a quantitative empirical paper, in which various numbers you extract from the papers in your dataset substitute your survey data.

Seeing literature review in this way has three important implications for how we think about our literature review, and how we can design it to increase its chances of being interesting to others - that is, of being published.

Start with a relevant research problem and an interesting research question

We learn early in our academic career that any empirical paper should have a clear research problem and a clear research question. We frequently hear from journal editors and reviewers that just having a gap in the literature, or the fact that something has not been researched before, are not good enough to justify doing yet another empirical study. They say: you need to have a problem that your study can address, and you need to have a question that we currently don’t have an answer to. Only then your empirical study can add value to existing research. 

When we think of a literature review as of an empirical study, just with the different type of data at hand, we realize that the very same rationale applies. From this perspective the arguments that I often see in literature reviews – that there is no literature review in this particular area or that the existing literature reviews are quite dated – are not sufficient in the journal’s eyes to justify the publication of a literature review on a topic. If you aim to publish your literature review, start by thinking – what is the problem I would like to address? What would be my research question about this problem, that other readers would find interesting?

Design a methodologically-sound data collection and analysis protocol

When we think of any empirical study, we know that if we want to have reliable findings that will be accepted by our peers as trustworthy, we need to follow a transparent and well-thought data collection protocol. We also need to carefully choose and correctly apply relevant data analysis method. This goes without saying, right?

The same applies to the literature review! If we want our readers to trust our conclusions from the literature review, we need to make sure that the data we collect speaks to our research question, is of good quality, representative of the field, etc. The growing attention in business and management field to the systematic approach to literature reviews (Denyer & Tranfield, 2009; Rojon et al., 2021) reflects the rising expectations of the quality of the data used in literature review papers. Indeed, this approach offers exactly that: a clear data collection protocol, transparently communicated, so that someone else could replicate your study. For example, do the very same thing in 10 years and see how thinking on the topic has changed.

published literature review

In the literature on doing literature reviews you will read that systematic literature review is only one of the types of literature reviews. Yet all recommendations on doing different types of the literature reviews share the idea that the data that you base your conclusions on has to be collected in a rigorous and transparent way (e.g., Callahan, 2014).  In this post you can find more references on how to ensure that your literature review “data collection” protocol meets the quality expectations.

So now you have all the papers you have carefully selected, how do you go about analysing them, so that peer academics would recognize your conclusions as reliable and robust? This is the trickiest part, and we have limited methodological advice published on this. In this post I’ve mentioned some papers that discuss specific methods of literature analysis. For example, I found that a sophisticated coding rubric leveraged our literature analysis to a different level (Sergeeva & Andreeva, 2016), but must acknowledge that developing this rubric was one of the most challenging tasks of this review paper. In O’Higgins et al. (forthcoming) we used a combination of qualitative content analysis with Pearson’s chi-squared (χ²) goodness of fit test in order to validate some of our conclusions. The trick is - as with any empirical study - your choice of the analytical method needs to fit with your research question. In sum, the message is: choose your method for analysis of the selected literature carefully, apply it rigorously, and explain it transparently.

Think of the theoretical contribution beyond description of the findings

When we think of our usual empirical work, be it qualitative or quantitative, we are well-aware that just the description of our data wouldn’t do. We know that we need to leverage what our data shows to explain how it informs the broader theory, how it compares to previous studies, what is new that we see from this data?

Again, the same logic applies to the literature reviews. In practice though, we often find it difficult to apply this advice to our literature review papers, because the description of the field in itself seems to be novel, especially if nobody did such a review before. In my experience, this argument does not persuade editors and reviewers of the journals, and often rightfully so.

published literature review

For example, think of a typical quantitative empirical paper: a descriptive statistics table must be provided, but no one would claim a contribution based on it, right? Cropanzano (2009:1306-1307) offers a good exercise that explains why reviewers often don’t buy the description of the field as a novel contribution. He suggests: imagine somebody who read all the primary articles in your dataset, would they still learn anything from your literature review? And if the answer is “no”, then it’s likely that your review paper doesn’t have yet the level of contribution that is needed to turn it into a publication.

I think this exercise can also help to stimulate your thinking of what a theoretical contribution of your literature review could be. For example, think – what it is that I see in this literature that others are not likely to see?  In this blogpost you can find some papers that offer insights on how to leverage your literature review to have a theoretical contribution.  

Callahan, J.L. (2014). Writing literature reviews: A reprise and update. Human Resource Development Review , 13(3), 271–275. https://doi.org/10.1177/1534484314536705

Cropanzano, R. (2009). Writing nonempirical articles for Journal of Management: General thoughts and suggestions. Journal of Management , 35(6), 1304–1311. https://doi.org/10.1177/0149206309344118

Denyer, D., Tranfield, D. (2009). Producing a systematic review. In Buchanan, D., Bryman, A. (Eds.), The Sage handbook of organizational research methods (pp. 671–689). London, UK: Sage.

O’Higgins, C., Andreeva, T., Aramburu, N. (forthcoming). International management challenges of professional service firms: a synthesis of the literature. Review of International Business and Strategy.

Rojon, C., Okupe, A., McDowall, A. (2021). Utilization and development of systematic reviews in management research: What do we know and where do we go from here? International Journal of Management Reviews, 1– 33. https://doi.org/10.1111/ijmr.12245

Sergeeva, A., Andreeva, T. (2016). Knowledge sharing: bringing the context back in, Journal of Management Inquiry , 25, 240-261. https://doi.org/10.1177/1056492615618271

Related blogposts

  • Resources on doing a literature review
  • Do you really want to publish your literature review? Advice for PhD students
  • How to keep up-to-date with the literature, but avoid information overload?
  • Is a literature review publication a low-cost project?
  • Using Publish or Perish to do a literature review
  • How to conduct a longitudinal literature review?
  • New: Publish or Perish now also exports abstracts

Find the resources on my website useful?

I cover all the expenses of operating my website privately. If you enjoyed this post and want to support me in maintaining my website, consider buying a copy of one of my books (see below) or  supporting the Publish or Perish software .

A story about perseverance and publication failures

Copyright © 2023 Tatiana Andreeva . All rights reserved. Page last modified on Fri 27 Oct 2023 17:26

published literature review

Tatiana Andreeva's profile and contact details >>

Book cover

Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches pp 503–527 Cite as

Publishing Literature Reviews

  • Rob Dekkers 4 ,
  • Lindsey Carey 5 &
  • Peter Langhorne 6  
  • First Online: 11 August 2022

1332 Accesses

1 Citations

With the credo of academics being ‘publish or perish’, much attention is given to publishing output of studies, which includes literature reviews. The foundation for conducting literature reviews has been laid in the previous chapters from setting the topic in Chapter 4 to the analysis and synthesis in later chapters and the reporting in Chapter 13 .

This is a preview of subscription content, log in via an institution .

Buying options

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
  • Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Paraphrased from electronic messages to Rob Dekkers dated 6th June 2020 and 19th October 2020.

Predatory journals are used by exploitive publishers that charge publication fees to authors without reviewing articles for quality and legitimacy, and without providing the other editorial and publishing services that legitimate academic journals provide. They are considered as predatory, because scholars can be tricked into publishing with them. See Beall ( 2016 ) for how these predatory journals operate and also which points to pay attention to.

Note that ranking, although often used for all kinds of purposes, is not directly a reflection of the importance or relevance of journals. An increasing number of academics and scientific institutions have signed the San Francisco Declaration on Research Assessment, published in 2013, also known by its abbreviation DORA. Specifically, it states that the impact factor (or any other ranking) is not to be used as a substitute ‘measure of the quality of individual research articles, or in hiring, promotion, or funding decisions.’

In addition to predatory journals there are predatory conferences, such as those organised by WASET (World Academy of Science, Engineering and Technology); there have been numerous publications in the media about these type of organisations and how they falsely claim to be credible conferences. Therefore, always check the credentials of a conference before submitting papers to avoid becoming victim of these.

These key tests have been adapted from a presentation given by Prof. David Bennett for a select group of doctoral students at the University of the West of Scotland in 2012.

Careful reading of the comments in the review is necessary, including the advice of the editor or associate editor, because sometimes the decision for a major revision can be interpreted as a polite way of rejecting (McKercher et al. 2007 , p. 458). If there is any doubt, it is better to contact the editor before submitting a revision; note that not all editors might reply.

Beall J (2016) Essential information about predatory publishers and journals. Int Higher Educ 86:2–3. https://doi.org/10.6017/ihe.2016.86.9358

Article   Google Scholar  

Bodolica V, Spraggon M (2018) An end-to-end process of writing and publishing influential literature review articles: do’s and don’ts. Manag Decis 56(11):2472–2486. https://doi.org/10.1108/MD-03-2018-0253

Callahan JL (2014) Writing literature reviews: a reprise and update. Hum Resour Dev Rev 13(3):271–275. https://doi.org/10.1177/1534484314536705

Green BN, Johnson CD, Adams A (2006) Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med 5(3):101–117. https://doi.org/10.1016/S0899-3467(07)60142-6

Holweg M (2007) The genealogy of lean production. J Oper Manag 25(2):420–437. https://doi.org/10.1016/j.jom.2006.04.001

Jennex ME (2015) Literature reviews and the review process: an editor-in-chief’s perspective. Commun Assoc Inf Syst 36:139–146. https://doi.org/10.17705/1CAIS.03608

MacLure M (2005) Clarity bordering on stupidity: where’s the quality in systematic review? J Educ Policy 20(4):393–416. https://doi.org/10.1080/02680930500131801

McKercher B, Law R, Weber K, Song H, Hsu C (2007) Why referees reject manuscripts. J Hosp Tour Res 31(4):455–470. https://doi.org/10.1177/1096348007302355

Rocco TS, Plakhotnik MS (2009) Literature reviews, conceptual frameworks, and theoretical frameworks: terms, functions, and distinctions. Hum Resour Dev Rev 8(1):120–130. https://doi.org/10.1177/1534484309332617

Shah R, Ward PT (2007) Defining and developing measures of lean production. J Oper Manag 25(4):785–805. https://doi.org/10.1016/j.jom.2007.01.019

Soilemezi D, Linceviciute S (2018) Synthesizing qualitative research: reflections and lessons learnt by two new reviewers. Int J Qual Methods 17(1):1–14. https://doi.org/10.1177/1609406918768014

Taylor EW, Beck J, Ainsworth E (2001) Publishing qualitative adult education research: a peer review perspective. Stud Educ Adults 33(2):163–179. https://doi.org/10.1080/02660830.2001.11661452

Torraco RJ (2005) Writing integrative literature reviews: guidelines and examples. Hum Resour Dev Rev 4(3):356–367. https://doi.org/10.1177/1534484305278283

Toussaint JS, Berry LL (2013) The promise of lean in health care. Mayo Clin Proc 88(1):74–82. https://doi.org/10.1016/j.mayocp.2012.07.025

Trafford V, Leshem S, Bitzer E (2014) Conclusion chapters in doctoral theses: some international findings. High Educ Rev 46(3):52–81

Google Scholar  

Womack JP, Jones DT (1996) Lean thinking. Simon & Schuster, New York

Womack JP, Jones DT, Roos D (1991) The machine that changed the world: the story of lean production. Free Press, New York

Young TP, McClean SI (2008) A critical look at lean thinking in healthcare. Qual Saf Health Care 17(5):382–386. https://doi.org/10.1136/qshc.2006.020131

Download references

Author information

Authors and affiliations.

University of Glasgow, Glasgow, UK

Rob Dekkers

Glasgow Caledonian University, Glasgow, UK

Lindsey Carey

Peter Langhorne

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Cite this chapter.

Dekkers, R., Carey, L., Langhorne, P. (2022). Publishing Literature Reviews. In: Making Literature Reviews Work: A Multidisciplinary Guide to Systematic Approaches. Springer, Cham. https://doi.org/10.1007/978-3-030-90025-0_16

Download citation

DOI : https://doi.org/10.1007/978-3-030-90025-0_16

Published : 11 August 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-90024-3

Online ISBN : 978-3-030-90025-0

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • J Grad Med Educ
  • v.8(3); 2016 Jul

The Literature Review: A Foundation for High-Quality Medical Education Research

a  These are subscription resources. Researchers should check with their librarian to determine their access rights.

Despite a surge in published scholarship in medical education 1 and rapid growth in journals that publish educational research, manuscript acceptance rates continue to fall. 2 Failure to conduct a thorough, accurate, and up-to-date literature review identifying an important problem and placing the study in context is consistently identified as one of the top reasons for rejection. 3 , 4 The purpose of this editorial is to provide a road map and practical recommendations for planning a literature review. By understanding the goals of a literature review and following a few basic processes, authors can enhance both the quality of their educational research and the likelihood of publication in the Journal of Graduate Medical Education ( JGME ) and in other journals.

The Literature Review Defined

In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth. Several organizations have published guidelines for conducting an intensive literature search intended for formal systematic reviews, both broadly (eg, PRISMA) 5 and within medical education, 6 and there are excellent commentaries to guide authors of systematic reviews. 7 , 8

  • A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact.
  • A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.
  • Literature reviews take time, are iterative, and should continue throughout the research process.
  • Researchers should maximize the use of human resources (librarians, colleagues), search tools (databases/search engines), and existing literature (related articles).
  • Keeping organized is critical.

Such work is outside the scope of this article, which focuses on literature reviews to inform reports of original medical education research. We define such a literature review as a synthetic review and summary of what is known and unknown regarding the topic of a scholarly body of work, including the current work's place within the existing knowledge . While this type of literature review may not require the intensive search processes mandated by systematic reviews, it merits a thoughtful and rigorous approach.

Purpose and Importance of the Literature Review

An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the “journal-as-conversation” metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: “Imagine yourself joining a conversation at a social event. After you hang about eavesdropping to get the drift of what's being said (the conversational equivalent of the literature review), you join the conversation with a contribution that signals your shared interest in the topic, your knowledge of what's already been said, and your intention.” 9

The literature review helps any researcher “join the conversation” by providing context, informing methodology, identifying innovation, minimizing duplicative research, and ensuring that professional standards are met. Understanding the current literature also promotes scholarship, as proposed by Boyer, 10 by contributing to 5 of the 6 standards by which scholarly work should be evaluated. 11 Specifically, the review helps the researcher (1) articulate clear goals, (2) show evidence of adequate preparation, (3) select appropriate methods, (4) communicate relevant results, and (5) engage in reflective critique.

Failure to conduct a high-quality literature review is associated with several problems identified in the medical education literature, including studies that are repetitive, not grounded in theory, methodologically weak, and fail to expand knowledge beyond a single setting. 12 Indeed, medical education scholars complain that many studies repeat work already published and contribute little new knowledge—a likely cause of which is failure to conduct a proper literature review. 3 , 4

Likewise, studies that lack theoretical grounding or a conceptual framework make study design and interpretation difficult. 13 When theory is used in medical education studies, it is often invoked at a superficial level. As Norman 14 noted, when theory is used appropriately, it helps articulate variables that might be linked together and why, and it allows the researcher to make hypotheses and define a study's context and scope. Ultimately, a proper literature review is a first critical step toward identifying relevant conceptual frameworks.

Another problem is that many medical education studies are methodologically weak. 12 Good research requires trained investigators who can articulate relevant research questions, operationally define variables of interest, and choose the best method for specific research questions. Conducting a proper literature review helps both novice and experienced researchers select rigorous research methodologies.

Finally, many studies in medical education are “one-offs,” that is, single studies undertaken because the opportunity presented itself locally. Such studies frequently are not oriented toward progressive knowledge building and generalization to other settings. A firm grasp of the literature can encourage a programmatic approach to research.

Approaching the Literature Review

Considering these issues, journals have a responsibility to demand from authors a thoughtful synthesis of their study's position within the field, and it is the authors' responsibility to provide such a synthesis, based on a literature review. The aforementioned purposes of the literature review mandate that the review occurs throughout all phases of a study, from conception and design, to implementation and analysis, to manuscript preparation and submission.

Planning the literature review requires understanding of journal requirements, which vary greatly by journal ( table 1 ). Authors are advised to take note of common problems with reporting results of the literature review. Table 2 lists the most common problems that we have encountered as authors, reviewers, and editors.

Sample of Journals' Author Instructions for Literature Reviews Conducted as Part of Original Research Article a

An external file that holds a picture, illustration, etc.
Object name is i1949-8357-8-3-297-t01.jpg

Common Problem Areas for Reporting Literature Reviews in the Context of Scholarly Articles

An external file that holds a picture, illustration, etc.
Object name is i1949-8357-8-3-297-t02.jpg

Locating and Organizing the Literature

Three resources may facilitate identifying relevant literature: human resources, search tools, and related literature. As the process requires time, it is important to begin searching for literature early in the process (ie, the study design phase). Identifying and understanding relevant studies will increase the likelihood of designing a relevant, adaptable, generalizable, and novel study that is based on educational or learning theory and can maximize impact.

Human Resources

A medical librarian can help translate research interests into an effective search strategy, familiarize researchers with available information resources, provide information on organizing information, and introduce strategies for keeping current with emerging research. Often, librarians are also aware of research across their institutions and may be able to connect researchers with similar interests. Reaching out to colleagues for suggestions may help researchers quickly locate resources that would not otherwise be on their radar.

During this process, researchers will likely identify other researchers writing on aspects of their topic. Researchers should consider searching for the publications of these relevant researchers (see table 3 for search strategies). Additionally, institutional websites may include curriculum vitae of such relevant faculty with access to their entire publication record, including difficult to locate publications, such as book chapters, dissertations, and technical reports.

Strategies for Finding Related Researcher Publications in Databases and Search Engines

An external file that holds a picture, illustration, etc.
Object name is i1949-8357-8-3-297-t03.jpg

Search Tools and Related Literature

Researchers will locate the majority of needed information using databases and search engines. Excellent resources are available to guide researchers in the mechanics of literature searches. 15 , 16

Because medical education research draws on a variety of disciplines, researchers should include search tools with coverage beyond medicine (eg, psychology, nursing, education, and anthropology) and that cover several publication types, such as reports, standards, conference abstracts, and book chapters (see the box for several information resources). Many search tools include options for viewing citations of selected articles. Examining cited references provides additional articles for review and a sense of the influence of the selected article on its field.

Box Information Resources

  • Web of Science a
  • Education Resource Information Center (ERIC)
  • Cumulative Index of Nursing & Allied Health (CINAHL) a
  • Google Scholar

Once relevant articles are located, it is useful to mine those articles for additional citations. One strategy is to examine references of key articles, especially review articles, for relevant citations.

Getting Organized

As the aforementioned resources will likely provide a tremendous amount of information, organization is crucial. Researchers should determine which details are most important to their study (eg, participants, setting, methods, and outcomes) and generate a strategy for keeping those details organized and accessible. Increasingly, researchers utilize digital tools, such as Evernote, to capture such information, which enables accessibility across digital workspaces and search capabilities. Use of citation managers can also be helpful as they store citations and, in some cases, can generate bibliographies ( table 4 ).

Citation Managers

An external file that holds a picture, illustration, etc.
Object name is i1949-8357-8-3-297-t04.jpg

Knowing When to Say When

Researchers often ask how to know when they have located enough citations. Unfortunately, there is no magic or ideal number of citations to collect. One strategy for checking coverage of the literature is to inspect references of relevant articles. As researchers review references they will start noticing a repetition of the same articles with few new articles appearing. This can indicate that the researcher has covered the literature base on a particular topic.

Putting It All Together

In preparing to write a research paper, it is important to consider which citations to include and how they will inform the introduction and discussion sections. The “Instructions to Authors” for the targeted journal will often provide guidance on structuring the literature review (or introduction) and the number of total citations permitted for each article category. Reviewing articles of similar type published in the targeted journal can also provide guidance regarding structure and average lengths of the introduction and discussion sections.

When selecting references for the introduction consider those that illustrate core background theoretical and methodological concepts, as well as recent relevant studies. The introduction should be brief and present references not as a laundry list or narrative of available literature, but rather as a synthesized summary to provide context for the current study and to identify the gap in the literature that the study intends to fill. For the discussion, citations should be thoughtfully selected to compare and contrast the present study's findings with the current literature and to indicate how the present study moves the field forward.

To facilitate writing a literature review, journals are increasingly providing helpful features to guide authors. For example, the resources available through JGME include several articles on writing. 17 The journal Perspectives on Medical Education recently launched “The Writer's Craft,” which is intended to help medical educators improve their writing. Additionally, many institutions have writing centers that provide web-based materials on writing a literature review, and some even have writing coaches.

The literature review is a vital part of medical education research and should occur throughout the research process to help researchers design a strong study and effectively communicate study results and importance. To achieve these goals, researchers are advised to plan and execute the literature review carefully. The guidance in this editorial provides considerations and recommendations that may improve the quality of literature reviews.

Jessie Ball duPont Library

Literature reviews.

  • Examples & Tutorials

Using Databases to Find Literature Reviews

  • Need Research Help?

Research Librarian

Profile Photo

Most specialized databases will have scholarly articles that are either literature reviews or have a review of the literature as an introduction to the research.  Keys to finding them in some of our best databases are:

Sewanee users only

  • << Previous: Examples & Tutorials
  • Next: Need Research Help? >>
  • Last Updated: Aug 28, 2023 4:40 PM
  • URL: https://library.sewanee.edu/litreview

Research Tools

  • Find Articles
  • Find Research Guides
  • Find Databases
  • Ask a Librarian
  • Learn Research Skills
  • How-To Videos
  • Borrow from Another Library (Sewanee ILL)
  • Find Audio and Video
  • Find Reserves
  • Access Electronic Resources

Services for...

  • College Students
  • The School of Theology
  • The School of Letters
  • Community Members

Spaces & Places

  • Center for Leadership
  • Center for Speaking and Listening
  • Center for Teaching
  • Floor Maps & Locations
  • Ralston Listening Library
  • Research Help
  • Study Spaces
  • Tech Help Desk
  • Writing Center

About the Library

  • Where is the Library?
  • Library Collections
  • New Items & Themed Collections
  • Library Policies
  • Library Staff
  • Friends of the Library

Jessie Ball duPont Library, University of the South

178 Georgia Avenue, Sewanee, TN 37383

931.598.1664

Logo for Alaska Digital Texts

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

What is a Literature Review?

What’s a literature review.

The straightforward answer is that a literature review is a review or synthesis of all the research published on a certain topic. But I’d rather explain it from a skateboarder’s perspective:

One of my favorite movies is the 1989 classic Back to the Future Part 2 where the bodacious skater Marty McFly time-travels to the future and sees a hoverboard. As a kid, I was a dabbling skateboarder and thought if I could just have one of those hoverboards, all my troubles would disappear. It was an optimistic time.

Trouble is, hoverboards are really hard to make. We’ve already passed the year 2015 when the “Future” of Back to the Future Part 2 takes place, and guess what? No hoverboards. I know you’ve seen a skateboard-like, two-wheeled device marketed with the name “Hoverboard” but that’s just an electric, no-handled scooter.

1024px-Hover_board_hovering.jpg

I want a real hoverboard. That you ride in the air . So how can we know when real hoverboards will be available? How can we know where the technology is now? Will we know a real hoverboard when we see one? Tony Hawk, the best skateboarder of all time (whose face was incidentally taped to my wall in the ’80s) recently filmed a 2-minute video of how far real hoverboard technology has come–filmed on the very day Marty McFly supposedly went to the future: October 21, 2015:

Tony Hawk and the cutting edge of hoverboard research

Image preview of a YouTube video

Although this “hoverboard” was really a huge black rectangle the size of Delaware floating only an inch off the ground, and although Tony Hawk fell off a lot, he was technically in the air, so I’m taking that as a good sign. Then recently, a professional jet ski rider broke the world record for longest time “hovering” in the air with a highly dangerous jet-engine-propelled contraption called Flyboard Air. It’s also definitely a step in the right direction, but there’s a big problem (beside extreme danger): it’s projected to cost around $250,000.

The good news is, now we’ve found the point where hoverboard research actually is. The bad news: we have to face the sad truth that it might still be a while before we get real flying hoverboards. But at least now we know.

The State of a Field on a Topic

That leads me to literature reviews. Whenever you want to know the state of a field of research like how far hoverboard technology has come, the best way to find out is probably not YouTube videos. It turns out you can do something much more reliable: conduct a literature review . In this case, “literature” doesn’t mean the Victorian novels you read in English class, it means all the research published on a certain topic. So a literature review is simply a review or a synthesis of the research published on a topic.

Researchers today don’t just start projects out of the blue–they do their homework first by finding out what others have already researched. So if you want to make a hoverboard, you don’t just go to Home Depot and buy random parts–you research what others have done and check out the conversation so you don’t have to reinvent the wheel.

“If I have seen further, it is by standing on the shoulders of giants.” –Sir Isaac Newton (and the motto of Google Scholar)

Before good researchers set up any surveys or experiments, or even write a proposal for funding, they figure out exactly which research questions have already been asked and answered. Same goes for anyone wanting to make a product that will sell. But more importantly, they look for the gaps in the research where answers have yet to be found. And then they focus their own research on filling in some of those gaps. That’ll be your job, too.

In other words, the goal of a literature review is to find the sweet spot where the most promising research is happening now–we call that the cutting edge.

How is a Literature Review different from a typical Research Paper?

You’ve probably been writing research papers most of your life– starting from the five-paragraph essay you learned in high school to the term paper you wrote last semester that had a thesis statement and lots of quotes. So it can seem daunting to switch gears to writing a literature review, but there are some distinct advantages to making the switch. The trick is first understanding the difference between the two.

Research Papers are Thesis Driven

The difference between a typical research paper and a literature review is your purpose and strategy. When you’re assigned to write a research paper, you start with a thesis or argument that you’d like to make. Your thesis has to do with changes you’d like to see in the future. Then you search for sources that support your point. You might adjust your thesis if you come across sources that challenge your claim, but generally, the sources you’ve gathered become evidence for your thesis and you use them to support your point. In other words, your argumentative research paper is driven by your thesis .

Literature Reviews are Source Driven

In contrast, when you write a Literature Review, the sources themselves dictate what you’ll say in your paper. Remember, your goal is to tell your audience the state of the field on a topic–what’s been happening in the published research–so you can find the cutting edge and where the research gaps are. Therefore, you need to find and evaluate the most relevant sources surrounding a topic and then write a review based on what you find . You can’t decide on a thesis statement or know what points you’ll make before you start because you have to find out what researchers are doing before you can report on that. Simply put, your literature review is driven by your sources .

You’ll still have an overarching point/thesis that controls your literature review paper structure, but it will be a claim about what patterns you found in the research– not an argument about a change you want to see in the future or a new way to look at something. And you’ll decide on your thesis much later in the writing process. Here’s a table that compares the writing process of a traditional research paper with that of a literature review:

Literature Reviews: Catching up with Old Friends

What do you do when you meet an old friend? You ask,

“How are you? What have you been up to? Fill me in!”

people-talking-908342_1280.jpg

Literature Reviews are like getting filled in by an old friend . Only this time, you’re explaining how a field of research has gotten to the present (like how far hoverboard technology has come). But like a conversation with an old friend, you want to review only the details most relevant to the situation. You don’t usually give a moment-by-moment chronology of what you’ve done in your life (no one has time for that); rather, you talk in terms of categories–work, family, travel, etc. This is like the synthesis that happens in a Literature Review. As you read sources about a specific topic, you’ll look for themes, for similarities and differences, for points of agreement and disagreement, for gaps in the research that haven’t been filled in yet. Those themes become the categories you’ll talk about in your literature review so your audience will understand the big picture about your topic.

Literature Reviews in the Sciences

Grant proposals.

Any grant proposal submitted to request research funding begins with an extensive literature review to justify the need for the research funds. If you can prove there’s a gap in knowledge, it makes it that much easier to convince your audience to give you funding to fill that gap.

IMRAD Articles

Wineglass_model_for_IMRaD_structure..png

The Introduction of an IMRAD article includes a literature review. Photo by Tom Toyosak i on Wikimedia Commons

IMRAD (pronounced “im-rad”) stands for Introduction, Methods, Results, and Discussion and is the most common genre published in the social sciences and sciences. Most of the sources you gather will likely be IMRAD-format papers. The I in IMRAD stands for Introduction and usually consists of a review of the literature on the authors’ research topic. The author(s) usually use the Introduction section to report on the published literature about their research topic and reveal the trends and gaps in current research. An added benefit to beginning an article this way is that by showing the gaps in the research, the author(s) can justify their own research and explain the significance of the topic they chose to examine. Clever!

As you might guess, the sections following the Introduction (Methods, Results, Discussion) describe the primary research the author(s) conducted to answer their research question. First they report on their quantitative and/or qualitative M ethods (M in IMRAD) including statistical analyses. Then they publish their R esults (R in IMRAD). Finally, the author(s) embark on a D iscussion (D in IMRAD) of their results in the context of the greater field of research and make suggestions for future research. This starts the research cycle over again as someone else reads their article as part of their own review of the literature and discovers a gap in the research that can be filled by new primary research. 

Published Literature Reviews

In the world of science and social science, literature reviews can also be published on their own. For example, if someone does an extensive investigation into an important topic, the publishers of academic journals will often publish that literature review on its own to help other researchers understand that topic better.

Popular Literature Reviews

196px-Wikipedia-logo-en-big.png

Lest you think nerdy academics are the only ones who rely on literature reviews, recall the last time you went on Wikipedia . If you think about it, Wikipedia is really just a giant literature review on millions of topics. Although the information on Wikipedia is not formally peer-reviewed like the reviews published in academic journals, they do cite all their sources and frequently revise to keep the information current. Clearly there’s a market for relevant information. If you really want your mind to explode Inception -style, look up “Wikipedia” on Wikipedia and you’ll find a literature review about a literature review.

Adapted from “What is a Literature Review?” in Writing in the Social Sciences. Authored by Christie Cowles Charles. Located at: https://edtechbooks.org/writing.

License: CC BY- SA

What is a Literature Review? Copyright © 2020 by Sara Rufner is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

  • Open access
  • Published: 15 February 2024

Improvement in work productivity among psoriatic arthritis patients treated with biologic or targeted synthetic drugs: a systematic literature review and meta-analysis

  • Laure Gossec   ORCID: orcid.org/0000-0002-4528-310X 1 , 2 ,
  • Brittany Humphries 3 , 4 ,
  • Megan Rutherford 5 ,
  • Vanessa Taieb 6 ,
  • Damon Willems 7 &
  • William Tillett   ORCID: orcid.org/0000-0001-7531-4125 8  

Arthritis Research & Therapy volume  26 , Article number:  50 ( 2024 ) Cite this article

Metrics details

Capacity to work is impacted by psoriatic arthritis (PsA). Our objective was to describe the course of work productivity and leisure activity in patients with PsA treated with biologic (b) and targeted synthetic (ts) disease-modifying antirheumatic drugs (DMARDs).

A systematic literature review identified all trials and observational studies published January 1, 2010–October 22, 2021, reporting work productivity using the Work Productivity and Activity Impairment Questionnaire (WPAI) in patients with PsA treated with b/tsDMARDs. Outcomes for WPAI domains (absenteeism, presenteeism, total work productivity, and activity impairment) were collected at baseline and time point closest to 24 weeks of treatment. A random effects meta-analysis of single means was conducted to calculate an overall absolute mean change from baseline for each WPAI domain.

Twelve studies (ten randomized controlled and two observational) assessing patients treated with adalimumab, bimekizumab, guselkumab, ixekizumab, risankizumab, secukinumab, or upadacitinib were analysed. Among 3741 employed patients, overall mean baseline scores were 11.4%, 38.7%, 42.7%, and 48.9% for absenteeism, presenteeism, total work productivity impairment, and activity impairment, respectively. Estimated absolute mean improvements (95% confidence interval) to week 24 were 2.4 percentage points (%p) (0.6, 4.1), 17.8%p (16.2,19.3), 17.6%p (15.9,19.4), and 19.3%p (17.6, 21.0) respectively, leading to a mean relative improvement of 41% for total work productivity. The change in work outcomes in the b/tsDMARDs appeared similar.

Conclusions

This systematic literature review and meta-analysis confirmed that patients with active PsA have a substantially reduced capacity to work and participate in leisure activities. Substantial improvements across various WPAI domains were noted after 24 weeks of b/tsDMARD treatment, especially in presenteeism, total work productivity, and activity impairment. These findings may be useful for reimbursement purposes and in the context of shared decision-making.

Key summary points

This systematic literature review (SLR) of randomized clinical trials and observational studies of biologic (b) and targeted synthetic (ts) disease-modifying antirheumatic drugs b/tsDMARDs in patients with PsA found that at treatment introduction, patients presented with a 42.7% mean productivity loss per week as assessed by the Work Productivity and Activity Impairment (WPAI) Questionnaire.

Through a meta-analysis comparing before/after values without adjustment for placebo response, we found that after 24 weeks of treatment with b/tsDMARDs, there was a mean absolute improvement of 17.6 percentage points and a mean relative improvement of 41% in total work productivity, with similar magnitudes of improvement in time spent at work and regular activities outside of work.

These results provide clinical-, regulatory- and reimbursement decision-makers with data on the potential societal and socio-economic benefits of b/tsDMARDs in PsA.

Plain language summary

Psoriatic arthritis (PsA) has a major impact on patients’ lives, including their ability to work by causing absence and reducing productivity. By pooling together published studies (12 studies, corresponding to 3741 patients) and comparing what patients reported before starting treatment to during treatment, we found that over the course of treatment with biologic (b) and targeted synthetic (ts) disease-modifying antirheumatic drugs (DMARDs), patients with PsA had an average of 18% higher total work productivity, translating to a 41% reduced impact of PsA at the group level (without looking at comparisons to a placebo response). It is important for health professionals and patients to know that work outcomes affected by PsA are improved when patients take b/tsDMARDS.

Introduction

Psoriatic arthritis (PsA) has a substantial impact on health-related quality of life (HRQoL) [ 1 ]. As the clinical presentation of PsA varies, its impact often extends beyond joint damage to include comorbidities such as obesity, depression, anxiety, and cardiovascular disease [ 2 , 3 ]. In particular, patients with PsA have reported its effect on individual activities and social participation, as well as physiological functioning as linked to the World Health Organization (WHO) International Classification of Functioning, Disability, and Health (ICF) [ 4 , 5 ]. The consequences of PsA on work are important and include deleterious effects such as hours of missed work (absenteeism), diminished productivity while at work (presenteeism), and increased economic burden due to indirect costs [ 4 , 6 , 7 , 8 , 9 , 10 ].

The availability of biologic (b) and targeted synthetic (ts) disease-modifying antirheumatic drugs (DMARDs) over the last 20 years has improved clinical outcomes in PsA [ 11 ]. However, work and work productivity following b/tsDMARD treatment initiation in PsA based on evidence from both randomized controlled trials and observational studies has not been comprehensively assessed.

The objective of this systematic literature review (SLR) and meta-analysis was to describe work and work productivity in patients with PsA prior to and following b/tsDMARD treatment, and to explore the potential economic impact of changes in productivity.

This SLR was conducted according to the methodological guidance of the Centre for Reviews and Dissemination and reporting requirements of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 12 , 13 ].

Search strategy and selection criteria

The search aimed to capture all trials of b/tsDMARDs in PsA reporting patient-reported outcomes. The eligibility criteria were defined according to Population, Intervention, Comparator, Outcome, and Study Design (PICOS) criteria as reported in Table S 1 . While the SLR was designed to assess HRQoL in addition to the work impact of PsA, for this analysis we only included studies reporting work impact, captured using the Work Productivity and Activity Impairment (WPAI) questionnaire [ 14 ]. Only studies reporting outcomes related to a specific intervention as listed in the eligibility criteria were included.

The search strategy was based on key terms and synonyms related to the WPAI, work (e.g. work, employment), and productivity (e.g. presenteeism, absenteeism, impact, loss, capacity). We searched MEDLINE, Embase, EconLit, and Cochrane from January 1, 2010, through the search date of October 22, 2021. A hand search was conducted in August 2022 to update the evidence base. Data sources and the full search strategy are provided in Table S 2 and Table S 3 , respectively.

All records were screened independently by two reviewers. Disagreements on a publication’s eligibility were resolved by discussion and/or arbitration provided by a third reviewer. Data extraction of study characteristics and outcomes of included studies was performed by a single reviewer (MR) and validated by a senior member of the research team (BH).

The primary outcome of interest for this analysis was health-related work impairment as measured using the WPAI, a patient-reported outcome [ 14 ].

The WPAI has four domains to assess absenteeism (the percentage of work time missed), presenteeism (the percentage of impairment experienced while at work), overall or total work productivity impairment (work impairment due to absenteeism and presenteeism), and leisure activity impairment (impairment of activities of daily living). The four domain scores are expressed as percentages, with high percentage scores indicating a high degree of impairment and less productivity over the past 7 days [ 14 ]. Only patients employed at baseline are included in the evaluation of WPAI, with the exception of the activity impairment domain which may also include non-employed patients.

We estimated changes in WPAI by comparing WPAI domains, when starting a b/tsDMARD and after around 6 months of treatment. Of note, this corresponds to a pre-post analysis but was not adjusted for placebo response (i.e. there was no comparison to improvements in the placebo group). Indeed, not all the studies had a placebo group, and to compare to placebo, a network meta-analysis would have been needed.

To explore the indirect costs attributable to PsA and the changes following treatment (again, this corresponds to ‘raw’ changes not adjusted to placebo response), we estimated a monetary value for the total productivity loss using the human capital approach, as explained below [ 15 , 16 ].

Evidence synthesis

A random effects meta-analysis of single means was performed using RStudio Version 2022.07.1 (meta package v4.17–0) [ 17 ]. The outcome of interest was the mean change in WPAI scores from baseline to the timepoint closest to 24 weeks (i.e. 24 weeks ± 4 weeks). Meta-analysis results were reported as absolute mean change from baseline in WPAI score (weighted by study sample size) and 95% confidence interval (CI), which equates to a pooled or summary estimate of the WPAI score across the included studies around 24 weeks. For illustrative purposes, relative change was also assessed. Results for absolute change are presented as percentage point (%p), whereas relative change is percent. The reference case analysis included any studies reporting WPAI outcomes as a mean change or least-squares mean (LSM) change from baseline. A scenario analysis was performed in which the adjusted means (i.e. LSM change) were excluded from the meta-analysis and only the raw values (i.e. mean change) were considered to examine internal validity.

The results from the meta-analyses were also used to estimate the indirect costs attributable to PsA [ 15 ]. For this analysis, we assigned a monetary value to lost productivity using the human capital approach, which takes the patient’s perspective by counting any hour not worked as an hour of lost productivity [ 16 ]. Domain scores were multiplied by 40 h (assuming a standard work week) to estimate the total lost productivity (total work impairment due to absenteeism and presenteeism). We then multiplied the lost hours of productivity by €29.10 [ 18 , 19 ], the reported 2021 average hourly labour costs in the European Union (27 countries, from 2020) and $40.35 [ 20 ], the 2021 employer cost for employee compensation for the United States (US), to broadly capture the international nature of the studies used in this analysis.

Of 6689 records, 751 publications were selected for full-text review, and 27 publications (from 14 unique studies) reported WPAI data. Two additional studies were identified from the hand search and one data on file from the sponsor (UCB) was also included, resulting in a total of 30 reports from 17 unique studies. Among these, 12 studies reported outcomes at a timepoint close to 24 weeks and were evaluated in the meta-analysis (Fig.  1 ).

figure 1

PRISMA flow of information for the systematic literature review. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PsA, psoriatic arthritis; WPAI, Work Productivity and Ativity Impairment Questionnaire

Of the 12 studies, 10 drew evidence from randomized controlled trials (RCTs) with a placebo comparator and two from prospective observational studies (Table  1 ). Study sample sizes ranged from 100 [ 21 ] to 1281 [ 22 ] patients (mean, 532 patients). The interventions assessed included adalimumab ( n  = 4 studies), secukinumab ( n  = 3), ixekizumab ( n  = 2), risankizumab ( n  = 2), upadacitinib ( n  = 2), bimekizumab ( n  = 1), and guselkumab ( n  = 1). Most RCTs allowed patients in the intervention and placebo arms to take concomitant conventional synthetic (cs)DMARDs such as methotrexate, corticosteroids, non-steroidal anti-inflammatory drugs (NSAIDs), or other analgesics. A risk of bias assessment was performed and presented in Table S 4 .

Overall, across all WPAI domains between 3683 and 5774 patients at baseline and 2425 and 3774 patients at week 24 were analysed. Among all studies, 48.8% of patients were male, the weighted mean age of patients was 50.1 years, and the weighted mean disease duration was 7.0 years (Table  1 ). Where reported, an average of 58.7% of patients were employed at baseline.

Absenteeism

Among the 3741 patients with baseline WPAI data included in the meta-analysis, the pooled estimated mean absenteeism score (percent of time missed from work over a 7-day period due to PsA) was 11.4%, range 5.8–16.3% (95% CI 10.2, 12.6) (Table  2 ). This can be illustrated as 4.6 h absent from work per week, based on a 40-h work week. At week 24, the pooled absolute mean change from baseline among patients treated with a b/tsDMARD was − 2.4%p, range − 12.5 to 6.1%p (95% CI − 4.1, − 0.6) (Fig.  2 ), i.e. a relative improvement of 21.0%. In general, there was a greater estimated improvement in absenteeism scores among patients receiving a b/tsDMARD compared to patients taking placebo (Figure S 1 ).

figure 2

Meta-analysis of percent mean CFB in absenteeism scores per intervention at 24 weeks. Note: Point estimates represent the mean CFB scores reported for each intervention. No direct comparison is intended nor was made across interventions. *Denotes observational study. CFB , change from baseline; CI , confidence interval; Q2W , every 2 weeks; Q4W , every 4 weeks; Q8W , every 8 weeks

Presenteeism

The meta-analysis estimated mean presenteeism score (percent of impairment while working due to PsA) at baseline was 38.7% ( n  = 3620), range 30.5–47.3% (95% CI 35.7, 43.2) (Table  2 ). This equates to 15.5 h of impaired or reduced work performance per week.

In the 12 studies (2425 patients), the pooled absolute mean change from baseline in presenteeism scores was − 17.8%p, range − 24.3 to − 8.7%p (95% CI − 19.3, − 16.2) among patients treated with a b/tsDMARD (Fig.  3 ) with a relative improvement of 46.0%. Improvements in presenteeism scores among patients taking placebo were smaller, with an estimated pooled mean change of − 5.5% from baseline (Figure S 2 ).

figure 3

Meta-analysis of percent mean CFB in presenteeism scores per intervention at 24 weeks. Note: Point estimates represent the mean CFB scores reported for each intervention. No direct comparison is intended nor was made across interventions. * Denotes observational study. CFB , change from baseline; CI , confidence interval; Q2W , every 2 weeks; Q4W , every 4 weeks; Q8W , every 8 weeks

Work productivity impairment

Baseline work productivity impairment scores were available for 3683 patients. The meta-analysis estimated mean loss of work productivity at baseline was 42.7%, range 34.2–50.8% (95% CI 40.6, 44.9) (Table  2 ), which can be estimated as 17.1 h of total work productivity lost per week. At 24 weeks, for patients treated with a b/tsDMARD, the pooled absolute mean change in total work productivity was − 17.6%p, range − 25.2 to − 12.2%p (95% CI − 19.4, − 15.9) (Fig.  4 ) with a mean relative improvement of 41.2%. Similar to other WPAI domains, improvements in total work productivity scores among patients taking placebo for up to 24 weeks were small (Figure S 3 ).

figure 4

Meta-analysis of percent mean CFB total work productivity impairment scores per intervention at 24 weeks. Note: Point estimates represent the mean CFB scores reported for each intervention. No direct comparison is intended nor was made across interventions. *Denotes observational study. CFB , change from baseline; CI , confidence interval; Q2W , every 2 weeks; Q4W , every 4 weeks; Q8W , every 8 weeks

Activity impairment

Baseline activity impairment was reported for 5774 patients, since unlike the other WPAI domains, this domain includes patients who were not employed. The estimated mean baseline impairment was 48.9%, range 41.7–55.1% (95% CI 47.5, 50.4), indicating the percent of impaired or reduced ability to participate in leisure activities over a 7-day period (Table  2 ). Overall, all patients treated with b/tsDMARDs reported a reduction in activity impairment from baseline with a pooled absolute mean change of − 19.3%p (95% CI − 21.4, − 17.6) (Fig.  5 ) and a mean relative improvement of 39.5%.

figure 5

Meta-analysis of percent mean CFB in activity impairment scores per intervention at 24 weeks. Note: Point estimates represent the mean CFB scores reported for each intervention. No direct comparison is intended nor was made across interventions. * Denotes observational study. CFB , change from baseline; CI , confidence interval; Q2W , every 2 weeks; Q4W , every 4 weeks; Q8W , every 8 weeks

Indirect costs

At baseline, the overall productivity loss among patients with PsA reported in this review ranged from 13.7 to 20.3 h per week. This equates to an estimated range of indirect costs for the European Union of €20,757 to €30,833 per patient per year (US$28,782 to US$42,753). After treatment with a b/tsDMARD, there was a pooled mean improvement in total work productivity of 7.0 h per week (mean change from baseline, − 17.6%). Based on the analysis of change from baseline among patients treated with b/tsDMARDs, and without adjusting on placebo response, the estimated absolute mean decrease in PsA-related indirect costs linked to productivity was €10,688 (US$14,820) per patient, per year.

This review reports important information on the burden of PsA on work prior to and following b/tsDMARD treatment. In studies of b/tsDMARDs among patients with PsA, at treatment introduction, patients presented with a high burden of their disease on work, with an estimated mean work productivity loss of 17.1 h per week, or a mean reduction of 42.7% in total work productivity. In all studies, presenteeism was a greater contributor to overall work productivity loss than absenteeism. Exploratory extrapolations of indirect costs associated with work productivity impairment yielded estimates between €20,000 to €30,000 (US$28,000 to US$42,000) per person annually. Beyond work outcomes, the additional burden was noted in the form of leisure activity impairment (mean, 48.9%). Through a meta-analysis corresponding to changes with treatment (without adjustments for placebo response), we found that after 24 weeks of treatment with b/tsDMARDs, there was a mean absolute improvement of 17.6%p in total work productivity, corresponding to a mean relative improvement of 41%. After treatment with a b/tsDMARD, the pooled mean improvement in total work productivity of 7.0 h per week led to an estimated absolute mean decrease in PsA-related indirect costs of €10,688 (US$14,820) per patient, per year. These results provide clinical-, regulatory- and reimbursement decision-makers with valuable data on the societal and socio-economic benefits of b/tsDMARDs in PsA.

The findings of our review are consistent with previous studies, confirming the significant impact of PsA on work outcomes [ 7 ] and how presenteeism is a higher contributor to overall lost work productivity than absenteeism among patients with PsA [ 15 ]. This highlights that when patients with PsA attend work, their productivity is considerably impacted by PsA [ 34 ]. The reported effects of diminished productivity include reduced personal and professional development. Furthermore, as work plays an important role in one’s social life and integration into society, reduced ability to participate in work may increase isolation and have deleterious effects on the wellbeing of patients; decreased work productivity has been linked to decrements in QoL and mental health [ 35 , 36 , 37 ].

PsA has a high cost for society. Published estimations of the annual direct PsA-related health care costs have been reported to be as high as US$1.9 billion [ 8 ]. Indirect costs are estimated to be even greater, accounting for 52 to 72% of total disease-related costs [ 8 ]. In a systematic review and meta-analysis [ 38 ], Kawalec and colleagues estimated that the average annual indirect costs of PsA range from US$1694 to $12,318 (using the friction cost approach) or from US$1751 to $50,271 (using the human capital approach). When using the human capital approach, this range of estimates is higher than our review and can be attributed to differences in the included studies and the scope of indirect costs calculated.

Overall, improvements in productivity were significant and clinically relevant after 24 weeks of b/tsDMARDs, though variation in different domains was observed. It is important to note that the results presented here for improvements after treatment were not adjusted for changes in a comparator or placebo arm; therefore, these improvements correspond to a ‘simple’ before-after analysis and cannot in any way be considered as due to treatment (no causality is claimed). Findings for the absenteeism domain presented mixed results, with several studies not reporting an improvement in scores at 24 weeks. In contrast, all studies reported an improvement in presenteeism at 24 weeks, with a pooled estimated mean change from baseline of − 17.8% p (95% CI − 19.3, − 16.2). This is close to the minimal clinically important difference (MCID) of 20%, which was estimated by Tillett and colleagues [ 39 ]. This reported MCID applies to patient-level data; however, it provides a useful benchmark for our review.

Previous studies have demonstrated associations between productivity and response (minimal disease activity [MDA] or low disease activity according to PsA Disease Activity Score [PASDAS]) [ 10 , 40 ] and found greater improvements in productivity with bDMARDs than with csDMARDs [ 41 ]. Generally, the estimated effects of the b/tsDMARDs in the current analysis were similar with respect to improving a patient’s capacity to work and participate in leisure activities. However, it is important to note that we did not directly compare the drugs as this would necessitate specific statistics such as network meta-analyses, which can provide rankings of relative effects of different treatments, but may also be influenced by heterogeneity between studies, leading to limitations in interpretability in some cases. Studies aimed to measure both the effectiveness and the impact of specific interventions on productivity and HRQoL in patients with PsA may be valuable in supporting optimized treatment selection from a holistic perspective.

This review has several strengths. The evidence was retrieved through a systematic search of the literature according to methodological guidance and reporting and included a recent update to account for the evolving treatment landscape in PsA. Analyses were based on both RCTs and observational studies, allowing for a wider scope than a previous review on this topic, which was limited to RCTs and included only five studies [ 42 ].

Despite these strengths, this review has some limitations. The analysis focused solely on the WPAI, a widely used measure of productivity that has been validated for use among patients with PsA [ 14 , 43 ]. While there is no gold standard measure for assessing productivity in PsA, the WPAI was one of six instruments identified by the OMERACT Worker Productivity Group as a candidate for assessing work productivity based on available evidence regarding psychometric properties (e.g. test–retest reliability, construct validity) [ 44 ]. However, other questionnaires, such as the Work Productivity Scale (WPS) [ 45 ] or the Work Limitations Questionnaire (WLQ) [ 43 , 46 ], may also be used to evaluate work impact. While this review excludes studies using these alternative assessment tools, the selection of a single measure facilitated the comparability of findings and allowed us to conduct a quantitative synthesis of findings. A network meta-analysis was not performed in which comparative treatment effects were estimated. Our approach to the analysis was taken to allow for the inclusion of observational studies that did not have a placebo arm; thus, a limitation is that no causal conclusions on the effect of b/tsDMARDs on work productivity can be derived. Future research using an NMA approach and limiting study inclusion to RCTs may provide more robust estimates.

The variation in reporting of WPAI outcomes required certain assumptions for the meta-analysis. For example, data reported as means and LSM were pooled following a scenario analysis to explore the internal validity of using raw and adjusted means. No major differences were observed between these analyses. As WPAI is often considered a secondary outcome measure, there was a lack of reporting of subgroup analyses related to patient characteristics (e.g. tumour necrosis factor inhibitors experience) and other factors that may contribute to work and/or activity impairment (e.g. pain, fatigue, participation in manual work). This limited our ability to conduct sensitivity analysis or additional analyses on subgroups of interest. This limitation reflects a shortcoming in the body of published evidence and not the methodology of the review itself.

A final limitation is the variation in the methods of estimating the costs attributable to lost productivity due to PsA. While the WPAI focuses on absenteeism and presenteeism, other components of productivity may be considered, such as early retirement due to disease or patients returning to work after successful treatment [ 38 ]. Lost productivity can also be valued using different approaches, with indirect costs typically calculated using the friction cost method or the human capital approach [ 16 ]. We estimated indirect costs using the human capital approach, which may overestimate the indirect costs incurred by employers given the limited amount of published data available. Cost conclusions are further limited by the level of heterogeneity across studies. However, these estimates are a reflection of the larger, societal burden of PsA.

Our review focused on patients with PsA treated with a b/tsDMARD. These patients could be considered as having a more severe form of disease due to failure of first-line treatment with NSAIDs and csDMARDs. Future research could consider productivity among a broader PsA population [ 47 , 48 ].

The review was also structured to consider both RCT and observational evidence. Only two observational studies provided WPAI scores at 24 weeks. These studies reported some of the largest [ 33 ] and lowest [ 21 ] improvements in mean change from baseline WPAI domain scores. This suggests potential variation in productivity outcomes in real-life settings. It is important to note that one study was from the US and the other from Japan, which may have inherent differences in working and productivity. However, it is not feasible to explore this potential “country-effect” due to limited evidence and the fact that both studies had limited sample sizes; therefore, the results should be interpreted with caution.

Measuring work impact should be considered an essential part of the overall assessment of the economic burden and the value assessment of therapies in PsA. This review systematically and comprehensively quantified the impact of PsA on work productivity and impairment using the WPAI among patients treated with a b/tsDMARD. The results demonstrate that patients with PsA suffered from substantial total work productivity impairment but report meaningful improvement after 24 weeks of treatment with a b/tsDMARD. This provides payers and other decision-makers with valuable data to inform decisions about the cost-effectiveness of b/tsDMARDs in PsA.

Availability of data and materials

The data that support the findings of this study are primarily from published literature. Data from BE OPTIMAL (NCT03895203) [ 25 ] may be requested by qualified researchers 6 months after product approval in the US and/or Europe, or global development is discontinued, and 18 months after trial completion. Investigators may request access to anonymized individual patient-level data and redacted trial documents which may include analysis-ready datasets, study protocol, annotated case report form, statistical analysis plan, dataset specifications, and clinical study report. Prior to the use of the data, proposals need to be approved by an independent review panel at www.Vivli.org and a signed data-sharing agreement will need to be executed. All documents are available in English only, for a pre-specified time, typically 12 months, on a password-protected portal. This plan may change if the risk of re-identifying trial participants is determined to be too high after the trial is completed; in this case and to protect participants, individual patient-level data would not be made available.

Abbreviations

Percentage point

Confidence interval

Conventional synthetic

Disease-modifying antirheumatic drugs

Health-related quality of life

International Classification of Functioning, Disability and Health

Least-squares mean

Minimal clinically important difference

Minimal disease activity

Non-steroidal anti-inflammatory drugs

Psoriatic Arthritis Disease Activity Score

Population, Intervention, Comparator, Outcome and Study Design

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

  • Psoriatic arthritis

Randomized controlled trials

Systematic literature review

Targeted synthetic

World Health Organization

Work Limitations Questionnaire

Work Productivity and Activity Impairment Questionnaire

Work Productivity Scale

Husni ME, Merola JF, Davin S. The psychosocial burden of psoriatic arthritis. Semin Arthritis Rheum. 2017;47(3):351–60. https://doi.org/10.1016/j.semarthrit.2017.05.010 .

Article   PubMed   Google Scholar  

Ogdie A, Weiss PF. The epidemiology of psoriatic arthritis. Rheum Dis Clin North Am. 2015;41(4):545–68.

Article   PubMed   PubMed Central   Google Scholar  

Karmacharya P, Ogdie A, Eder L. Psoriatic arthritis and the association with cardiometabolic disease: a narrative review. Ther Adv Musculoskelet Dis. 2021;13:1759720X21998279. https://doi.org/10.1177/1759720X21998279 .

Gudu T, Gossec L. Quality of life in psoriatic arthritis. Expert Rev Clin Immunol. 2018;14(5):405–17. https://doi.org/10.1080/1744666X.2018.1468252 .

Article   CAS   PubMed   Google Scholar  

Gudu T, Kiltz U, de Wit M, Kvien TK, Gossec L. Mapping the effect of psoriatic arthritis using the International Classification of Functioning, Disability and Health. J Rheumatol. 2017;44(2):193–200. https://doi.org/10.3899/jrheum.160180 .

Tillett W, Shaddick G, Askari A, et al. Factors influencing work disability in psoriatic arthritis: first results from a large UK multicentre study. Rheumatology. 2014;54(1):157–62. https://doi.org/10.1093/rheumatology/keu264 .

Tillett W, de Vries C, McHugh NJ. Work disability in psoriatic arthritis: a systematic review. Rheumatology (Oxford). 2012;51(2):275–83. https://doi.org/10.1093/rheumatology/ker216 .

Lee S, Mendelsohn A, Sarnes E. The burden of psoriatic arthritis: a literature review from a global health systems perspective. P t. 2010;35(12):680–9.

PubMed   PubMed Central   Google Scholar  

Gossec L, Walsh JA, Michaud K, et al. Effect of fatigue on health-related quality of life and work productivity in psoriatic arthritis: Findings From a Real-World Survey. J Rheumatol. 2022;49(11):1221–8. https://doi.org/10.3899/jrheum.211288 .

van Hal TW, Mulder MLM, Wenink MH, Vriezekolk JE. Determinants of work and social participation in patients with psoriatic arthritis in the Netherlands: an observational study. BMC Rheumatol. 2022;6(1):49. https://doi.org/10.1186/s41927-022-00279-7 .

Coates L, Gossec L. The updated GRAPPA and EULAR recommendations for the management of psoriatic arthritis: similarities and differences. Joint Bone Spine. 2023;90(1):105469. https://doi.org/10.1016/j.jbspin.2022.105469 .

Systematic Reviews: CRD's guidance for undertaking reviews in health care. University of York. https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf . Accessed 19 May 2022

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. 2021;10(1):89. https://doi.org/10.1186/s13643-021-01626-4 .

Reilly MC, Zbrozek AS, Dukes EM. The validity and reproducibility of a work productivity and activity impairment instrument. Pharmacoeconomics. 1993;4(5):353–65. https://doi.org/10.2165/00019053-199304050-00006 .

Kruntorádová K, Klimeš J, Šedová L, Štolfa J, Doležal T, Petříková A. Work productivity and costs related to patients with ankylosing spondylitis, rheumatoid arthritis, and psoriasis. Value in Health Reg Issues. 2014;4:100–6. https://doi.org/10.1016/j.vhri.2014.07.008 .

Article   Google Scholar  

van den Hout WB. The value of productivity: human-capital versus friction-cost method. Ann Rheum Dis. 2010;69(Suppl 1):i89-91. https://doi.org/10.1136/ard.2009.117150 .

Balduzzi S, Rucker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. 2019;22(4):153–60. https://doi.org/10.1136/ebmental-2019-300117 .

Labour costs levels by NACE Rev. 2 activity. European Commission. Updated 17 May 2022. https://ec.europa.eu/eurostat/databrowser/view/LC_LCI_LEV__custom_709491/bookmark/table?lang=en&bookmarkId=eb78a6c4-aa9b-4210-ad86-6f9d003e1952 . Accessed Aug 18 2022

Low and high pay in the UK: 2021 - The distribution of hourly earnings of high-paid and low-paid jobs and jobs paid below the National Minimum Wage. Office for National Statistics. Updated 26 October 2021. https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/earningsandworkinghours/bulletins/lowandhighpayuk/2021 . Accessed 08 Aug 2022

Total compensation cost per hour worked for civilian workers (Q4, 2021). U.S. Bureau of Labor Statistics. https://www.bls.gov/ncs/ect/ . Accessed 11 Jan 2023

Mease PJ, Blachley T, Glynn M, et al. Secukinumab improves clinical and patient-reported outcomes at 6 months among patients with psoriatic arthritis in the US-based corrona psoriatic arthritis/spondyloarthritis (PSA/SPA) registry. Conference Abstract. Ann Rheum Dis. 2020;79(SUPPL 1):1169. https://doi.org/10.1136/annrheumdis-2020-eular.1014 .

Strand V, Mease PJ, Soriano ER, et al. Improvement in patient-reported outcomes in patients with psoriatic arthritis treated with upadacitinib versus placebo or adalimumab: results from SELECT-PsA 1. Rheumatol Ther. 2021 https://doi.org/10.1007/s40744-021-00379-9

Kristensen LE, Soliman AM, Papp K, et al. Risankizumab improved health-related quality of life, fatigue, pain and work productivity in psoriatic arthritis: results of KEEPsAKE 1. Rheumatology. 2022:keac342. https://doi.org/10.1093/rheumatology/keac342

Ostor AJK, Soliman AM, Papp KA, et al. Improved patient-reported outcomes in patients with psoriatic arthritis treated with risankizumab: analysis of the Phase 3 trial KEEPsAKE 2. RMD Open. 2022;8(2):e002286.

McInnes IB, Asahina A, Coates LC, et al. Bimekizumab in patients with psoriatic arthritis, naive to biologic treatment: a randomised, double-blind, placebo-controlled, phase 3 trial (BE OPTIMAL). Lancet. 2023;401(10370):25–37. https://doi.org/10.1016/S0140-6736(22)02302-9 .

Curtis J, McInnes I, Rahman P, et al. Guselkumab provides sustained improvements in work productivity and non-work activity in psoriatic arthritis patients: Phase 3 results through year 1. Conference Abstract. Int J Rheum Dis. 2021;24(SUPPL 2):72–3. https://doi.org/10.1111/1756-185X.14199 .

Strand V, Van den Bosch F, Ranza R, et al. Patient-reported outcomes in psoriatic arthritis patients with an inadequate response to biologic disease-modifying antirheumatic drugs: SELECT-PsA 2. Rheumatol Ther. 2021; https://doi.org/10.1007/s40744-021-00377-x

Strand V, Mease P, Gossec L, et al. Secukinumab improves patient-reported outcomes in subjects with active psoriatic arthritis: results from a randomised phase III trial (FUTURE 1). J Article Ann Rheum Dis. 2017;76(1):203–7.

Article   CAS   Google Scholar  

Tillett W, Lin CY, TrevelinSprabery A, Birt JA, Kavanaugh A. Clinically meaningful improvement in work productivity loss in active psoriatic arthritis: post-hoc analysis of SPIRIT-P1 and SPIRIT-P2 trials. Clinical Trial, Phase III Randomized Controlled Trial. Clin Exp Rheum. 2020;38(6):1227–30.

Google Scholar  

Kavanaugh A, Marzo-Ortega H, Vender R, et al. Ixekizumab improves patient-reported outcomes in patients with active psoriatic arthritis and inadequate response to tumour necrosis factor inhibitors: SPIRIT-P2 results to 52 weeks. Randomized Controlled Trial. Clin Exp Rheum. 2019;37(4):566–74.

Gottlieb AB, Strand V, Kishimoto M, et al. Ixekizumab improves patient-reported outcomes up to 52 weeks in bDMARD-naive patients with active psoriatic arthritis (SPIRIT-P1). Randomized Controlled Trial; Research Support, Non-U.S. Gov’t. Rheumatol. 2018;57(10):1777–88. https://doi.org/10.1093/rheumatology/key161 .

McInnes IB, Mease PJ, Ritchlin CT, et al. Secukinumab sustains improvement in signs and symptoms of psoriatic arthritis: 2 year results from the phase 3 FUTURE 2 study. Article. Rheumatology (oxford, england). 2017;56(11):1993–2003.

Nakagawa H, Tanaka Y, Sano S, et al. Real-world postmarketing study of the impact of adalimumab treatment on work productivity and activity impairment in patients with psoriatic arthritis. Adv Ther. 2019;36(3):691–707. https://doi.org/10.1007/s12325-018-0866-y .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Drouet JMS, Gossec L. Work-related issues and a high burden of disease are frequent in axial spondyloarthritis: what should we be doing better? J Rheumatol. 2023 https://doi.org/10.3899/jrheum.221252

Garrido-Cumbrera M, Hillmann O, Mahapatra R, et al. Improving the management of psoriatic arthritis and axial spondyloarthritis: roundtable discussions with healthcare professionals and patients. Rheumatol Ther. 2017;4(2):219–31. https://doi.org/10.1007/s40744-017-0066-2 .

Haglund E, Bremander A, Bergman S, Jacobsson LT, Petersson IF. Work productivity in a population-based cohort of patients with spondyloarthritis. Rheumatology (Oxford). 2013;52(9):1708–14. https://doi.org/10.1093/rheumatology/ket217 .

Ramonda R, Marchesoni A, Carletto A, et al. Patient-reported impact of spondyloarthritis on work disability and working life: the ATLANTIS survey. Arthritis Res Ther. 2016;18:78. https://doi.org/10.1186/s13075-016-0977-2 .

Kawalec P, Malinowski KP. The indirect costs of psoriatic arthritis: systematic review and meta-analysis. Expert Rev Pharmacoecon Outcomes Res. 2015;15(1):125–32. https://doi.org/10.1586/14737167.2015.965154 .

Tillett W, Lin C-Y, Zbrozek A, Sprabery AT, Birt J. A threshold of meaning for work disability improvement in psoriatic arthritis measured by the work productivity and activity impairment questionnaire. Rheumatol Ther. 2019;6(3):379–91. https://doi.org/10.1007/s40744-019-0155-5 .

Wervers K, Luime JJ, Tchetverikov I, et al. Time to minimal disease activity in relation to quality of life, productivity, and radiographic damage 1 year after diagnosis in psoriatic arthritis. Arthritis Res Ther. 2019;21(1):25. https://doi.org/10.1186/s13075-019-1811-4 .

Tillett W, Shaddick G, Jobling A, et al. Effect of anti-TNF and conventional synthetic disease-modifying anti-rheumatic drug treatment on work disability and clinical outcome in a multicentre observational cohort study of psoriatic arthritis. Rheumatology (Oxford). 2017;56(4):603–12. https://doi.org/10.1093/rheumatology/kew433 .

Iragorri N, Hofmeister M, Spackman E, Hazlewood GS. The effect of biologic and targeted synthetic drugs on work- and productivity-related outcomes for patients with psoriatic arthritis: a systematic review. J Rheumatol. 2018;45(8):1124–30. https://doi.org/10.3899/jrheum.170874 .

Prasad M, Wahlqvist P, Shikiar R, Tina Shih Y-C. A review of self-report instruments measuring health-related work productivity. PharmacoEconomics. 2004;22(4):225–44. https://doi.org/10.2165/00019053-200422040-00002 .

Verstappen SMM, Lacaille D, Boonen A, et al. Considerations for evaluating and recommending worker productivity outcome measures: an update from the OMERACT Worker Productivity Group. J Rheumatol. 2019;46(10):1401–5. https://doi.org/10.3899/jrheum.181201 .

Osterhaus JT, Purcaru O. Discriminant validity, responsiveness and reliability of the arthritis-specific Work Productivity Survey assessing workplace and household productivity in patients with psoriatic arthritis. Arthritis Res Ther. 2014;16(4):R140. https://doi.org/10.1186/ar4602 .

Kennedy M, Papneja A, Thavaneswaran A, Chandran V, Gladman DD. Prevalence and predictors of reduced work productivity in patients with psoriatic arthritis. Clin Exp Rheumatol May-Jun. 2014;32(3):342–8.

CAS   Google Scholar  

Albrecht K, Regierer AC, Strangfeld A, Marschall U, Callhoff J. High burden of polypharmacy and comorbidity in persons with psoriatic arthritis: an analysis of claims data, stratified by age and sex. RMD Open. 2023;9(1) https://doi.org/10.1136/rmdopen-2022-002960

Puche-Larrubia MA, Ladehesa-Pineda L, Lopez-Montilla MD, et al. Differences between early vs. Late-onset of psoriatic arthritis: data from the respondia and regisponser registries. Joint Bone Spine. 2023:105563. https://doi.org/10.1016/j.jbspin.2023.105563

Download references

Acknowledgements

The authors acknowledge Heather Edens (UCB Pharma, Smyrna, GA, USA) for publication coordination and editorial assistance and Elizabeth Hubscher (Cytel, Inc., Waltham, MA) for medical writing and editorial assistance based on the authors’ input and direction. The authors also acknowledge Costello Medical for administrative and submission support.

This study was sponsored by UCB Pharma, including support for third-party writing assistance.

Author information

Authors and affiliations.

Sorbonne Université, INSERM, Institut Pierre Louis d’Epidémiologie et de Santé Publique, Paris, France

Laure Gossec

Rheumatology Department, Pitié-Salpêtrière Hospital, AP-HP, 47-83 Bd de l’Hôpital, Paris, 75013, France

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Canada

Brittany Humphries

Cytel Inc, Ottawa, Canada

Cytel Inc, London, UK

Megan Rutherford

UCB Pharma, Colombes, France

Vanessa Taieb

UCB Pharma, Brussels, Belgium

Damon Willems

Department of Life Sciences, Royal National Hospital for Rheumatic Diseases, Centre for Therapeutic Innovation University of Bath, Bath, UK

William Tillett

You can also search for this author in PubMed   Google Scholar

Contributions

DW, VT, BH, and MR conceptualized and designed the study. BH and MR were involved in the data collection and analysis. All authors were involved in the interpretation of the results. All authors were involved in preparation of the manuscript, critical review and approved the final version.

Corresponding author

Correspondence to Laure Gossec .

Ethics declarations

Ethics approval and consent to participate.

This SLR and meta-analysis was based on published literature and de-identified results of the BE OPTIMAL (NCT03895203) trial [ 25 ]; therefore, additional ethics approval was not required. This paper was developed in accordance with the Good Publication Practice (GPP3) guidelines ( https://www.ismpp.org/gpp3 ).

Consent for publication

All authors agreed to the publication of this manuscript.

Competing interests

LG: Received grants or contracts from Pfizer, Sandoz and UCB, consulting fees from AbbVie, Bristol-Myers Squibb, Celltrion, Galapagos, Janssen, Novartis, Pfizer and UCB, honoraria for lectures from AbbVie, Amgen, Galapagos, Gilead, Janssen, Lilly, MSD, Novartis, Pfizer, Sandoz and UCB, support for attending meetings and/or travel from MSD, Novartis, and Viatris, and has received medical writing support from AbbVie, Janssen, Pfizer and UCB. All unrelated to the present study.

BH: Consultant to UCB Pharma.

MR: Consultant to UCB Pharma.

VT: Employee of UCB Pharma.

DW: Employee and shareholder of UCB.

WT: Research funding, consulting or speaker fees from AbbVie, Amgen, Celgene, Eli Lilly, Janssen, Pfizer and UCB. Consulting and speaker fees from AbbVie, Amgen, Celgene, Eli Lilly, Janssen, MSD, Novartis, ONO Pharma, Pfizer and UCB.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

PICOS eligibility criteria. Figure S1. Meta-analysis of percent mean CFB in absenteeism scores for placebo at 24 weeks. Figure S2. Meta-analysis of percent mean CFB in presenteeism scores for placebo at 24 weeks. Figure S3. Meta-analysis of percent mean CFB in total work productivity scores for placebo at 24 weeks. Figure S4. Meta-analysis of percent mean CFB activity impairment scores for placebo at 24 weeks. Table S2. Data sources. Table S3. Search strategy with results. Table S4. Risk of bias assessment.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gossec, L., Humphries, B., Rutherford, M. et al. Improvement in work productivity among psoriatic arthritis patients treated with biologic or targeted synthetic drugs: a systematic literature review and meta-analysis. Arthritis Res Ther 26 , 50 (2024). https://doi.org/10.1186/s13075-024-03282-0

Download citation

Received : 04 May 2023

Accepted : 30 January 2024

Published : 15 February 2024

DOI : https://doi.org/10.1186/s13075-024-03282-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Work productivity
  • Productivity cost

Arthritis Research & Therapy

ISSN: 1478-6362

published literature review

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • For authors
  • Browse by collection
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Volume 14, Issue 2
  • Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • http://orcid.org/0000-0001-6546-0778 Li Jiu 1 ,
  • Michiel Hartog 1 ,
  • Junfeng Wang 1 ,
  • Rick A Vreman 1 ,
  • Olaf H Klungel 1 ,
  • http://orcid.org/0000-0002-8782-0698 Aukje K Mantel-Teeuwisse 1 ,
  • Wim G Goettsch 1 , 2
  • 1 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences , Utrecht University , Utrecht , Netherlands
  • 2 National Health Care Institute , Diemen , Netherlands
  • Correspondence to Dr Wim G Goettsch; w.g.goettsch{at}uu.nl ; Dr Junfeng Wang; j.wang5{at}uu.nl ; Dr Junfeng Wang; j.wang5{at}uu.nl

Objectives We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.

Design Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.

Data sources Systematic search: Medline; Snowballing: starting from three articles (D’Andrea et al , Quigley et al and Faria et al ); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.

Eligibility criteria for selecting studies We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.

Data extraction and synthesis Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.

Results Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.

Conclusion Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.

OSF registration number OSF registration DOI ( https://doi.org/10.17605/OSF.IO/KCSGX ).

  • Systematic Review
  • EPIDEMIOLOGY
  • HEALTH ECONOMICS

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2023-075173

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

This literature review identified 49 appraisal tools for non-randomised studies of interventions, through both the systematic approach (ie, database search) and the non-systematic approaches (ie, snowballing and grey literature search).

Our study compared sufficient descriptions of appraisal tools at quality-item levels, for either methodological quality or reporting.

We only searched health technology assessment agencies for grey literature, so some tools only mentioned by clinical guideline or regulatory organisations might have been overlooked.

Usefulness of categorising a quality item as ‘sufficient’ or ‘brief’ for each tool, based on whether an explanation was provided for the criteria, has not been tested by previous studies.

Introduction

Real-world data (RWD) generally refer to data collected during routine clinical practice, but their definitions could vary in settings. 1 According to Makady et al , one of the RWD definitions is data collected without interference with treatment assignment. 1 RWD that fit this definition are normally analysed in non-randomised studies of interventions (NRSIs), which estimate effectiveness of a health intervention without randomising intervention groups. 2 3

NRSIs provide evidence on clinical and cost-effectiveness of health interventions for decision-making, in clinical and health technology assessment (HTA) settings. 4–9 For example, NRSIs could inform clinicians on what diagnosis or treatment strategies to adopt. 4 5 Also, with NRSIs, HTA agencies could gain more certainty on validity of evidence from randomised controlled trials (RCTs), when deciding on which health intervention to reimburse and on which pricing strategy to adopt. 6 7 Also, HTA stakeholders could exploit NRSIs to evaluate highly innovative or complex interventions, for which RCTs may be considered infeasible or unethical. 8 9 Generally speaking, NRSIs have become increasingly useful, as they complement and sometimes replace RCTs, when RCTs are scarce or even infeasible to conduct. 2 10

However, the usefulness of NRSIs is often questioned due to quality concerns, in terms of risk of bias (RoB) and reporting. According to the Cochrane Handbook, NRSIs have higher RoB than RCTs and are vulnerable to various types of bias, such as confounding, selection and information bias. 11 Also, the Professional Society for Health Economics and Outcomes Research (ISPOR) published a report in 2020, which stated that insufficient reporting on how an NRSI was generated was a major barrier for decision-makers to adopt NRSIs. 12

To address NRSI’s quality concerns and to build decision-makers’ confidence, NRSIs need to be rigorously appraised, and this rationalises the development and use of appraisal tools. According to systematic reviews of appraisal tools for NRSIs, tens of tools have been developed in the past five decades. 13–15 The growing number of tools has then brought a new challenge to users: how to select the best tool. To address this challenge, previous reviews have summarised quality items (ie, a group of criteria or signalling questions for methodological quality or reporting) and compared whether existing tools addressed these items. 13–15 Some example items include ‘measurement of outcomes’, ‘loss to follow-up bias’, ‘inclusion and exclusion criteria of target population’, ‘sampling strategies to correct selection bias’, etc. 13 In addition, these reviews provided some general recommendations on tool selection, such as referring to multiple tools for quality appraisal. 14 However, information is still lacking on to what extent the tools address each quality item and the heterogeneity of tools at the quality-item level. To take outcome measurement as an example, the Academy of Nutrition and Dietetics Quality Criteria (ANDQ) checklist mentions that outcomes should be measured with ‘standard, valid and reliable data collection instruments, tests and procedures’ and ‘at an appropriate level of precision’. 16 In contrast, the Good ReseArch for Comparative Effectiveness (GRACE) checklist considers the ‘valid and reliable’ measurement as ‘objective rather than subject to clinical judgement’ 17 ; while the Risk Of Bias In Non-randomised Studies—of Interventions (ROBINS-I) checklist interprets the ‘standard’ way as ‘comparable across study groups’ and ‘valid and reliable’ as low detection bias without ‘systematic errors’ in outcome measurement. 18 In summary, the heterogeneity in level of detail with which a tool addresses a quality item and the heterogeneity in content and format of signalling questions can pose a challenge when tools are selected, or even merged.

Hence, our study aimed to summarise and compare signalling questions or criteria in the tools provided at the quality-item level, through a content analysis. This research was performed as part of the HTx project. 19 The project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825162.

To ensure credibility of the review and the content analysis, we registered a study protocol in the OSF registry (registration DOI: https://doi.org/10.17605/OSF.IO/KCSGX ) on 30 June 2022. The OSF registry is an online repository that accepts registration of all types of research projects, including reviews and content analyses. 20

Patient and public involvement

Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

In our study, appraisal tools refer to tools, guidelines, instruments or standards that provide guidance on how to report or assess any quality concern of NRSIs. NRSIs, according to the Cochrane Handbook, refer to any quantitative study estimating the effectiveness of an intervention without randomisation to allocate patients to intervention groups. 2 According to Makady et al , data collected in such NRSIs belong to the second category of RWD, that is, those collected without interference with treatment assignment, patient monitoring or follow-up, or selection of study population. 1

Search strategy

To identify appraisal tools for NRSIs from various potential sources, we adopted three approaches. A diagram illustrating how the three approaches complemented each other is shown in online supplemental appendix 1 .

Supplemental material

Database search.

In the first approach, we conducted a systematic review to identify articles on appraisal tools, through a database search using Medline. Since D’Andrea et al have already conducted a systematic review to identify appraisal tools for all types of non-randomised studies published before November 2019, 13 we updated their review by searching for articles published between November 2019 and April 2022, with their strings.

Snowballing

In the second approach, we searched for published reviews on appraisal tools for NRSIs. To identify all published reviews, we adopted a snowballing approach described by Wohlin. 21 Snowballing refers to using the citations of articles to identify additional articles, and it is considered a good extension of a database search. 21 To implement the snowballing approach, three researchers (LJ, MH and JW) first conducted a pilot search of articles using Google Scholar, reviewed full-text, judged eligibility through a group discussion, then identified three reviews (ie, those by D’Andrea et al , 13 Quigley et al 14 and Faria et al 15 ). Next, the three reviews were used as a starting set and were uploaded to the website Connected Papers, which provides an online tool for snowballing. 22 With each uploaded review, Connected Papers analysed approximately 50 000 articles and finally returned 40 articles with the highest level of similarity, based on factors such as overlapping citations. After judging eligibility of the returned articles, eligible articles were uploaded to the website Connected Papers for a second round of snowballing.

Grey literature

In the third approach, we searched for grey literature on the websites of European HTA agencies. Our rationale was that some appraisal tools may exist in the format of grey literature, such as agency reports and technical support documents. The list of European HTA agencies was derived from the International Network of Agencies for Health Technology Assessment. 23 On each agency website, two researchers (MH and LJ) independently searched for grey literature with four concepts, respectively, ‘quality’, ‘RoB’, ‘appraisal’ and ‘methodology’. For each concept, only the first 10 hits sorted by relevance, if optional, were included (ie, a maximum of 40 hits for each website).

Eligibility criteria for articles and grey literature to identify relevant tools

An article or grey literature document was included if it described one or more appraisal tools. It was excluded if it only described tools for RCTs or only described tools for diagnostic, prognostic, qualitative or secondary studies (eg, systematic reviews and cost-effectiveness analyses). We only included articles identified through the database search and snowballing if published in English, while included grey literature could be published in all languages, as many HTA agencies tend to only use languages of their nations. Relevant documents obtained through this approach were translated using Google Translate.

The process of identifying studies and appraisal tools

Two researchers (MH and LJ) independently scanned all titles and abstract of the identified hits, then reviewed the full-text with Rayyan 24 and Excel. After identifying the eligible studies, one researcher (MH) extracted the name of the tools and downloaded them by tracking study citations. A pilot search with Google was conducted to ensure we downloaded the most up-to-date version. Next, two researchers (MH and LJ) independently reviewed full-text and judged eligibility of the tools. An appraisal tool was included if it (1) was designed for non-randomised studies, (2) was used for assessing either methodological quality or reporting and (3) was developed or updated after 2002. A tool was excluded if it was designed for non-randomised studies of exposures which were not controlled by investigators (eg, diets). All discrepancies were solved through discussion among the three researchers (MH, LJ and JW).

One researcher (MH) extracted tool characteristics using a prespecified Excel form. The data items included publication year, tool format (eg, checklist or rating scale), targeted study design (eg, all NRSIs, cohort studies, etc), target interventions (eg, all or surgical interventions), originality (ie, whether a tool was developed based on an existing tool) and scope. The scope referred to whether the tools were designed for assessing methodological quality (eg, RoB and external validity) and/or for ensuring adequate reporting of research details that could be used for assessing methodological quality. 25

For the content analysis, we adopted both deductive and inductive coding techniques. 26 First, we derived a list of candidate quality items from the three reviews, the starting set for the snowballing. 13–15 Then, in a pilot coding process, we reviewed all identified appraisal tools and judged whether a candidate quality item was described. After the pilot coding, we summarised signalling questions or criteria that were not covered by the candidate items and coded them as new items. After updating the list of candidate items, three researchers (JW, LJ and MH) finalised the items in four group meetings. During the meetings, we merged items with overlapping content, split items containing too much content and renamed items so they could be self-explanatory.

To score whether and to what extent a quality item was described by a tool, we again reviewed all identified tools. If an item was described by a tool in one or several signalling questions, we judged whether the question(s) was related to methodological quality, reporting or both, independently of what original studies claimed to be. Additionally, we judged whether an item was described sufficiently or briefly. A description was scored as ‘brief’, if the corresponding signalling question(s) did not explain how to improve or assess methodological quality or specify elements needed for reporting. For example, ‘outcomes should be measured appropriately’ or ‘outcome measurement should be adequately described’ are ‘brief’ descriptions, if no additional explanations were provided. The scoring process was independently conducted by two researchers (LJ and MH) using NVivo V.12, and all discrepancies were solved through discussion between the two.

Tool selection

As shown in figure 1 , we identified 1738 articles after removing duplicates and excluded 1645 articles after subsequently reviewing titles, abstracts and full-text. From the remaining 27 eligible studies, we identified 417 appraisal tools. After removing duplicates and reviewing full-texts, we included 49 tools which met our criteria. References of the included studies and appraisal tools are shown in online supplemental appendix 2 and 3 , respectively.

  • Download figure
  • Open in new tab
  • Download powerpoint

Flow chart for the inclusion and exclusion of appraisal tools for non-randomised studies of interventions

Characteristics of appraisal tools

As shown in table 1 , 18 (37%) tools were published between 2002 and 2010, while 31 (63%) tools were published thereafter. Among these, 30 (61%), 6 (12%) and 5 (10%) tools were designed for addressing methodological quality, reporting and both, respectively, while 7 (14%) tools did not report intended use of the tools. About three quarters of the tools were designed for all types of NRSIs, while others were designed for one or several NRSI types, such as cohort (16%) and case–control studies (16%). Regarding sources, 44 (90%) tools were described in articles that developed a tool, in grey literature (eg, online checklist or report), or in both, while the other five tools were extended from existing tools, when researchers conducted systematic reviews on non-randomised studies. Finally, 9 (18%) tools were designed for specific interventions or diseases while all other tools were generic in nature.

  • View inline

Characteristics of the 49 included appraisal tools for non-randomised studies of interventions

Quality domains and items

We identified 44 criteria to describe study quality from three previous reviews. 13–15 After merging criteria with similar content (eg, ‘Follow-up’ and ‘Loss to follow-up’) and incorporating items into those with wider meanings (eg, ‘Loss to follow-up bias’ into ‘Loss to follow-up’), we obtained a list of 18 items. After the pilot coding, we summarised criteria of appraisal tools not covered by the 18 items into another 8 items. According to the general order of conducting an NRSI (eg, study design and data analysis, etc.), these 26 items were categorised into four domains: Study design, Data quality, Data analysis and Results presentation. As shown in figure 2 and table 2 , all domains and most items were addressed by existing tools, but for each item, the number of tools with sufficient descriptions was relatively small. For three items in methodology and nine items in reporting, less than five tools addressed them, and none of the tools sufficiently described them.

The extent to which the appraisal tools addressed quality items on methodological quality or reporting.

Overview of the 4 domains and 26 quality items, with numbers and proportions of appraisal tools that addressed or sufficiently described them

Figure 2 illustrates whether and to what extent the identified tools addressed quality items in terms of methodological quality or reporting. The 26 columns represent the 26 quality items as shown in table 2 . The ranking of appraisal tools based on the number of items addressed or sufficiently described, either general or segmented by quality domains, is shown in online supplemental appendix 4–6 . Regarding methodological quality, Research Triangle Institute Item Bank (RTI Item Bank) 27 addressed (n=20) and sufficiently described (n=18) the highest number of items. In addition, the tools that ranked both top 10, based on number of items addressed or sufficiently described, included Methodology Index for Non-randomized Studies (MINORS), 28 , Faillie et al , 29 ROBINS-I, 18 ANDQ, 16 Comparative Effectiveness Research Collaborative Initiative Questionnaire (CER-CI) 30 and Joanna Briggs Institute’s Critical Appraisal Tool (JBI). 31 These tools addressed at least 10 items and sufficiently described at least 5 items. In the study-design domain, RTI Item Bank 27 sufficiently described the most items (n=7), while in the Data quality domain, RTI Item Bank 27 and MINOR 28 ranked the top two, which sufficiently described at least 5 of the 10 items. In the Data analysis domain, only Faillie et al 29 and Handu et al 32 sufficiently described all the three included items. In the Results presentation domain, the relevant two items sufficiently described by Faillie et al 29 and Handu et al , 32 and ANDQ. 16 Regarding reporting, STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) 33 addressed (n=17) and sufficiently described (n=14) the highest number of items. Also, the tools that ranked both top 10, based on the two criteria, included Transparent Reporting of Evaluations with Non-randomized Designs (TREND), 34 the tool by Genaidy et al , 35 REporting of studies Conducted using Observational Routinely-collected Data (RECORD), 36 European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP), 36 International Society of Pharmacoepidemiology (ISPE), 37 the tool by Tseng et al 38 and Joint Task Force between the International Society for Pharmacoepidemiology and the International Society for Pharmacoeconomics and Outcomes Research (ISPE-ISPOR). 39 These tools at least addressed and sufficiently described seven and three quality items, respectively. In all the four quality domains, STROBE 32 sufficiently described the (equally) most items, compared with other tools. Besides, in the Study design domain, ENCePP 36 and RECORD 40 sufficiently described at least 4 of the 11 items, while in the Data quality domain, TREND 34 and Genaidy et al 35 sufficiently described at least 4 of the 10 items. In the Data analysis and Results presentation domain, STROBE was the only tool that sufficiently described two of the thee items, while 7 and 12 other tools sufficiently described only one item, respectively.

Methodological quality

Among the four domains, the Study design domain was the most ignored domain by appraisal tools, as only 4 of the 11 relevant items were described with sufficient details by more than four tools. More specifically, no tool described methodological quality on Ethical approval or Study objective with sufficient detail. For example, the guidelines manual of the National Institute for Health and Care Excellence (NICE) stated that: “The study addresses an appropriate and clearly focused question”. 41 The tool did not explain the standard of appropriateness and clearness.

In addition, although one-third of tools discussed what a good study design was, only three tools defined the goodness. 42–44 For example, the NHS Wales Questions to Assist with the Critical Appraisal of a Cross-Sectional Study (NHS Wales) stated that the choice of study design should be appropriate to the research question and ensure the reliability of study results. 44 Outcome selection was also ignored by most tools, as only three tools (ie, RTI Item Bank, 27 MINORS 28 and the tool by Faillie et al 29 ) sufficiently described them. Similarly, only RTI Item Bank, 27 the tool by Genaidy et al 35 and NICE 41 sufficiently described the item Outcome definition. For example, Genaidy et al 35 stated that a definition was clear only if ‘definitions of all outcome variables were clearly described’, and was partially clear if not all variables were clearly described, but ‘sufficient information was provided for the reader to understand the intent’. 35 Other items that were rarely addressed or insufficiently described included Intervention definition and Data source. The respective tools with sufficient descriptions included SURE, 45 ROBINS-I, 18 MINORS, 28 CER-CI, 30 GRACE 17 and the tools described by Faillie et al . 29

The Data quality domain was ignored by most tools, as 4 of the 10 relevant items were sufficiently addressed by less than three tools. In particular, the item Intervention measurement and Length of follow-up were sufficiently addressed by none of the tools, JBI was the only tool stating that method of measuring interventions should be clearly reported, 31 while 19 tools addressing Intervention measurement only focused on methodological quality. Some other items that were rarely addressed or insufficiently addressed included Outcome blinding and Loss to follow-up. Regarding Outcome blinding, only three tools provided sufficient descriptions, that is, MINORS, TREND and ISPE. 28 34 37 . Similarly, only the tool by Genaidy et al , 35 TREND and STROBE sufficiently described Loss to follow-up. 32 35 36

We conducted a review of appraisal tools for NRSIs and assessed whether and how sufficiently these tools addressed quality concerns, in terms of methodological quality or reporting, in 4 quality domains and across 26 items. Our study identified 49 tools and showed that the RTI Item Bank and STROBE were most comprehensive, with the highest number of items addressed and sufficiently described, respectively, on methodological quality and reporting. However, none of the tools addressed concerns in all items, not even briefly. The items least addressed for methodological quality included Outcome selection, Outcome definition and Ethical approval, and for reporting included Intervention selection, Intervention measurement and Length of follow-up.

To our knowledge, this is the first study that compared level of sufficient descriptions of appraisal tools at quality-item levels. Previous reviews also compared appraisal tools but from different perspectives. D’Andrea et al identified 44 tools evaluating the comparative safety and effectiveness of medications, and only assessed whether or not these tools addressed methodological quality in eight domains. 13 In another review, Ma et al elaborated for what types of study design a tool was suited. 46 For example, for cohort studies, they encouraged using five tools, while discouraged the use of another two. However, they did not clarify why some tools were more suitable than the others. Quigley et al identified 48 tools for appraising quality of systematic reviews of non-randomised studies, listed the five most commonly used tools and assessed whether they addressed the 12 quality domains, such as ‘appropriate design’ and ‘appropriate statistical analysis’. 14 Although the tools were compared using different criteria, some results were consistent among all studies. For example, both D’Andrea et al 13 and our study found that intervention measurement, outcome measurement and confounding were frequently addressed by existing tools. Also, Ma et al 46 and Quigley et al 14 both recommended ROBINS-I, MINORS and JBI, and all these tools ranked top 10 for addressing and sufficiently describing methodological quality in our study. With detailed information on level of sufficient descriptions of appraisal tools at the quality-item level, we add value to previous reviews by listing quality concerns that such commonly recommended tools could not adequately address.

We also found some discrepancies in the tools identified or recommended. For example, of the 44 tools identified by D’Andrea et al , 13 27 were published between 2003 and 2019; while in our study, 47 were identified as published between 2003 and 2019. This discrepancy could be explained by additional tools identified through other reviews, tools from grey literature and differences in eligibility criteria (eg, exclusion of non-pharmacological interventions or assessing only one or a few specific types of bias). Another discrepancy was that some tools that ranked top in our study were less recommended by previous reviews, such as RTI Item Bank 27 and the tool by Faillie et al 29 for methodological quality and by Genaidy et al 35 for reporting. This might be explained by the novel criteria (ie, how sufficiently quality items were addressed) we used to evaluate these tools.

We discovered that, with information on how sufficiently a tool described a quality item, tool users might broaden their horizons on quality concerns of non-randomised studies to be considered. For example, if ROBINS-I 18 is used for assessing methodological quality, the quality concerns known to users will be RoB in eight domains (eg, confounding and selection bias). However, as shown in figure 2 , quality concerns in 16 items (eg, Intervention selection and Outcome definition) may not be sufficiently described in ROBINS-I but in other tools, such as RTI Item Bank, 27 the NICE checklist 41 and the tool by NHS Wales. 44 Similarly, if users check the ENCePP 36 and ISPE tools, 37 in addition to STROBE, for reporting quality concerns, they may more comprehensively understand concerns on Ethical approval, Outcome definition, Study objective and Data source. Tool users who may benefit from such information are not only researchers who conduct non-randomised studies and decision-makers who assess study quality, but also tool developers who may identify a research gap.

While the needs of tool users may vary, they could all be somewhat satisfied by our research. For example, it is important for researchers to ensure sufficient reporting of the strengths and weaknesses of an NRSI, as such information will be ultimately used for determining the eligibility of their studies for a decision-making. 32 47 For HTA agencies, NRSIs can be used to extrapolate long-term drug effectiveness and to identify drug-related costs, and a deep and consistent understanding of how to assess NRSI quality among the agencies is important for promoting the use of RWD. 48 For regulators, a comprehensive understanding of how to evaluate NRSI quality may promote a structured pattern of using RWD to support drug regulation. 49 While researchers focus more on reporting, and decision-makers (eg, HTA agencies) have emphasis on methodological quality, we suggest all users pay attention to the linkage between methodology and reporting for each quality item, as illustrated in our research, as it could help understand the necessity of investigating each item.

Another finding of our research was that whether and to what extent a quality concern was addressed by a tool partly depended on the tool purpose. For example, the GRACE checklist was designed as a ‘screening tool’ to exclude studies that did not meet basic quality requirements, 17 and ROBINS-I focused on RoB, rather than all methodological quality issues, such as appropriateness of study objectives or statistical analyses for patient matching. 18 Some tools, such as JBI Cohort, 31 were specific to a type of study design. While they addressed less than half of quality items defined in our research, they were proven robust in many studies. 14 Additionally, for several quality items we found some heterogeneity in content of signalling questions or criteria among the tools with sufficient description. For example, to assess methodological quality of sensitivity analysis, CER-CI 30 stated that key assumptions or definitions of outcomes should be tested, while the tool by Viswanathan et al 50 emphasised the importance of reducing uncertainty in individual judgements. Given the heterogeneity of tools, we suggest users following a two-step approach when selecting a tool. First, users may narrow down the scope of tools based on their own needs, for example, excluding tools for a different study design. This step could be achieved by referring to synthesised results and recommendations from existing reviews. 13 14 Second, users could use the overview we provide ( figure 2 ) to see which tool(s) could provide complementary insights the tool of their first choice is lacking.

Furthermore, we found that appraisal tools designed for specific interventions had potential to be transferred for general interventions. In our research, the tools described by Tseng et al 38 and Blagojevic et al 51 and ANDQ 16 were originally designed for a surgical intervention, knee osteoarthritis and for the field of diabetes, respectively. All these tools ranked top 15 in our study for addressing either methodological quality or reporting ( online supplemental file appendix 4–6 ), and many of their criteria could be generalisable. For example, Tseng et al 38 stated that interventions could be adequately described with specifically referenced articles ( online supplemental file appendix 7 ). 38 Though such tools could be transferred, they often used disease-or-intervention-specific concepts in their criteria, which might be adjusted before being applied more widely.

Moreover, we noticed that, some quality items were less frequently addressed, such as Study objective, Ethical approval or Sensitivity analysis, compared with other items. This might be explained by the fact that, some items were more related to a certain need of users than the others. For example, a tool addressing concerns on RoB may focus less on Study objective, which is relatively more difficult to be directly linked to a well-defined type of bias. Still, since these quality items are related to NRSI quality, and they are rarely sufficiently described, particular efforts investigating these quality items may be needed in future tool development. In contrast, while some quality items have been frequently addressed, such as Length of follow-up and Intervention measurement, they are not necessarily relevant to all types of user needs. For example, as shown in table 2 and online supplemental file appendix 7 , 14 tools highlighted that the follow-up should be sufficiently substantial for detecting an association between intervention and outcome, but none of these tools linked Length of follow-up to RoB. Therefore, we recommend tool developers to clarify not only the purpose of their tools but also the relevance of their signalling questions to any user needs (eg, RoB assessment). We also advise that in future research the relationships between quality items and user needs will be investigated in more detail.

Our study has a number of limitations. One limitation is that, some tools identified by our study were originally developed for purposes beyond assessing methodological quality of reporting of NRSIs, so our study could not cover all potentials of these tools. For example, the GRADE framework was mainly designed for addressing certainty of evidence, such as indirectness (ie, whether interventions were compared directly), and for making relevant clinical practice recommendations. While it mentions RoB (eg, publication bias), its main purpose is to illustrate how to grade quality of evidence, rather than to function as an exact quality appraisal tool. In other words, the GRADE allows users to use any additional tools to assess NRSI quality. 52 Also, the GRADE checklist was designed for both RCTs and NRSIs, so some criteria might be relatively brief, compared with specifically designed tools, such as RTI Item Bank. 27 Finally, GRADE can be used to estimate and score the quality of evidence for the full body of evidence and not only for individual primary studies. Therefore, tool users who assess NRSIs beyond methodological quality or reporting should consider criteria in addition to those mentioned in our study, for selecting a tool. Another limitation is that, some tools were predecessors of others, but we did not exclude them if they met the inclusion criteria. For example, the ROBINS-I tool was developed from the Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI), 53 and some of their signalling questions differed. Such information on tool linkage may also be considered for tool selection, if available from the tools. Another limitation is that we only searched HTA agencies for grey literature, and the returned hits on the snowballing approach depended on the starting-set articles, so some tools only mentioned by clinical guideline or regulatory organisations, or tools missed by the previous reviews might have been overlooked. Also, only one researcher (MH) traced versions of tools, by following reference lists of the identified studies and by visiting websites of the online tools. Consequently, the most up-to-date version of a tool might be missing, and the extent to which a quality item was described by a tool might be underestimated. As existing appraisal tools are improved continuously and new tools are being developed (eg, the HARmonized Protocol Template to Enhance Reproducibility (HARPER) and Authentic Transparent Relevant Accurate Track-Record (ATRAcTR)), 54 55 an online platform that automatically identifies appraisal tools and summarises tool information is promising. Such platforms have already been established for tools for assessing observational studies for exposures that were not controlled by investigators (eg, dietary patterns). 56 Another limitation is that we categorised criteria of a quality item as ‘sufficient’ or ‘brief’ for each tool, based on whether an explanation was provided for the criteria. Though consensus was reached among authors, and all tool criteria were independently reviewed by two researchers, tool users might question the feasibility of such categorisation when selecting a tool. Additionally, as we categorised quality items based on the order of conducting an NRSI (ie, from study design to results presentation), we did not provide specific suggestions on how to select tools based on bias categories. For example, motivational bias, which would occur when judgements are influenced by the desirability or undesirability of events or outcomes, may affect reporting and measurement of patient outcomes and adherence to healthcare interventions. 57 58 Although the items Conflict of interest and Outcome measurement are relevant to motivational bias, we did not investigate their relationships. Hence, we recommend for future research to bridge our quality items to all potential categories of bias, then test whether a tool selected based on such categorisation, together with recommendations from previous reviews, can really satisfy tool users. It is also worth noting that, the target audience of this review and content analysis could be decision-makers who assess the general quality of an NRSI, NRSI performers who may report quality of their studies, or developers of relevant appraisal tools. However, when users focus on a specific type of concern (eg, causal effect or data quality), some methodological guidance investigating the specific issue or tools beyond the healthcare field (eg, social science) really exist 59 60 and may be referred to by users. In addition, the tools for diagnosis studies, prognosis studies and secondary studies were beyond the scope of our study, and relevant users may refer to other studies, such as Quigley et al 14 , for further information. Moreover, some frameworks specifically designed for assessing data quality, for example, in terms of data structures and completeness, have been published, and some of their instructions may also be considered as criteria for assessing NRSI quality. 61–65 While evaluating these frameworks is beyond the scope of this study, we recommend tool developers to refer to these frameworks when they define relevant criteria or signalling questions in the future.

Most of the appraisal tools for NRSIs have their own strengths, but none of them could address all quality concerns relevant to these studies. Even the most comprehensive tools could be complemented with items from other tools. With information on how sufficiently a tool describes a quality item, tool users might broaden their horizons on quality concerns of non-randomised studies to be considered and might select a tool that more completely satisfies their needs. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity when selecting a tool or identifying a research gap.

Acknowledgments

This research was once published as an abstract in 2022-11, ISPOR Europe 2022, Vienna, Austria. The citation was: Jiu L, Hartog MK, Wang J, et al . OP18 applicability of appraisal tools of real-world evidence in health technology assessment: a literature review and content analysis. Value Health . 2022 Dec 1;25(12):S389.

  • de Boer A ,
  • Hillege H , et al
  • Reeves BC ,
  • Higgins JPT , et al
  • Higgins J ,
  • Rooney A , et al
  • Katkade VB ,
  • Sanders KN ,
  • Baumfeld Andre E ,
  • Carrington N ,
  • Siami FS , et al
  • van Veelen A ,
  • Jonsson P , et al
  • Salcher-Konrad M ,
  • Boccia S , et al
  • Rannanheimo P ,
  • Batchelor L , et al
  • Hogervorst MA ,
  • Vreman RA , et al
  • Schünemann HJ ,
  • Tugwell P ,
  • Reeves BC , et al
  • Sterne JA ,
  • Hernán MA ,
  • McAleenan A , et al
  • Orsini LS ,
  • Crown W , et al
  • D’Andrea E ,
  • Patorno E , et al
  • Quigley JM ,
  • Thompson JC ,
  • Halfpenny NJ , et al
  • University of Sheffield
  • Evidence Analysis
  • Dreyer NA ,
  • Velentgas P ,
  • Westrich K , et al
  • ↵ Open science framework (OSF) Registry . 2022 . Available : https://osf.io/dashboard
  • ↵ Connected papers . 2022 . Available : https://www.connectedpapers.com/about
  • ↵ The International network of agencies for health technology assessment members . 2022 . Available : https://www.inahta.org/members/members_list/
  • Ouzzani M ,
  • Hammady H ,
  • Fedorowicz Z , et al
  • Whiting P ,
  • Mallett S , et al
  • Nowell LS ,
  • Norris JM ,
  • White DE , et al
  • Viswanathan M ,
  • Forestier D , et al
  • Faillie J-L ,
  • Gouverneur A , et al
  • Berger ML ,
  • Martin BC ,
  • Husereau D , et al
  • Joanna Briggs Institute
  • Moloney L ,
  • Wolfram T , et al
  • Vandenbroucke JP ,
  • von Elm E ,
  • Altman DG , et al
  • Des Jarlais DC ,
  • Crepaz N , et al
  • Genaidy AM ,
  • Lemasters GK ,
  • Lockey J , et al
  • ENCePP Guide on Methodological Standards in Pharmacoepidemiology (Revision 10)
  • Public Policy Committee
  • Fesperman SF , et al
  • Schneeweiss S ,
  • Berger ML , et al
  • Benchimol EI ,
  • Guttmann A , et al
  • ↵ National institute for health and care excellence (NICE) ]. The guidelines manual: Appendices B–I . 2012 . Available : https://www.nice.org.uk/process/pmg6/resources/the-guidelines-manual-appendices-bi-pdf-3304416006853
  • Kennedy CE ,
  • Fonner VA ,
  • Armstrong KA , et al
  • ↵ Critical appraisal tools . 2022 . Available : https://www.cardiff.ac.uk/specialist-unit-for-review-evidence/resources/critical-appraisal-checklists
  • Yang Z-H , et al
  • Franklin JM ,
  • Martin D , et al
  • Patnode CD ,
  • Berkman ND , et al
  • Blagojevic M ,
  • Jeffery A , et al
  • Guyatt GH ,
  • Vist G , et al
  • University of Bristol [Internet
  • Pottegård A ,
  • Li JZ , et al
  • Allman-Farinelli M , et al
  • Arjmand EM ,
  • Shapiro JA ,
  • Shah RK , et al
  • Montibeller G ,
  • von Winterfeldt D
  • WILEY online library
  • Sharma Waddington H ,
  • Cairncross S
  • Food and Drug Administration (FDA)
  • European Medicines Agency (EMA)
  • Duke Margolis Center for Health Policy
  • Callahan TJ ,
  • Barnard J , et al
  • Schmidt CO ,
  • Struckmann S ,
  • Enzenbach C , et al
  • Bishop FL ,
  • Prescott P ,
  • Chan YK , et al
  • Gagnon M-P ,
  • Griffiths F , et al
  • Falco FJE , et al
  • Heller RF ,
  • Gemmell I , et al
  • Weightman AL, Mann MK, Sander L, Turley RL
  • Thomas BH ,
  • Ciliska D ,
  • Dobbins M , et al
  • Rangel SJ ,
  • Colby CE , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Data supplement 1

Contributors LJ designed the study protocol, identified appraisal tools, conducted the content analysis and wrote the manuscript; MH identified appraisal tools, collected data on appraisal tools and conducted the content analysis; JW designed the study protocol, solved the discrepancies on identification of appraisal tools and edited the manuscript; RAV designed the study protocol and edited the manuscript; OK provided assistance on coding of quality items and edited the manuscript; AM-T edited the manuscript; WG edited the manuscript, and was responsible for the overall content as the guarantor.

Funding This research was performed as part of the HTx project. The project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825162.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

ORIGINAL RESEARCH article

Pneumocystis jirovecii pneumonia associated with immune checkpoint inhibitors: a systematic literature review of published case reports and disproportionality analysis based on the faers database.

Shuang Xia,,

  • 1 Department of Pharmacy, The Second Xiangya Hospital, Central South University, Hunan, China
  • 2 International Research Center for Precision Medicine, Transformative Technology and Software Services, Hunan, China
  • 3 Toxicology Counseling Center of Hunan Province, Hunan, China
  • 4 Hunan University of Chinese Medicine, Hunan, China
  • 5 Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, United States
  • 6 Division of Oncology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, United States
  • 7 Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH, United States
  • 8 Laboratory of Clinical Pharmacy, Gifu Pharmaceutical University, Gifu, Japan

Background: Pneumocystis jirovecii pneumonia (PJP) has been reported with ICIs but limited to case reports. The clinical features of PJP with ICIs remain mostly unknown. This study aims to investigate the association of PJP with ICIs and describe clinical features.

Methods: Reports of PJP recorded in FAERS (January 2004–December 2022) were identified through the preferred term “Pneumocystis jirovecii pneumonia”. Demographic and clinical features were described, and disproportionality signals were assessed through the Reporting Odds Ratio (ROR) and Information Component (IC), using traditional chemotherapy and targeted therapy as comparators, and adjusting signals by excluding contaminant immunosuppressive drugs and pre-existing diseases. A systematic literature review was conducted to describe clinical features of published PJP reports with ICIs. Bradford Hill criteria was adopted for global assessment of the evidence.

Results: We identified 677 reports of PJP associated with ICIs, in which 300 (44.3%) PJP cases with fatal outcome. Nivolumab (IC 025 2.05), pembrolizumab (IC 025 1.88), ipilimumab (IC 025 1.43), atezolizumab (IC 025 0.36), durvalumab (IC 025 1.65), nivolumab plus ipilimumab (IC 025 1.59) have significant signals compared to other drugs in FAERS database. After excluding pre-existing diseases and immunosuppressive agents which may increase susceptibility of PJP, the signals for PJP associated with nivolumab, pembrolizumab, durvalumab, nivolumab plus ipilimumab remained robust (IC 025 > 0). When compared to other anticancer regimens, although all ICIs showed a lower disproportionate signal for PJP than chemotherapy, nivolumab (IC025 0.33, p < 0.001), pembrolizumab (IC025 0.16, p < 0.001), both PD-1 inhibitors, presented a higher signal for PJP than targeted therapy. Male gender (IC 025 0.26, p < 0.001) and age >65 years (IC 025 0.38, p < 0.001) were predominant in PJP cases associated with across all ICIs. In literature, 15 PJP cases associated with ICIs were reported in 10 published case reports. 12 of 15 (80.0%) of cases received PD-1 inhibitors before PJP was diagnosed.

Conclusion: By the combined analysis of post-marketing data from FAERS and published case reports, we identified ICIs may be associated with PJP, especially in males aged >65years. After accounting for confounders, PD-1 inhibitors emerged with a robust disproportionality signal when compared to PD-L1/CTLA-4 inhibitors as well as targeted therapy. Further research is warranted to validate our findings.

1 Introduction

Immune checkpoint inhibitors (ICIs) have dramatically changed the treatment landscape of many types of malignancies. As of November 2022, FDA has approved 10 ICI agents including PD-1 inhibitors: nivolumab, pembrolizumab, cemiplimab, dostarlimab, PD-L1 inhibitors: atezolizumab, avelumab, durvalumab, CTLA-4 inhibitors: ipilimumab and tremelimumab, LAG-3 inhibitors: relatlimab. These ICIs generate durable efficacy in certain types of cancer, however, there is a significant concern ICIs-mediated toxicities. Toxic effects from these ICI agents are related to removing nodes of self-tolerance and unleashing autoimmune-like phenomena ( Postow et al., 2018 ). Although usually manageable with corticosteroid and immunosuppressants administration, clinically severe events leading to morbidity and even mortality may complicate ICI treatment ( Wang et al., 2018 ).

Pneumocystis jirovecii pneumonia is a form of pneumonia that is caused by the yeast-like fungus Pneumocystis jirovecii. A recent study ( Lee et al., 2019 ) showed that the 3-month mortality rate of lung cancer patients with the PJP was as high as 61.6%, suggesting the severity of PJP infection in lung cancer patients. Moreover, the Infectious Diseases Working Party (AGIHO) of the German Society for Hematology and Medical Oncology (DGHO) recommended that primary prophylaxis of Pneumocystis jirovecii pneumonia in patients with hematologic malignancies and solid tumors are warranted ( Classen et al., 2021 ). To the best of our knowledge, only a few case reports/series have described Pneumocystis jirovecii pneumonia (PJP) during pembrolizumab ( Liu et al., 2020 ; Sadek et al., 2020 ; Rath et al., 2021 ; Si et al., 2021 ; Hiba et al., 2022 ; Sanka and Hsu, 2023 ), nivolumab ( Schwarz et al., 2019 ), ipilimumab ( Arriola et al., 2015 ; Finbar Slevin et al., 2016 ; Sadek et al., 2020 ), nivolumab plus ipilimumab ( Moujaess et al., 2020 ) and other anti-PD-1 inhibitors ( Liu et al., 2020 ; Feng et al., 2021 ).

However, case reports can only provide a partial epidemiological perspective. There are no meta-analysis or systematic reviews to investigate the association of PJP and ICIs. In addition, the clinical manifestations of PJP in patients on ICIs are not well known. The analysis of spontaneous reporting databases, such as the FDA Adverse Even Reporting System (FAERS) and WHO Vigibase, allows a broader perspective by collecting unpublished reports of adverse events submitted all over the world occurring in real-world unselected subjects with comorbidities, poly-pharmacotherapy and complex anticancer combination regimens ( Raschi et al., 2020 ), which ensures rapid detection of even rare adverse events. Previous disproportionality analysis based on the FAERS database indicated that interstitial lung disease ( Guo et al., 2023 ), pulmonary tuberculosis ( Zhu et al., 2022a ) and pneumonitis ( Cui et al., 2022 ) were found to be significantly associated with ICIs exposure. But it is still unclear if ICIs were associated with increased reporting frequency of PJP in the real-world clinical setting.

Herein, we conducted a large-scale pharmacovigilance study by using the US Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) database to investigate the link between PJP and ICIs. A systematic literature review was conducted to incorporate the information from published papers and compare with post-marketing data from FAERS database.

2 Materials and methods

2.1 data sources and study design.

FAERS is one of the largest publicly available databases designed to support the United States Food and Drug Administration (FDA) post-marketing safety surveillance program, which gathering more than 20 million reports worldwide, including the United States, Europe, and Asia. It allows for the signal detection and quantification of the association between drugs and reporting of AEs. AERS Mine is a validated multi-cohort analyzing application designed to mine data across millions of patient reports (currently 19,089,556) from the FDA’s Adverse Event Reporting System ( Sarangdhar et al., 2016 ).A recent high-impact study which combined clinical cardiotoxicity of kinase inhibitors with cell line-derived transcriptomic datasets to identify a gene signature that can predict risk of cardiotoxicity by leveraging AERS Mine to visit FAERS data ( van Hasselt et al., 2020 ). Firstly, we performed a retrospective disproportionality analysis of PJP cases with ICIs using data from the AERSMine. Secondly, we conducted a systematic literature review to confirm whether there is an association between PJP and ICIs therapies and provide a comprehensive clinical description of PJP induced by ICIs. The flow chart of this study was displayed in Figure 1 . Ethical approval was not required because this study was conducted by using deidentified data.

www.frontiersin.org

FIGURE 1 . Flow chart to combine the FAERS disproportionality analysis with the systematic literature review. FAERS, FDA Adverse Event Reporting System; ICI, immune checkpoint inhibitor; Q1, quarter 1; PJP, Pneumocystis jirovecii pneumonia.

2.2 Pharmacovigilance analysis

A pharmacovigilance study was conducted from 2011 Quarter 1 (Q1, ipilimumab was approved by FDA on 25 March 2011) to 2022 Q4 with the FAERS data in AERS Mine to evaluate the disproportionate signal of PJP correlated with ICIs in a large-scale population. We included FDA approved 10 ICI agents (nivolumab, pembrolizumab, ipilimumab, atezolizumab, avelumab, durvalumab, dostarlimab, cemiplimab, tremelimumab, relatlimab) and one ICI combination therapies (nivolumab plus ipilimumab). PJP cases were identified by searching the Medical Dictionary for Regulatory Activities (MedDRA) (version 25.0), for preferred term “Pneumocystis jirovecii pneumonia”. Only case number more than 5 were included in this study. We collected the following information for each detected PJP reports: report year, demographic information (gender, age), drugs, indications, and co-administrated drugs. The demographics features including age and gender distribution of PJP cases associated with ICIs were investigated by comparing with all other adverse events of ICIs.

We performed the case/non-case analysis, a validated concept in pharmacovigilance, to investigate whether suspected PJP (cases) are differentially reported with immune check point inhibitors as compared to other adverse events (non-cases) ( Faillie, 2019 ). We used two different disproportionate indicators, reporting odds ratios (RORs) and information components (IC) to reduce likelihood of false positive signals ( Noguchi et al., 2021 ). When the lower limit of the 95% confidence interval of ROR (ROR 025 ) >1 ( Rothman et al., 2004 ) or the lower limit of the 95% confidence interval of IC (IC 025 ) >0 ( Bate et al., 1998 ), significant adverse events were detected. If the proportion of adverse events of interest is greater in patients exposed to a specific drug (cases) than in patients not exposed to this drug (non-cases), a disproportionality signal emerges, and subsequent analytical investigations are usually required before taking regulatory actions ( Raschi et al., 2020 ). Both approaches were firstly conducted by using all other drugs in the FAERS database as a comparator, a common exploratory disproportionality analysis method.

Considering confounders such as comorbidities may affect the safety signal of PJP, we conducted primary sensitivity analyses to confirm the PJP safety signals correlated with ICIs. We excluded diseases (“renal transplant” “liver transplant” “stem cell transplant” “hiv infection” “bone marrow transplant” “inflammatory bowel disease” “organ transplant” “chronic obstructive pulmonary disease”) as PJP may occur preferentially in those immunocompromised patients with aforementioned conditions. We also excluded some drugs that cause an immunosuppression state which may further induce infection (“corticosteroids” “glucocorticoids” “immunosuppressants”). To assess the robustness of disproportionality signals and account for underlying confounders of the drug-event association, we conducted secondary sensitivity analysis by using anticancer drugs, such as traditional chemotherapy and molecular targeted therapy, as a comparator (to reduce confounding by indication and provide a clinical perspective). Firstly, we identified relevant National Comprehensive Cancer Network (NCCN) guidelines ( Supplementary Figure S1 ), according to FDA-approved indications of ICIs. Then we extracted different cancer regimens from those selected NCCN guidelines and compared the safety signal of PJP between ICIs and traditional chemotherapy/molecular targeted therapy. Differences in categorical variables were assessed using a chi-squared test of independence performed on a 2 × 2 contingency table with Yates’ continuity correction or fisher exact test. Significance was assumed when the p -value less than 0.05. Data analyses were conducted by using the IBM SPSS (26.0) and Microsoft Excel (2021).

2.3 Systematic literature review

A comprehensive literature review was conducted through PubMed, Google Scholar, Web of Science, Embase from inception to 20 February 2023 (We have registered our protocol in PROSPERO with a number CRD42022376162 before the formal search). The search strategy included the keywords (“immunotherapy OR immune checkpoint inhibitors OR PD-1 inhibitors OR PD-L1 inhibitors OR CTLA-4 inhibitors OR nivolumab OR pembrolizumab OR cemiplimab OR dostarlimab OR atezolizumab OR avelumab OR durvalumab OR ipilimumab OR tremelimumab OR relatlimab”) AND (“Pneumocystis jirovecii pneumonia” OR “Pneumocystis pneumonia” OR “Pneumocystosis”). Mesh terms (“Immune Checkpoint Inhibitors” [Mesh]) AND “Pneumonia, Pneumocystis” [Mesh] were also used in the search process of PubMed. We only included literature written in English. Meeting abstracts were excluded. Case reports case series, case-control studies, observational studies, single-arm studies with detailed clinical information of PJP were retained in our final analyses. Two reviewers independently searched the literature and examined the relevant studies for further assessment of data, and collected clinical characteristics including age, gender, indication, absolute lymphocyte count, comorbidities, first/second-line regimens, immune-related adverse events, immunosuppressive agents, and outcome of PJP cases associated with ICIs. The quality of the reports was assessed following the recommended guidelines for publishing an adverse event report ( Kelly et al., 2007 ).

2.4 Global assessment of the evidence

Although disproportionality analysis per se is not an estimate, it could be evaluated and used for signal prioritization, which further provides clues for regimens management clinically and regulatory actions. Previous research ( Pacurariu et al., 2017 ) concluded criteria used for signal prioritization and of the associated decision support frameworks, including strength of evidence, public health impact, novelty and general public and media attention. This study adopted novelty of the drug event association, seriousness, and disproportionate reporting as criteria for the signal prioritization. Moreover, a causal relationship appraisal was carried out on the entire body of evidence by using adapted Bradford Hill criteria used in epidemiology that have been applied to pharmacovigilance data ( van Hunsel et al., 2018 ; Raschi et al., 2022 ), including biological plausibility, strength, consistency, specificity, coherence, and analogy. The rechallenge/de-challenge item were not analyzed in this study due to missing data.

3.1 PJP is associated with ICIs in the FAERS database

From Q1, 2011 to Q4, 2022, we detected 677 patients on ICIs with PJP. In the primary analysis, we found that nivolumab, pembrolizumab, ipilimumab, atezolizumab, durvalumab, nivolumab plus ipilimumab had a significant safety signal (ROR 025 > 1, IC 025 > 0) for PJP ( Table 1 ). After the sensitivity analyses, the above six ICI regimens consistently emerged with strong disproportionality, through both, ROR and IC approaches. ( Table 2 ). When compared with all other anti-cancer drugs, nivolumab (IC 025 0.09, p = 0.001) showed significant higher safety signal of PJP. Then we found that across all ICIs (IC 025 0.18, p < 0.001), nivolumab (IC 025 0.33, p < 0.001), pembrolizumab (IC 025 0.16, p < 0.001), durvalumab (IC 025 –0.01, p = 0.019) presented a significant signal for PJP compared to targeted therapy. However, all ICIs showed a low signal of disproportionate reporting for PJP when compared to traditional chemotherapy ( Figure 2 ).

www.frontiersin.org

TABLE 1 . Pneumocystis jirovecii pneumonia signals of different immune checkpoint inhibitors (Primary analysis).

www.frontiersin.org

TABLE 2 . Pneumocystis jirovecii pneumonia signals of different immune checkpoint inhibitors after sensitivity analysis.

www.frontiersin.org

FIGURE 2 . The comparison of Pneumocystis jirovecii pneumonia signal between ICIs and targeted therapy/chemotherapy in FAERS database. The list of control group (targeted therapy and chemotherapy) was extracted from NCCN guidelines for FDA approved indication of ICIs. Abbreviation: PJP, Pneumocystis jirovecii pneumonia. ROR, reporting odds ratio. IC, information component. 95%CI, 95% confidence interval. N, number. AEs, adverse events. ICIs, immune checkpoint inhibitors. NCCN, National Comprehensive Cancer Network.

3.2 Clinical features of PJP cases related to ICIs in FAERS database

We analyzed the clinical characteristics of PJP cases correlated with ICIs in the FAERS database. A total of 85.3% of PJP reports associated with ICIs were reported from 2018 to 2022.68.5% of cases were reported by health professionals. 300 of 677 (44.3%) of PJP cases died and 123 of 677 (18.2%) PJP cases experienced life-threatening situation. 66.3% (403 of 677) were elderly, age >65 years. A total of 77.0% (502/677) of PJP cases related to ICIs were males. To assess whether age >65 years is a factor may increase reporting of PJP in patients on ICIs, we calculated the ROR and IC by comparing age >65 years PJP cases/all PJP cases with age >65 years cases/all adverse events cases on ICIs. We found that elderly whose age more than 65 years old was predominant in PJP reports of overall ICIs (IC 025 0.38, p < 0.001), nivolumab (IC 025 0.31, p < 0.001), pembrolizumab (IC 025 0.01, p = 0.001), ipilimumab (IC 025 0.32, p < 0.001), durvalumab (IC 025 –0.13, p = 0.008), nivolumab plus ipilimumab (IC 025 0.26, p < 0.001). Similarly, we also assessed the gender distribution in PJP associated with ICIs. Males is the predominance in PJP case reports of across all ICIs (IC 025 0.26, p < 0.001), nivolumab (IC 025 0.20, p < 0.001), pembrolizumab (IC 025 0.23, p < 0.001), ipilimumab (IC 025 –0.13, p = 0.015), nivolumab plus ipilimumab (IC 025 –0.18, p = 0.029). ( Figure 3 ). The top indications in PJP cases were lung cancer (171/667, 40.1%) and melanoma (160/677, 37.6%). We further analyzed the co-administrated regimens of ICIs in PJP cases. A total of 41.4% (280/677) cases reported to receiving glucocorticoids or corticosteroids during ICIs therapy. 69 out of 677 (10.2%) cases received immunosuppressants when they were on ICIs. Additional details of clinical features of PJP cases associated with specific ICIs regimens are shown in Table 3 .

www.frontiersin.org

FIGURE 3 . Age and gender distribution in Pneumocystis jirovecii pneumonia signal cases of ICIs. Abbreviation: PJP, Pneumocystis jirovecii pneumonia. ROR, reporting odds ratio. IC, information component. 95%CI, 95% confidence interval. N, number. AEs, adverse events. ICIs, immune checkpoint inhibitors.

www.frontiersin.org

TABLE 3 . Patient characteristics of PJP reports with immune checkpoint inhibitors in FAERS database.

3.3 Published PJP case reports related to ICIs

Our literature review identified 15 PJP cases related to ICIs. 8 of 15 (53.3%) cases were lung cancer and 4 of 15 (26.7%) cases were melanoma. 8 cases (53.3%) were male while 7 cases (46.7%) were female. 8 PJP cases (53.3%) were patients with an age >60 years old. 7 of 15 (46.7%) PJP cases were co-reported with other immune-related adverse events such as colitis and hepatitis. 12 of 15 (80.0%) of cases received PD-1 inhibitors before PJP was diagnosed. 11 of 15 (73.3%) of PJP cases received glucocorticoids, corticosteroids or other immunosuppressants before PJP occurred. 5 of 15 PJP patients (33.3%) died. Additional details about ICIs regimens (and previous treatments) are listed in Table 4 . The quality appraisal of the cases identified from the literature are summarized in Supplementary Figure S2 .

www.frontiersin.org

TABLE 4 . Summaries of published case reports of ICI-related Pneumocystis jirovecii pneumonia.

3.4 Global assessment of evidence

By the combined analysis from post-marketing adverse reports submitted to FAERS and published case reports, this study detected novel signals of PJP with ICIs and provided comprehensive evidence for the signal prioritization. Bradford Hill criteria were fulfilled, as indicated by the strength of disproportionality and its consistency throughout the analyses, thus supporting a likely causal association between PJP and ICIs (The signal priority and causal relationship appraisal could be found in Supplementary Table S2 ).

4 Discussion

To the best of our knowledge, this is the first large-scale pharmacovigilance study to investigate the association of PJP and ICIs by combining FAERS data mining and literature review. There are three main findings of this study:

Firstly, our post-marketing pharmacovigilance analysis showed that PJP was significantly associated with ICIs. Previous case reports ( Arriola et al., 2015 ; Finbar Slevin et al., 2016 ; Schwarz et al., 2019 ; Liu et al., 2020 ; Moujaess et al., 2020 ; Sadek et al., 2020 ; Feng et al., 2021 ; Rath et al., 2021 ; Si et al., 2021 ; Hiba et al., 2022 ) showed that PJP may be a complication in immunotherapy. But the sample size is small. We retrospectively analyzed 677 case reports in FAERS database, conducted primary analysis and sensitivity analysis and found that all ICIs had significant higher disproportionate signals for PJP than other drugs in the FAERS database. We subsequently compared the signals of PJP between ICIs and other anticancer regimens, including targeted therapy, and chemotherapy, both were extracted from NCCN’s guideline for ICIs’ indications in order to enhance the clinical perspective of this signal comparison. Although all ICIs were detected a lower signal of PJP than chemotherapy, we found that nivolumab and pembrolizumab, both PD-1 inhibitors, showed significant higher reporting frequency for PJP than targeted therapy. And our case collections also showed that use of PD-1 inhibitors was correlated with PJP cases (12 of 15, 80.0%). However, CTLA-4 inhibitors or PD-L1 inhibitors were not associated with increased PJP reports compared with other anticancer drugs from FAERS data analysis. A previous meta-analysis ( Su et al., 2019 ) showed that PD-L1 but not PD-1/CTLA4 inhibitors increased the risk of pneumonia compared to chemotherapy/placebo. However, another recent study ( Tong et al., 2021 ) showed that both PD-1 and PD-L1 inhibitors significantly increase the risk of all-grade and high-grade pneumonia in NSCLC patients compared to conventional chemotherapy. Considering the conflicting research evidence, our pharmacovigilance analysis and literature collections support that PD-1 inhibitors may carry a clear potential for PJP. Further research is warranted to explore this clinical association of ICIs and PJP.

Secondly, we systematically investigated the clinical features of PJP cases associated with ICIs. Across all ICIs (which included regimens in table 1 ), our data analysis from FAERS database showed that males had a higher PJP report frequency than female (IC 025 0.26, p < 0.001), elderly patients >65 years had more reports than younger patients (<65 years) on ICIs (IC 025 0.38, p < 0.001). This is consistent with our literature review on published case reports (53.3% PJP cases associated with ICIs were male and age more than 65).

Thirdly, our study provides more evidence to support that PJP may be unmasked in cancer patients in the process of immune reconstitution induced by immune checkpoint inhibitors. An earlier review ( Morelli et al., 2022 ) showed that PJP, could be categorized into a kind of opportunistic infections, and maybe associated with irAE treatment (corticosteroid, infliximab, etc.). However, there were also case reports ( Inthasot et al., 2020 ; Feng et al., 2021 ) of severe pulmonary infections induced by Mycobacterium tuberculosis , Aspergillus fumigatus and Pneumocystis jirovecii were outside the context of immunosuppressive therapy. Our FAERS data anallysis showed that only 41.4% and 10.2% of cases received glucocorticoids/corticosteroids, or immunosuppressants during ICIs therapy, respectively. Our literature review also showed that 26.7% (4 of 15) of cases did not receive glucocorticoids/corticosteroids, infliximab, mycophenolate mofetil or other immunosuppressive agents before the occurrence of PJP. Both the FAERS data and published case reports showed that PJP cases associated with ICIs may be independent of immunosuppression. The subclinical colonization of Pneumocystis jirovecii may unmask and progress into PJP in non-HIV-infected immunosuppressed populations ( Morris and Norris, 2012 ). Previous research ( Wu et al., 2004 ) showed that Pneumocystis jiroveci pneumonia will be unmasked during reversal of immunosuppression in non-HIV infected patients. Emerging research ( Feng et al., 2021 ; Lin et al., 2021 ) has reported that immune-related pneumonitis could be induced by immune reconstitution inflammatory syndrome. Previous preclinical study ( Kauffman et al., 2021 ) showed that immune checkpoint blockade may cause an exaggerated immune response to fungal colonisation, which could promote fungal growth similar to recent studies in Mycobacterium tuberculosis infection. With the data from FAERS and published case reports, as well as previous research on mechanism of immune-related pneumonitis, we believe immune checkpoint inhibitors especially PD-1 inhibitors, may “unmask” low level PJP colonisation in the same way that antiretroviral therapy can reveal subclinical tuberculosis as immune reconstitution occurs. This is inherently counter-intuitive as developing PJP (a disease of immunocompromise) in association with ICI (which boosts immunity) but indeed be a possible mechanism for PJP.

Another study ( Mansharamani et al., 2000 ) indicated that in immunosuppressed persons without HIV infection, CD4 + counts may be a useful clinical marker to identify specific individuals at particularly high clinical risk for PJP. Previous prospective study ( Agrawal et al., 2016 ) showed that Absolute Lymphocyte Count <1643 μl but not 1200 μl could be the cost-effective surrogate marker for CD4 cell counts <200 cells/μL in monitoring HIV infected individuals. Moreover, a recent meeting abstract ( Joshi et al., 2019 ) in 2019 ASCO Annual Meeting indicated that low ALC (750/μL, median) and prolonged steroid therapy are more likely to result in PJP infection as opposed to steroid therapy alone. Our case series identified 9 PJP cases with ALC amount, 7 of 9 (77.8%) had an ALC <1200 μL. The above literature provided literature evidence of monitoring ALC value as an indicator of PJP infection when CD4 T Cells amount is not accessible.

To increase the robustness of disproportionate signals detected from spontaneous reporting systems, some researchers tried to combine pharmacovigilance analysis with literature review. Wang et al. investigated fatal toxic effects associated with immune checkpoint inhibitors by using data from large academic medical centers, global WHO pharmacovigilance data, and all published ICI clinical trials ( Wang et al., 2018 ). Stevens-Johnson syndrome/toxic epidermal necrolysis ( Zhu et al., 2021 ) and type 1 diabetes ( Zhu et al., 2022b ) were found highly associated with immune checkpoint inhibitors by analyzing data from clinical trials and post-marketing data from the FAERS database. Raschi et al. (2019) confirmed that ICIs are associated with a multitude of irAEs, especially respiratory, endocrine, and hepatic toxicities by conducting parallel approach through contemporary post-marketing analysis and overview of systematic reviews. With the above research experience, this study presented that ICIs were strongly associated with PJP by the combined analysis of pharmacovigilance and systematic literature review.

Our study has several limitations. Firstly, FAERS database is a spontaneous reporting system whose data source is heterogeneous (both non-health care and healthcare practitioners) and therefore reporting bias exists. Secondly, FAERS data could not be used to calculate the incidence of PJP because of the under-reporting phenomenon and not having the full number of patients who have received the drug. Thirdly, detailed clinical information such as previous treatment regimens and stages of cancer are missing, so the reports do not confirm causality of the drug-induced event. However, this large-scale pharmacovigilance analysis did provide comprehensive information about the link between PJP and ICIs.

5 Conclusion

Our literature and FAERS analysis indicated that ICIs may be associated with a safety signal of PJP, especially in males aged >65years. PD-1 inhibitors emerged with a robust disproportionality signal when compared to PD-L1/CTLA-4 inhibitors as well as targeted therapy, even accounting for confounders such as concomitant immunosuppressive drugs. Anti-PD-1 therapy may unmask low level Pneumocystis jirovecii colonization and cause PJP infection in patients. More pre-clinical or clinical studies are warranted to confirm the association of PJP and ICIs and explore the potential new mechanism of ICIs-related PJP.

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary Material , further inquiries can be directed to the corresponding author.

Author contributions

SX: formal analysis, data curation, writing—original draft, review and editing, and visualization. MS and YN: software, methodology, and writing—review and editing. LL, LG and YCZ: resources. HG and Y-KW: software. B-KZ: supervision. MY: conceptualization, methodology and supervision. All authors participated in the interpretation of the results. The final manuscript was read, checked, and approved by all authors.

This research was funded by Health Commission of Hunan Province, grant number 202113012480.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2023.1129730/full#supplementary-material

Agrawal, P. B., Rane, S. R., and Jadhav, M. V. (2016). Absolute lymphocyte count as a surrogate marker of CD4 count in monitoring HIV infected individuals: A prospective study. J. Clin. Diagn Res. 10 (5), EC17–9. doi:10.7860/JCDR/2016/19263.7765

CrossRef Full Text | Google Scholar

Arriola, E., Wheater, M., Krishnan, R., Smart, J., Foria, V., and Ottensmeier, C. (2015). Immunosuppression for ipilimumab-related toxicity can cause pneumocystis pneumonia but spare antitumor immune control. Oncoimmunology 4 (10), e1040218. doi:10.1080/2162402X.2015.1040218

PubMed Abstract | CrossRef Full Text | Google Scholar

Bate, A., Lindquist, M., Edwards, I. R., Olsson, S., Orre, R., Lansner, A., et al. (1998). A Bayesian neural network method for adverse drug reaction signal generation. Eur. J. Clin. Pharmacol. 54 (4), 315–321. doi:10.1007/s002280050466

Classen, A. Y., Henze, L., von Lilienfeld-Toal, M., Maschmeyer, G., Sandherr, M., Graeff, L. D., et al. (2021). Primary prophylaxis of bacterial infections and pneumocystis jirovecii pneumonia in patients with hematologic malignancies and solid tumors: 2020 updated guidelines of the infectious diseases working party of the German society of Hematology and medical Oncology (AGIHO/DGHO). Ann. Hematol. 100 (6), 1603–1620. doi:10.1007/s00277-021-04452-9

Cui, C., Deng, L., Wang, W., Ren, X., Wang, Y., and Cui, W. (2022). Respiratory system toxicity induced by immune checkpoint inhibitors: A real-world study based on the FDA adverse event reporting system database. Front. Oncol. 12, 941079. doi:10.3389/fonc.2022.941079

Faillie, J. L. (2019). Case-non-case studies: Principle, methods, bias and interpretation. Therapie 74 (2), 225–232. doi:10.1016/j.therap.2019.01.006

Feng, Y., Chen, C., Zhao, L., Zhu, X., Zhu, X., and Li, Q. (2021). A potential mechanism of the onset of immune-related pneumonitis triggered by anti-PD-1 treatment in a patient with advanced adenocarcinoma lung cancer: Case report. BMC Pulm. Med. 21 (1), 291. doi:10.1186/s12890-021-01649-6

Finbar Slevin, C. M., Marples, M., and Marples, M. (2016). Pneumocystis jirovecii pneumonia in a patient with melanoma treated with infliximab and corticosteroids for ipilimumab-associated colitis. Glob. Dermatol. 3, 1000193. doi:10.15761/god.1000193

Guo, X. J., Cai, X. T., Rong, Z. X., Zhang, Y. P., Wen, Y. X., Bai, X., et al. (2023). Interstitial pneumonitis associated with combined regimen of immunotherapy and conventional therapies-pharmacovigilance database analysis with real-world data validation. BMC Med. 21 (1), 6. doi:10.1186/s12916-022-02713-6

Hiba, Z., Abdelmoughit, H., Zaynab, I. H., Hounaida, J., Rachida, L., and Youssef, O. (2022). Pneumocystis pneumonia in patient with lung adenocarcinoma: Early side effects from pembrolizumab. Radiol. Case Rep. 17 (10), 3979–3981. doi:10.1016/j.radcr.2022.07.083

Inthasot, V., Bruyneel, M., Muylle, I., and Ninane, V. (2020). Severe pulmonary infections complicating nivolumab treatment for lung cancer: A report of two cases. Acta Clin. Belg 75 (4), 308–310. doi:10.1080/17843286.2019.1629078

Joshi, K. A., Atwal, D., Siegel, E. R., Jillella, A., Konda, M., and Parikh, R. (2019). Absolute lymphocyte count (ALC) as predictor of Pneumocystis Jiroveci Pneumonia (PCP) infection in patients on immune checkpoint inhibitors (ICPi). J. Clin. Oncol. 37, e14255. doi:10.1200/JCO.2019.37.15_suppl.e14255

Kauffman, K. D., Sakai, S., Lora, N. E., Namasivayam, S., Baker, P. J., Kamenyeva, O., et al. (2021). PD-1 blockade exacerbates Mycobacterium tuberculosis infection in rhesus macaques. Sci. Immunol. 6 (55), eabf3861. doi:10.1126/sciimmunol.abf3861

Kelly, W. N., Arellano, F. M., Barnes, J., Bergman, U., Edwards, R. I., Fernandez, A. M., et al. (2007). Guidelines for submitting adverse event reports for publication. Drug Saf. 30 (5), 367–373. doi:10.2165/00002018-200730050-00001

Lee, E. H., Kim, E. Y., Lee, S. H., Roh, Y. H., Leem, A. Y., Song, J. H., et al. (2019). Risk factors and clinical characteristics of Pneumocystis jirovecii pneumonia in lung cancer. Sci. Rep. 9 (1), 2094. doi:10.1038/s41598-019-38618-3

Lin, X., Lu, T., Li, S., Xie, X., Chen, X., Jiang, J., et al. (2021). Cytomegalovirus infection as an underestimated trigger for checkpoint inhibitor-related pneumonitis in lung cancer: A retrospective study. Clin. Transl. Oncol. 23 (2), 389–396. doi:10.1007/s12094-020-02432-5

Liu, Z., Liu, T., Zhang, X., Si, X., Wang, H., Zhang, J., et al. (2020). Opportunistic infections complicating immunotherapy for non-small cell lung cancer. Thorac. Cancer 11 (6), 1689–1694. doi:10.1111/1759-7714.13422

Mansharamani, N. G., Balachandran, D., Vernovsky, I., GaRland, R., and Koziel, H. (2000). Peripheral blood CD4 + T-lymphocyte counts during Pneumocystis carinii pneumonia in immunocompromised patients without HIV infection. Chest 118 (3), 712–720. doi:10.1378/chest.118.3.712

Morelli, T., Fujita, K., Redelman-Sidi, G., and Elkington, P. T. (2022). Infections due to dysregulated immunity: An emerging complication of cancer immunotherapy. Thorax 77 (3), 304–311. doi:10.1136/thoraxjnl-2021-217260

Morris, A., and Norris, K. A. (2012). Colonization by Pneumocystis jirovecii and its role in disease. Clin. Microbiol. Rev. 25 (2), 297–317. doi:10.1128/CMR.00013-12

Moujaess, E., El Haddad, E., and Joseph, K. (2020). Pneumocystis jiroveci mimicking COVID-19 pneumonia in a patient who is receiving ipilimumab and nivolumab combination therapy: A case report. Eurasian J. Med. Oncol. 2020, 89845. doi:10.14744/ejmo.2020.89845

Noguchi, Y., Tachi, T., and Teramachi, H. (2021). Detection algorithms and attentive points of safety signal using spontaneous reporting systems as a clinical data source. Brief. Bioinform 22 (6), bbab347. doi:10.1093/bib/bbab347

Pacurariu, A. C., Coloma, P. M., Gross-Martirosyan, L., Sturkenboom, M. C., and Straus, S. M. (2017). Decision making in drug safety-a literature review of criteria used to prioritize newly detected safety issues. Pharmacoepidemiol Drug Saf. 26 (3), 327–334. doi:10.1002/pds.4128

Postow, M. A., Sidlow, R., and Hellmann, M. D. (2018). Immune-related adverse events associated with immune checkpoint blockade. N. Engl. J. Med. 378 (2), 158–168. doi:10.1056/NEJMra1703481

Raschi, E., Fusaroli, M., Giunchi, V., Repaci, A., Pelusi, C., Mollica, V., et al. (2022). Adrenal insufficiency with anticancer tyrosine kinase inhibitors targeting vascular endothelial growth factor receptor: Analysis of the FDA adverse event reporting system. Cancers (Basel). 14 (19), 4610. doi:10.3390/cancers14194610

Raschi, E., Gatti, M., Gelsomino, F., Ardizzoni, A., Poluzzi, E., and De Ponti, F. (2020). Lessons to be learnt from real-world studies on immune-related adverse events with checkpoint inhibitors: A clinical perspective from pharmacovigilance. Target Oncol. 15 (4), 449–466. doi:10.1007/s11523-020-00738-6

Raschi, E., Mazzarella, A., Antonazzo, I. C., Bendinelli, N., Forcesi, E., Tuccori, M., et al. (2019). Toxicities with immune checkpoint inhibitors: Emerging priorities from disproportionality analysis of the FDA adverse event reporting system. Target Oncol. 14 (2), 205–221. doi:10.1007/s11523-019-00632-w

Rath, P., Bashir, H. I., Diwakar, A., and Murray, T. (2021). Pneumocystis jiroveci pneumonitis: A rare pneumonia in a patient treated with chemoimmunotherapy. Am. J. Respir. Crit. Care Med. 203, 1277676.

Rothman, K. J., Lanes, S., and Sacks, S. T. (2004). The reporting odds ratio and its advantages over the proportional reporting ratio. Pharmacoepidemiol Drug Saf. 13 (8), 519–523. doi:10.1002/pds.1001

Sadek, M., Loizidou, A., Drowart, A., Van den Wijngaert, S., Gomez-Galdon, M., and Aspeslagh, S. (2020). Pneumocystis infection in two patients treated with both immune checkpoint inhibitor and corticoids. J. Immunother. Precis. Oncol. 3 (1), 27–30. doi:10.4103/JIPO.JIPO_23_19

Sanka, P., and Hsu, A. (2023). A case of pneumocystis jirovecci in a patient with non-small cell lung cancer treated with immunotherapy. R I Med. J. (2013) 106 (1), 11–13.

PubMed Abstract | Google Scholar

Sarangdhar, M., Tabar, S., Schmidt, C., Kushwaha, A., Shah, K., Dahlquist, J. E., et al. (2016). Data mining differential clinical outcomes associated with drug regimens using adverse event reporting data. Nat. Biotechnol. 34 (7), 697–700. doi:10.1038/nbt.3623

Schwarz, M., Kocher, F., Niedersuess-Beke, D., Rudzki, J., Hochmair, M., Widmann, G., et al. (2019). Immunosuppression for immune checkpoint-related toxicity can cause pneumocystis jirovecii pneumonia (PJP) in non-small-cell lung cancer (NSCLC): A report of 2 cases. Clin. Lung Cancer 20 (3), e247–e250. doi:10.1016/j.cllc.2018.12.006

Si, S., Erickson, K., Evageliou, N., Silverman, M., and Kersun, L. (2021). An usual presentation of pneumocystis jirovecii pneumonia in a woman treated with immune checkpoint inhibitor. J. Pediatr. Hematol. Oncol. 43 (2), e163–e164. doi:10.1097/MPH.0000000000001757

Su, Q., Zhu, E. C., Wu, J. B., Li, T., Hou, Y. L., Wang, D. Y., et al. (2019). Risk of pneumonitis and pneumonia associated with immune checkpoint inhibitors for solid tumors: A systematic review and meta-analysis. Front. Immunol. 10, 108. doi:10.3389/fimmu.2019.00108

Tong, Z. Q., Wu, D. Y., Liu, D., and Dong, M. (2021). Incidence risk of PD-1/PD-L1-related pneumonia and diarrhea in non-small cell lung cancer (NSCLC) patients: A systematic review and meta-analysis of randomized controlled trials. Eur. J. Clin. Pharmacol. 77 (8), 1079–1088. doi:10.1007/s00228-020-03083-9

van Hasselt, J. G. C., Rahman, R., Hansen, J., Stern, A., Shim, J. V., Xiong, Y., et al. (2020). Transcriptomic profiling of human cardiac cells predicts protein kinase inhibitor-associated cardiotoxicity. Nat. Commun. 11 (1), 4809. doi:10.1038/s41467-020-18396-7

van Hunsel, F., van de Koppel, S., van Puijenbroek, E., and Kant, A. (2018). Vitamin B(6) in health supplements and neuropathy: Case series assessment of spontaneously reported cases. Drug Saf. 41 (9), 859–869. doi:10.1007/s40264-018-0664-0

Wang, D. Y., Salem, J. E., Cohen, J. V., Chandra, S., Menzer, C., Ye, F., et al. (2018). Fatal toxic effects associated with immune checkpoint inhibitors: A systematic review and meta-analysis. JAMA Oncol. 4 (12), 1721–1728. doi:10.1001/jamaoncol.2018.3923

Wu, A. K., Cheng, V. C., Tang, B. S., Hung, I. F. N., Lee, R. A., Hui, D. S., et al. (2004). The unmasking of pneumocystis jiroveci pneumonia during reversal of immunosuppression: Case reports and literature review. BMC Infect. Dis. 4 (1), 57. doi:10.1186/1471-2334-4-57

Zhu, J., Chen, G., He, Z., Zheng, Y., Gao, S., Li, J., et al. (2021). Stevens-johnson syndrome/toxic epidermal necrolysis in patients treated with immune checkpoint inhibitors: A safety analysis of clinical trials and FDA pharmacovigilance database. EClinicalMedicine 37, 100951. doi:10.1016/j.eclinm.2021.100951

Zhu, J., He, Z., Liang, D., Yu, X., Qiu, K., and Wu, J. (2022). Pulmonary tuberculosis associated with immune checkpoint inhibitors: A pharmacovigilance study. Thorax 77 (7), 721–723. doi:10.1136/thoraxjnl-2021-217575

Zhu, J., Luo, M., Liang, D., Gao, S., Zheng, Y., He, Z., et al. (2022). Type 1 diabetes with immune checkpoint inhibitors: A systematic analysis of clinical trials and a pharmacovigilance study of postmarketing data. Int. Immunopharmacol. 110, 109053. doi:10.1016/j.intimp.2022.109053

Keywords: immune checkpoint inhibitors, pneumocystis jirovecii pneumonia, pharmacovigilance, FAERS, systematic literature review

Citation: Xia S, Gong H, Wang Y-k, Liu L, Zhao Y-c, Guo L, Zhang B-k, Sarangdhar M, Noguchi Y and Yan M (2023) Pneumocystis jirovecii pneumonia associated with immune checkpoint inhibitors: A systematic literature review of published case reports and disproportionality analysis based on the FAERS database. Front. Pharmacol. 14:1129730. doi: 10.3389/fphar.2023.1129730

Received: 22 December 2022; Accepted: 03 March 2023; Published: 15 March 2023.

Reviewed by:

Copyright © 2023 Xia, Gong, Wang, Liu, Zhao, Guo, Zhang, Sarangdhar, Noguchi and Yan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Miao Yan, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Online First
  • Rapid reviews methods series: guidance on rapid qualitative evidence synthesis
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • http://orcid.org/0000-0003-4808-3880 Andrew Booth 1 , 2 ,
  • Isolde Sommer 3 , 4 ,
  • Jane Noyes 2 , 5 ,
  • Catherine Houghton 2 , 6 ,
  • Fiona Campbell 1 , 7
  • The Cochrane Rapid Reviews Methods Group and Cochrane Qualitative and Implementation Methods Group (CQIMG)
  • 1 EnSyGN Sheffield Evidence Synthesis Group , University of Sheffield , Sheffield , UK
  • 2 Cochrane Qualitative and Implementation Methods Group (CQIMG) , London , UK
  • 3 Department for Evidence-based Medicine and Evaluation , University for Continuing Education Krems , Krems , Austria
  • 4 Cochrane Rapid Reviews Group & Cochrane Austria , Krems , Austria
  • 5 Bangor University , Bangor , UK
  • 6 University of Galway , Galway , Ireland
  • 7 University of Newcastle upon Tyne , Newcastle upon Tyne , UK
  • Correspondence to Professor Andrew Booth, Univ Sheffield, Sheffield, UK; a.booth{at}sheffield.ac.uk

This paper forms part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group and addresses rapid qualitative evidence syntheses (QESs), which use modified systematic, transparent and reproducible methodsu to accelerate the synthesis of qualitative evidence when faced with resource constraints. This guidance covers the review process as it relates to synthesis of qualitative research. ‘Rapid’ or ‘resource-constrained’ QES require use of templates and targeted knowledge user involvement. Clear definition of perspectives and decisions on indirect evidence, sampling and use of existing QES help in targeting eligibility criteria. Involvement of an information specialist, especially in prioritising databases, targeting grey literature and planning supplemental searches, can prove invaluable. Use of templates and frameworks in study selection and data extraction can be accompanied by quality assurance procedures targeting areas of likely weakness. Current Cochrane guidance informs selection of tools for quality assessment and of synthesis method. Thematic and framework synthesis facilitate efficient synthesis of large numbers of studies or plentiful data. Finally, judicious use of Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research assessments and of software as appropriate help to achieve a timely and useful review product.

  • Systematic Reviews as Topic
  • Patient Care

Data availability statement

No data are available. Not applicable. All data is from published articles.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjebm-2023-112620

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Rapid Qualitative Evidence Synthesis (QES) is a relatively recent innovation in evidence synthesis and few published examples currently exists.

Guidance for authoring a rapid QES is scattered and requires compilation and summary.

WHAT THIS STUDY ADDS

This paper represents the first attempt to compile current guidance, illustrated by the experience of several international review teams.

We identify features of rapid QES methods that could be accelerated or abbreviated and where methods resemble those for conventional QESs.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

This paper offers guidance for researchers when conducting a rapid QES and informs commissioners of research and policy-makers what to expect when commissioning such a review.

Introduction

This paper forms part of a series from the Cochrane Rapid Reviews Methods Group providing methodological guidance for rapid reviews. While other papers in the series 1–4 focus on generic considerations, we aim to provide in-depth recommendations specific to a resource-constrained (or rapid) qualitative evidence synthesis (rQES). 5 This paper is accompanied by recommended resources ( online supplemental appendix A ) and an elaboration with practical considerations ( online supplemental appendix B ).

Supplemental material

The role of qualitative evidence in decision-making is increasingly recognised. 6 This, in turn, has led to appreciation of the value of qualitative evidence syntheses (QESs) that summarise findings across multiple contexts. 7 Recognition of the need for such syntheses to be available at the time most useful to decision-making has, in turn, driven demand for rapid qualitative evidence syntheses. 8 The breadth of potential rQES mirrors the versatility of QES in general (from focused questions to broad overviews) and outputs range from descriptive thematic maps through to theory-informed syntheses (see table 1 ).

  • View inline

Glossary of important terms (alphabetically)

As with other resource-constrained reviews, no one size fits all. A team should start by specifying the phenomenon of interest, the review question, 9 the perspectives to be included 9 and the sample to be determined and selected. 10 Subsequently, the team must finalise the appropriate choice of synthesis. 11 Above all, the review team should consider the intended knowledge users, 3 including requirements of the funder.

An rQES team, in particular, cannot afford any extra time or resource requirements that might arise from either a misunderstanding of the review question, an unclear picture of user requirements or an inappropriate choice of methods. The team seeks to align the review question and the requirements of the knowledge user with available time and resources. They also need to ensure that the choice of data and choice of synthesis are appropriate to the intended ‘knowledge claims’ (epistemology) made by the rQES. 11 This involves the team asking ‘what types of data are meaningful for this review question?’, ‘what types of data are trustworthy?’ and ‘is the favoured synthesis method appropriate for this type of data?’. 12 This paper aims to help rQES teams to choose methods that best fit their project while understanding the limitations of those choices. Our recommendations derive from current QES guidance, 5 evidence on modified QES methods, 8 13 and practical experience. 14 15

This paper presents an overview of considerations and recommendations as described in table 2 . Supplemental materials including additional resources details of our recommendations and practical examples are provided in online supplemental appendices A and B .

Recommendations for resource-constrained qualitative evidence synthesis (rQES)

Setting the review question and topic refinement

Rapid reviews summarise information from multiple research studies to produce evidence for ‘the public, researchers, policymakers and funders in a systematic, resource-efficient manner’. 16 Involvement of knowledge users is critical. 3 Given time constraints, individual knowledge users could be asked only to feedback on very specific decisions and tasks or on selective sections of the protocol. Specifically, whenever a QES is abbreviated or accelerated, a team should ensure that the review question is agreed by a minimum number of knowledge users with expertise or experience that reflects all the important review perspectives and with authority to approve the final version 2 5 11 ( table 2 , item R1).

Involvement of topic experts can ensure that the rQES is responsive to need. 14 17 One Cochrane rQES saved considerable time by agreeing the review topic within a single meeting and one-phase iteration. 9 Decisions on topics to be omitted are also informed by a knowledge of existing QESs. 17

An information specialist can help to manage the quantity and quality of available evidence by setting conceptual boundaries and logistic limits. A structured question format, such as Setting-Perspective-Interest, phenomenon of-Comparison-Evaluation or Population-Interest, phenomenon of-Context helps in communicating the scope and, subsequently, in operationalising study selection. 9 18

Scoping (of review parameters) and mapping (of key types of evidence and likely richness of data) helps when planning the review. 5 19 The option to choose purposive sampling over comprehensive sampling approaches, as offered by standard QES, may be particularly helpful in the context of a rapid QES. 8 Once a team knows the approximate number and distribution of studies, perhaps mapping them against country, age, ethnicity, etc), they can decide whether or not to use purposive sampling. 12 An rQES for the WHO combined purposive with variation sampling. Sampling in two stages started by reducing the initial number of studies to a more manageable sampling frame and then sampling approximately a third of the remaining studies from within the sampling frame. 20

Sampling may target richer studies and/or privilege diversity. 8 21 A rich qualitative study typically illustrates findings with verbatim extracts from transcripts from interviews or textual responses from questionnaires. Rich studies are often found in specialist qualitative research or social science journals. In contrast, less rich studies may itemise themes with an occasional indicative text extract and tend to summarise findings. In clinical or biomedical journals less rich findings may be placed within a single table or box.

No rule exists on an optimal number of studies; too many studies makes it challenging to ‘maintain insight’, 22 too few does not sustain rigorous analysis. 23 Guidance on sampling is available from the forthcoming Cochrane-Campbell QES Handbook.

A review team can use templates to fast-track writing of a protocol. The protocol should always be publicly available ( table 2 , item R2). 24 25 Formal registration may require that the team has not commenced data extraction but should be considered if it does not compromise the rQES timeframe. Time pressures may require that methods are left suitably flexible to allow well-justified changes to be made as a detailed picture of the studies and data emerge. 26 The first Cochrane rQES drew heavily on text from a joint protocol/review template previously produced within Cochrane. 24

Setting eligibility criteria

An rQES team may need to limit the number of perspectives, focusing on those most important for decision-making 5 9 27 ( table 2 , item R3). Beyond the patients/clients each additional perspective (eg, family members, health professionals, other professionals, etc) multiplies the additional effort involved.

A rapid QES may require strict date and setting restrictions 17 and language restrictions that accommodate the specific requirements of the review. Specifically, the team should consider whether changes in context over time or substantive differences between geographical regions could be used to justify a narrower date range or a limited coverage of countries and/or languages. The team should also decide if ‘indirect evidence’ is to substitute for the absence of direct evidence. An rQES typically focuses on direct evidence, except when only indirect evidence is available 28 ( table 2 , item R4). Decisions on relevance are challenging—precautions for swine influenza may inform precautions for bird influenza. 28 A smoking ban may operate similarly to seat belt legislation, etc. A review team should identify where such shared mechanisms might operate. 28 An rQES team must also decide whether to use frameworks or models to focus the review. Theories may be unearthed within the topic search or be already known to team members, fro example, Theory of Planned Behaviour. 29

Options for managing the quantity and quality of studies and data emerge during the scoping (see above). In summary, the review team should consider privileging rich qualitative studies 2 ; consider a stepwise approach to inclusion of qualitative data and explore the possibility of sampling ( table 2 , item R5). For example, where data is plentiful an rQES may be limited to qualitative research and/or to mixed methods studies. Where data is less plentiful then surveys or other qualitative data sources may need to be included. Where plentiful reviews already exist, a team may decide to conduct a review of reviews 5 by including multiple QES within a mega-synthesis 28 29 ( table 2 , item R6).

Searching for QES merits its own guidance, 21–23 30 this section reinforces important considerations from guidance specific to qualitative research. Generic guidance for rapid reviews in this series broadly applies to rapid QESs. 1

In addition to journal articles, by far the most plentiful source, qualitative research is found in book chapters, theses and in published and unpublished reports. 21 Searches to support an rQES can (a) limit the number of databases searched, deliberately selecting databases from diverse disciplines, (b) use abbreviated study filters to retrieve qualitative designs and (c) employ high yield complementary methods (eg, reference checking, citation searching and Related Articles features). An information specialist (eg, librarian) should be involved in prioritising sources and search methods ( table 2 , item R7). 11 14

According to empirical evidence optimal database combinations include Scopus plus CINAHL or Scopus plus ProQuest Dissertations and Theses Global (two-database combinations) and Scopus plus CINAHL plus ProQuest Dissertations and Theses Global (three-database combination) with both choices retrieving between 89% and 92% of relevant studies. 30

If resources allow, searches should include one or two specialised databases ( table 2 , item R8) from different disciplines or contexts 21 (eg, social science databases, specialist discipline databases or regional or institutional repositories). Even when resources are limited, the information specialist should factor in time for peer review of at least one search strategy ( table 2 , item R9). 31 Searches for ‘grey literature’ should selectively target appropriate types of grey literature (such as theses or process evaluations) and supplemental searches, including citation chaining or Related Articles features ( table 2 , item R10). 32 The first Cochrane rQES reported that searching reference lists of key papers yielded an extra 30 candidate papers for review. However, the team documented exclusion of grey literature as a limitation of their review. 15

Study selection

Consistency in study selection is achieved by using templates, by gaining a shared team understanding of the audience and purpose, and by ongoing communication within, and beyond, the team. 2 33 Individuals may work in parallel on the same task, as in the first Cochrane rQES, or follow a ‘segmented’ approach where each reviewer is allocated a different task. 14 The use of machine learning in the specific context of rQES remains experimental. However, the possibility of developing qualitative study classifiers comparable to those for randomised controlled trials offers an achievable aspiration. 34

Title and abstract screening

The entire screening team should use pre-prepared, pretested title and abstract templates to limit the scale of piloting, calibration and testing ( table 2 , item R11). 1 14 The first Cochrane rQES team double-screened titles and abstracts within Covidence review software. 14 Disagreements were resolved with reference to a third reviewer achieving a shared understanding of the eligibility criteria and enhancing familiarity with target studies and insight from data. 14 The team should target and prioritise identified risks of either over-zealous inclusion or over-exclusion specific to each rQES ( table 2 , item R12). 14 The team should maximise opportunities to capture divergent views and perspectives within study findings. 35

Full-text screening

Full-text screening similarly benefits from using a pre-prepared pretested standardised template where possible 1 14 ( table 2 , item R11). If a single reviewer undertakes full-text screening, 8 the team should identify likely risks to trustworthiness of findings and focus quality control procedures (eg, use of additional reviewers and percentages for double screening) on specific threats 14 ( table 2 , item R13). The Cochrane rQES team opted for double screening to assist their immersion within the topic. 14

Data extraction

Data extraction of descriptive/contextual data may be facilitated by review management software (eg, EPPI-Reviewer) or home-made approaches using Google Forms, or other survey software. 36 Where extraction of qualitative findings requires line-by-line coding with multiple iterations of the data then a qualitative data management analysis package, such as QSR NVivo, reaps dividends. 36 The team must decide if, collectively, they favour extracting data to a template or coding direct within an electronic version of an article.

Quality control must be fit for purpose but not excessive. Published examples typically use a single reviewer for data extraction 8 with use of two independent reviewers being the exception. The team could limit data extraction to minimal essential items. They may also consider re-using descriptive details and findings previously extracted within previous well-conducted QES ( table 2 , item R14). A pre-existing framework, where readily identified, may help to structure the data extraction template. 15 37 The same framework may be used to present the findings. Some organisations may specify a preferred framework, such as an evidence-to-decision-making framework. 38

Assessment of methodological limitations

The QES community assess ‘methodological limitations’ rather than use ‘risk of bias’ terminology. An rQES team should pick an approach appropriate to their specific review. For example, a thematic map may not require assessment of individual studies—a brief statement of the generic limitations of the set of studies may be sufficient. However, for any synthesis that underpins practice recommendations 39 assessment of included studies is integral to the credibility of findings. In any decision-making context that involves recommendations or guidelines, an assessment of methodological limitations is mandatory. 40 41

Each review team should work with knowledge users to determine a review-specific approach to quality assessment. 27 While ‘traffic lights’, similar to the outputs from the Cochrane Risk of Bias tool, may facilitate rapid interpretation, accompanying textual notes are invaluable in highlighting specific areas for concern. In particular, the rQES team should demonstrate that they are aware (a) that research designs for qualitative research seek to elicit divergent views, rather than control for variation; (b) that, for qualitative research, the selection of the sample is far more informative than the size of the sample; and (c) that researchers from primary research, and equally reviewers for the qualitative synthesis, need to be thoughtful and reflexive about their possible influences on interpretation of either the primary data or the synthesised findings.

Selection of checklist

Numerous scales and checklists exist for assessing the quality of qualitative studies. In the absence of validated risk of bias tools for qualitative studies, the team should choose a tool according to Cochrane Qualitative and Implementation Methods Group (CQIMG) guidance together with expediency (according to ease of use, prior familiarity, etc) ( table 2 , item R15). 41 In comparison to the Critical Appraisal Skills Programme checklist which was never designed for use in synthesis, 42 the Cochrane qualitative tool is similarly easy to use and was designed for QES use. Work is underway to identify an assessment process that is compatible with QESs that support decision-making. 41 For now the choice of a checklist remains determined by interim Cochrane guidance and, beyond this, by personal preference and experience. For an rQES a team could use a single reviewer to assess methodological limitations, with verification of judgements (and support statements) by a second reviewer ( table 2 , item R16).

The CQIMG endorses three types of synthesis; thematic synthesis, framework synthesis and meta-ethnography ( box 1 ). 43 44 Rapid QES favour descriptive thematic synthesis 45 or framework synthesis, 46 47 except when theory generation (meta-ethnography 48 49 or analytical thematic synthesis) is a priority ( table 2 , item R17).

Choosing a method for rapid qualitative synthesis

Thematic synthesis: first choice method for rQES. 45 For example, in their rapid QES Crooks and colleagues 44 used a thematic synthesis to understand the experiences of both academic and lived experience coresearchers within palliative and end of life research. 45

Framework synthesis: alternative where a suitable framework can be speedily identified. 46 For example, Bright and colleagues 46 considered ‘best-fit framework synthesis’ as appropriate for mapping study findings to an ‘a priori framework of dimensions measured by prenatal maternal anxiety tools’ within their ‘streamlined and time-limited evidence review’. 47

Less commonly, an adapted meta-ethnographical approach was used for an implementation model of social distancing where supportive data (29 studies) was plentiful. 48 However, this QES demonstrates several features that subsequently challenge its original identification as ‘rapid’. 49

Abbrevations: QES, qualitative evidence synthesis; rQES, resource-constrained qualitative evidence synthesis.

The team should consider whether a conceptual model, theory or framework offers a rapid way for organising, coding, interpreting and presenting findings ( table 2 , item R18). If the extracted data appears rich enough to sustain further interpretation, data from a thematic or framework synthesis can subsequently be explored within a subsequent meta-ethnography. 43 However, this requires a team with substantial interpretative expertise. 11

Assessments of confidence in the evidence 4 are central to any rQES that seeks to support decision-making and the QES-specific Grading of Recommendations Assessment, Development and Evaluation approach for assessing the Confidence of Evidence from Reviews of Qualitative research (GRADE-CERQual) approach is designed to assess confidence in qualitative evidence. 50 This can be performed by a single reviewer, confirmed by a second reviewer. 26 Additional reviewers could verify all, or a sample of, assessments. For a rapid assessment a team must prioritise findings, using objective criteria; a WHO rQES focused only on the three ‘highly synthesised findings’. 20 The team could consider reusing GRADE-CERQual assessments from published QESs if findings are relevant and of demonstrable high quality ( table 2 , item R19). 50 No rapid approach to full application of GRADE-CERQual currently exists.

Reporting and record management

Little is written on optimal use of technology. 8 A rapid review is not a good time to learn review management software or qualitative analysis management software. Using such software for all general QES processes ( table 2 , item R20), and then harnessing these skills and tools when specifically under resource pressures, is a sounder strategy. Good file labelling and folder management and a ‘develop once, re-use multi-times’ approach facilitates resource savings.

Reporting requirements include the meta-ethnography reporting guidance (eMERGe) 51 and the Enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) statement. 52 An rQES should describe limitations and their implications for confidence in the evidence even more thoroughly than a regular QES; detailing the consequences of fast-tracking, streamlining or of omitting processes all together. 8 Time spent documenting reflexivity is similarly important. 27 If QES methodology is to remain credible rapid approaches must be applied with insight and documented with circumspection. 53 54 (56)

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

  • Klerings I ,
  • Robalino S ,
  • Booth A , et al
  • Nussbaumer-Streit B ,
  • Hamel C , et al
  • Garritty C ,
  • Tricco AC ,
  • Smith M , et al
  • Gartlehner G ,
  • Devane D , et al
  • NHS Scotland
  • Campbell F ,
  • Flemming K , et al
  • Glenton C ,
  • Lubarsky S ,
  • Varpio L , et al
  • Meskell P ,
  • Glenton C , et al
  • Houghton C ,
  • Delaney H , et al
  • Beecher C ,
  • Maeso B , et al
  • McKenzie JE , et al
  • Harris JL ,
  • Cargo M , et al
  • Varley-Campbell J , et al
  • Downe S , et al
  • Shamseer L ,
  • Clarke M , et al
  • Nussbaumer-Streit B , et al
  • Finlayson KW ,
  • Lawrie TA , et al
  • Lewin S , et al
  • Frandsen TF ,
  • Gildberg FA ,
  • Tingleff EB
  • Mshelia S ,
  • Analo CV , et al
  • Husk K , et al
  • Carmona C ,
  • Carroll C ,
  • Ilott I , et al
  • Meehan B , et al
  • Munthe-Kaas H ,
  • Bohren MA ,
  • Munthe-Kaas HM ,
  • French DP ,
  • Flemming K ,
  • Garside R , et al
  • Shulman C , et al
  • Dixon-Woods M
  • Bright KS ,
  • Norris JM ,
  • Letourneau NL , et al
  • Sadjadi M ,
  • Mörschel KS ,
  • Petticrew M
  • France EF ,
  • Cunningham M ,
  • Ring N , et al
  • McInnes E , et al
  • Britten N ,
  • Garside R ,
  • Pope C , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Data supplement 1

Contributors All authors (AB, IS, JN, CH, FC) have made substantial contributions to the conception and design of the guidance document. AB led on drafting the work and revising it critically for important intellectual content. All other authors (IS, JN, CH, FC) contributed to revisions of the document. All authors (AB, IS, JN, CH, FC) have given final approval of the version to be published. As members of the Cochrane Qualitative and Implementation Methods Group and/or the Cochrane Rapid Reviews Methods Group all authors (AB, IS, JN, CH, FC) agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests AB is co-convenor of the Cochrane Qualitative and Implementation Methods Group. In the last 36 months, he received royalties from Systematic Approaches To a Successful Literature Review (Sage 3rd edition), honoraria from the Agency for Healthcare Research and Quality, and travel support from the WHO. JN is lead convenor of the Cochrane Qualitative and Implementation Methods Group. In the last 36 months, she has received honoraria from the Agency for Healthcare Research and Quality and travel support from the WHO. CH is co-convenor of the Cochrane Qualitative and Implementation Methods Group.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; internally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

IMAGES

  1. 50 Smart Literature Review Templates (APA) ᐅ TemplateLab

    published literature review

  2. 50 Smart Literature Review Templates (APA) ᐅ TemplateLab

    published literature review

  3. How To Write A Literature Review

    published literature review

  4. literature review article examples Sample of research literature review

    published literature review

  5. Helping You in Writing a Literature Review Immaculately

    published literature review

  6. 50 Smart Literature Review Templates (APA) ᐅ TemplateLab

    published literature review

VIDEO

  1. Literature Review

  2. Chapter 5 Review of Literature PART 02

  3. Methods L04

  4. Systematic Literature Review Paper

  5. How to write literature review #academicsuccess #thesiswriting #school #students

  6. Effective Review of Literature

COMMENTS

  1. How to Write a Literature Review

    Step 1 - Search for relevant literature Step 2 - Evaluate and select sources Step 3 - Identify themes, debates, and gaps Step 4 - Outline your literature review's structure Step 5 - Write your literature review Free lecture slides Other interesting articles Frequently asked questions Introduction Quick Run-through Step 1 & 2 Step 3 Step 4 Step 5

  2. Writing a Literature Review

    A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays).

  3. Literature Reviews

    A literature review discusses published information in a particular subject area, and sometimes information in a particular subject area within a certain time period. A literature review can be just a simple summary of the sources, but it usually has an organizational pattern and combines both summary and synthesis.

  4. What is a literature review?

    A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations.

  5. Literature review as a research methodology: An overview and guidelines

    A literature review can broadly be described as a more or less systematic way of collecting and synthesizing previous research ( Baumeister & Leary, 1997Tranfield, Denyer, & Smart, 2003 ).

  6. Literature Review: The What, Why and How-to Guide

    "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are.

  7. Writing a literature review

    Rather than just being an exhaustive list of all that has been published, a literature review should be an informative, personal but unbiased synopsis of the information, providing a balanced view that includes conflicting findings and inconsistencies, as well as established and current thinking.

  8. Literature Review: Examples of Published Literature Reviews

    Literature Review (Historiographic Essay): Making sense of what has been written on your topic. To find examples of published literature reviews in your field or niche, try searching ProQuest Dissertations and Theses by keyword, advisor, or subject. Dissertations & Theses Theses and dissertations from around the world covering all areas of study.

  9. How to write a superb literature review

    The best proposals are timely and clearly explain why readers should pay attention to the proposed topic. It is not enough for a review to be a summary of the latest growth in the literature: the ...

  10. What is a Literature Review?

    A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research. There are five key steps to writing a literature review: Search for relevant literature. Evaluate sources. Identify themes, debates and gaps.

  11. Steps in Conducting a Literature Review

    A literature review is an integrated analysis-- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

  12. 5. The Literature Review

    A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated.

  13. Literature Review Overview

    A literature review discusses published information in a particular subject area. Often part of the introduction to an essay, research report or thesis, the literature review is literally a "re" view or "look again" at what has already been written about the topic, wherein the author analyzes a segment of a published body of knowledge through summary, classification, and comparison of prior ...

  14. Literature review

    A literature review is a piece of academic writing demonstrating knowledge and understanding of the academic literature on a specific topic placed in context. A literature review also includes a critical evaluation of the material; this is why it is called a literature review rather than a literature report.

  15. Ten Simple Rules for Writing a Literature Review

    Rule 1: Define a Topic and Audience How to choose which topic to review? There are so many issues in contemporary science that you could spend a lifetime of attending conferences and reading the literature just pondering what to review.

  16. Creating a Literature Review

    A literature review requires the writer to perform extensive research on published work in one's field in order to explain how one's own work fits into the larger conversation regarding a particular topic. This task requires the writer to spend time reading, managing, and conveying information; the complexity of literature reviews can make ...

  17. Guidance on Conducting a Systematic Literature Review

    Literature review is an essential feature of academic research. Fundamentally, knowledge advancement must be built on prior existing work. To push the knowledge frontier, we must know where the frontier is. By reviewing relevant literature, we understand the breadth and depth of the existing body of work and identify gaps to explore.

  18. Want to publish a literature review? Think of it as an empirical paper

    The literature review publications come in so many forms, and there is no single cheat-sheet or established format like for empirical papers that you could follow to ensure success in publication.

  19. Writing a Literature Review Research Paper: A step-by-step approach

    ... Analysing literature gives an overview of the "WHs": WHat has been reported in a particular feld or topic, WHo the key writers are, WHat are the prevailing theories and hypotheses, WHat...

  20. Chapter 9 Methods for Literature Reviews

    Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generat...

  21. Publishing Literature Reviews

    The foundation for conducting literature reviews has been laid in the previous chapters from setting the topic in Chapter 4 to the analysis and synthesis in later chapters and the reporting in Chapter 13. Download chapter PDF. With the credo of academics being 'publish or perish', much attention is given to publishing output of studies ...

  22. The Literature Review: A Foundation for High-Quality Medical Education

    The Literature Review Defined In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth.

  23. Finding Published Literature Reviews

    Step 1: PsycInfo uses Literature Review as a Form/Content Type. Step 2: In EBSCO Select Methodology: Literature Review and combine with terms for your topic. Step 3: Publication Type - Review has book, software, etc. reviews.

  24. What is a Literature Review?

    What's a Literature Review? The straightforward answer is that a literature review is a review or synthesis of all the research published on a certain topic. But I'd rather explain it from a skateboarder's perspective: One of my favorite movies is the 1989 classic Back to the Future Part 2 where the bodacious skater Marty McFly time-travels to the future and sees a hoverboard.

  25. Improvement in work productivity among psoriatic arthritis patients

    Background Capacity to work is impacted by psoriatic arthritis (PsA). Our objective was to describe the course of work productivity and leisure activity in patients with PsA treated with biologic (b) and targeted synthetic (ts) disease-modifying antirheumatic drugs (DMARDs). Methods A systematic literature review identified all trials and observational studies published January 1, 2010 ...

  26. Tools for assessing quality of studies investigating health

    Objectives We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level. Design Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment ...

  27. Frontiers

    This is consistent with our literature review on published case reports (53.3% PJP cases associated with ICIs were male and age more than 65). Thirdly, our study provides more evidence to support that PJP may be unmasked in cancer patients in the process of immune reconstitution induced by immune checkpoint inhibitors.

  28. Rapid reviews methods series: guidance on rapid qualitative evidence

    Setting the review question and topic refinement. Rapid reviews summarise information from multiple research studies to produce evidence for 'the public, researchers, policymakers and funders in a systematic, resource-efficient manner'.16 Involvement of knowledge users is critical.3 Given time constraints, individual knowledge users could be asked only to feedback on very specific ...