a systematic literature review is

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism. Run a free check.

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved July 4, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

Introduction to Systematic Reviews

• Reference work entry
• First Online: 20 July 2022
• pp 2159–2177
• Cite this reference work entry

• Tianjing Li 3 ,
• Ian J. Saldanha 4 &
• Karen A. Robinson 5  

276 Accesses

A systematic review identifies and synthesizes all relevant studies that fit prespecified criteria to answer a research question. Systematic review methods can be used to answer many types of research questions. The type of question most relevant to trialists is the effects of treatments and is thus the focus of this chapter. We discuss the motivation for and importance of performing systematic reviews and their relevance to trialists. We introduce the key steps in completing a systematic review, including framing the question, searching for and selecting studies, collecting data, assessing risk of bias in included studies, conducting a qualitative synthesis and a quantitative synthesis (i.e., meta-analysis), grading the certainty of evidence, and writing the systematic review report. We also describe how to identify systematic reviews and how to assess their methodological rigor. We discuss the challenges and criticisms of systematic reviews, and how technology and innovations, combined with a closer partnership between trialists and systematic reviewers, can help identify effective and safe evidence-based practices more quickly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or Ebook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

What Is the Difference Between a Systematic Review and a Meta-analysis?

Systematic Reviewing

AHRQ (2015) Methods guide for effectiveness and comparative effectiveness reviews. Available from https://effectivehealthcare.ahrq.gov/products/cer-methods-guide/overview . Accessed on 27 Oct 2019

Andersen MZ, Gülen S, Fonnes S, Andresen K, Rosenberg J (2020) Half of Cochrane reviews were published more than two years after the protocol. J Clin Epidemiol 124:85–93. https://doi.org/10.1016/j.jclinepi.2020.05.011

Article   Google Scholar  

Berkman ND, Lohr KN, Ansari MT, Balk EM, Kane R, McDonagh M, Morton SC, Viswanathan M, Bass EB, Butler M, Gartlehner G, Hartling L, McPheeters M, Morgan LC, Reston J, Sista P, Whitlock E, Chang S (2015) Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol 68(11):1312–1324

Borah R, Brown AW, Capers PL, Kaiser KA (2017) Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JP, Oliver S (2014) How to increase value and reduce waste when research priorities are set. Lancet 383(9912):156–165. https://doi.org/10.1016/S0140-6736(13)62229-1

Clarke M, Chalmers I (1998) Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 280(3):280–282

Cooper NJ, Jones DR, Sutton AJ (2005) The use of systematic reviews when designing studies. Clin Trials 2(3):260–264

Djulbegovic B, Kumar A, Magazin A, Schroen AT, Soares H, Hozo I, Clarke M, Sargent D, Schell MJ (2011) Optimism bias leads to inconclusive results-an empirical study. J Clin Epidemiol 64(6):583–593. https://doi.org/10.1016/j.jclinepi.2010.09.007

Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, Salanti G, Meerpohl J, MacLehose H, Hilton J, Tovey D, Shemilt I, Thomas J (2017) Living systematic review network. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol 91:23–30

Equator Network. Reporting guidelines for systematic reviews. Available from https://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=+ . Accessed 9 Mar 2020

Garner P, Hopewell S, Chandler J, MacLehose H, Schünemann HJ, Akl EA, Beyene J, Chang S, Churchill R, Dearness K, Guyatt G, Lefebvre C, Liles B, Marshall R, Martínez García L, Mavergames C, Nasser M, Qaseem A, Sampson M, Soares-Weiser K, Takwoingi Y, Thabane L, Trivella M, Tugwell P, Welsh E, Wilson EC, Schünemann HJ (2016) Panel for updating guidance for systematic reviews (PUGs). When and how to update systematic reviews: consensus and checklist. BMJ 354:i3507. https://doi.org/10.1136/bmj.i3507 . Erratum in: BMJ 2016 Sep 06 354:i4853

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ (2011) GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4):383–394

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) (2019a) Cochrane handbook for systematic reviews of interventions, 2nd edn. Wiley, Chichester

Google Scholar  

Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R (2019b) Standards for the conduct of new Cochrane intervention reviews. In: JPT H, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R (eds) Methodological expectations of Cochrane intervention reviews. Cochrane, London

IOM (2011) Committee on standards for systematic reviews of comparative effectiveness research, board on health care services. In: Eden J, Levit L, Berg A, Morton S (eds) Finding what works in health care: standards for systematic reviews. National Academies Press, Washington, DC

Jonnalagadda SR, Goyal P, Huffman MD (2015) Automating data extraction in systematic reviews: a systematic review. Syst Rev 4:78

Krnic Martinic M, Pieper D, Glatt A, Puljak L (2019) Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol 19(1):203. Published 4 Nov 2019. https://doi.org/10.1186/s12874-019-0855-0

Lasserson TJ, Thomas J, Higgins JPT (2019) Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook

Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC (1992) Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med 327(4):248–254

Lau J (2019) Editorial: systematic review automation thematic series. Syst Rev 8(1):70. Published 11 Mar 2019. https://doi.org/10.1186/s13643-019-0974-z

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6(7):e1000100. https://doi.org/10.1371/journal.pmed.1000100

Lund H, Brunnhuber K, Juhl C, Robinson K, Leenaars M, Dorch BF, Jamtvedt G, Nortvedt MW, Christensen R, Chalmers I (2016) Towards evidence based research. BMJ 355:i5440. https://doi.org/10.1136/bmj.i5440

Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC (2018) Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide. Res Synth Methods 9(4):602–614. https://doi.org/10.1002/jrsm.1287

Michelson M, Reuter K (2019) The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun 16:100443. https://doi.org/10.1016/j.conctc.2019.100443 . Erratum in: Contemp Clin Trials Commun 2019 16:100450

Moher D, Liberati A, Tetzlaff J (2009) Altman DG; PRISMA group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151(4):264–269. W64

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA-P Group (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4(1):1. https://doi.org/10.1186/2046-4053-4-1

NIHR HTA Stage 1 guidance notes. Available from https://www.nihr.ac.uk/documents/hta-stage-1-guidance-notes/11743 ; Accessed 10 Mar 2020

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catalá-López F, Li L, Reid EK, Sarkis-Onofre R, Moher D (2016) Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 13(5):e1002028. https://doi.org/10.1371/journal.pmed.1002028

Page MJ, Higgins JPT, Sterne JAC (2019) Chapter 13: assessing risk of bias due to missing results in a synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ et al (eds) Cochrane handbook for systematic reviews of interventions, 2nd edn. Wiley, Chichester, pp 349–374

Chapter   Google Scholar  

Robinson KA (2009) Use of prior research in the justification and interpretation of clinical trials. Johns Hopkins University

Robinson KA, Goodman SN (2011) A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 154(1):50–55. https://doi.org/10.7326/0003-4819-154-1-201101040-00007

Rouse B, Cipriani A, Shi Q, Coleman AL, Dickersin K, Li T (2016) Network meta-analysis for clinical practice guidelines – a case study on first-line medical therapies for primary open-angle glaucoma. Ann Intern Med 164(10):674–682. https://doi.org/10.7326/M15-2367

Saldanha IJ, Lindsley K, Do DV et al (2017) Comparison of clinical trial and systematic review outcomes for the 4 most prevalent eye diseases. JAMA Ophthalmol 135(9):933–940. https://doi.org/10.1001/jamaophthalmol.2017.2583

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008. https://doi.org/10.1136/bmj.j4008

Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147(4):224–233

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JP (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernán MA, Hopewell S, Hróbjartsson A, Junqueira DR, Jüni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S (2019) Chapter 2: determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook

USPSTF U.S. Preventive Services Task Force Procedure Manual (2017). Available from: https://www.uspreventiveservicestaskforce.org/uspstf/sites/default/files/inline-files/procedure-manual2017_update.pdf . Accessed 21 May 2020

Whitaker (2015) UCSF guides: systematic review: when will i be finished? https://guides.ucsf.edu/c.php?g=375744&p=3041343 , Accessed 13 May 2020

Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J (2016) Churchill R; ROBIS group. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005

Download references

Author information

Authors and affiliations.

Department of Ophthalmology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA

Tianjing Li

Department of Health Services, Policy, and Practice and Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA

Ian J. Saldanha

Department of Medicine, Johns Hopkins University, Baltimore, MD, USA

Karen A. Robinson

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tianjing Li .

Editor information

Editors and affiliations.

Department of Surgery, Division of Surgical Oncology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

Steven Piantadosi

Department of Epidemiology, School of Public Health, Johns Hopkins University, Baltimore, MD, USA

Curtis L. Meinert

Section Editor information

Department of Epidemiology, University of Colorado Denver Anschutz Medical Campus, Aurora, CO, USA

The Johns Hopkins Center for Clinical Trials and Evidence Synthesis, Johns Hopkins University, Baltimore, MD, USA

Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Cite this entry.

Li, T., Saldanha, I.J., Robinson, K.A. (2022). Introduction to Systematic Reviews. In: Piantadosi, S., Meinert, C.L. (eds) Principles and Practice of Clinical Trials. Springer, Cham. https://doi.org/10.1007/978-3-319-52636-2_194

Download citation

DOI : https://doi.org/10.1007/978-3-319-52636-2_194

Published : 20 July 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-52635-5

Online ISBN : 978-3-319-52636-2

eBook Packages : Mathematics and Statistics Reference Module Computer Science and Engineering

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

What is a Systematic Literature Review?

A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.

An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.

In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms . As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.

• Charts of BSWL workshop (pdf, 2.88 MB)
• Listen to the interview (mp4, 12.35 MB)

Differences to "common" literature reviews

What are the objectives of SLRs?

• Avoidance of research redundancies despite a growing amount of publications
• Identification of research areas, gaps and methods
• Input for evidence-based management, which allows to base management decisions on scientific methods and findings
• Identification of links between different areas of researc

Process steps of an SLR

A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:

1. Defining research questions

Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:

C – CONTEXT:  Which individuals, relationships, institutional frameworks and systems are being investigated?

I – Intervention:  The effects of which event, action or activity are being investigated?

M – Mechanisms:  Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?

O – Outcomes:  What are the effects of the intervention? How are the results measured? What are intended and unintended effects?

The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.

With the aid of the scheme, the following concrete research questions can be derived with regard to this example:

Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?

Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?          

Research questions do not necessarily need to follow the CIMO scheme, but they should:

• ... be formulated in a clear, focused and comprehensible manner and be answerable;
• ... have been determined prior to carrying out the SLR;
• ... consist of general and specific questions.

As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).

2. Selecting databases and other research sources

The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.

In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:

• Business Source Complete
• ProQuest One Business
• EconBiz        

Our video " Selecting the right databases " explains how to find relevant databases for your topic.

Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias ).  Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias ( tutorial on publication bias ).

Our online self-study course " Searching economic databases " explains step 2 und 3.

3. Defining search terms

Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.

A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example .

Identification of headwords and keywords

When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:

• appear in the title, abstract and/or text
• sometimes specified by the author, but in most cases automatically generated
• non-standardized
• different spellings and forms (singular/plural) must be searched separately

Subject headings

• describe the content
• are generated by an editorial team
• are listed in a standardized list (thesaurus)
• may comprise various keywords
• include different spellings
• database-specific

Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.

Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords. 

To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).

The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics . Headwords are integrated into the search by being used in the search query.

Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).

Searching in literature databases

Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.

Since the search query must be set up separately for each database, tools such as  LitSonar  have been developed to enable a systematic search across different databases. LitSonar was created by  Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.

Advanced search

Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.

Additional searches in sources other than literature databases

In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:

• the topic is new and not yet included in indexes of subject headings;
• search terms are not used congruently in articles because uniform definitions do not exist;
• some studies are still in the process of being published, or have been completed, but not published.

Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias” ) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.

The PRESS-Checklist can support you to check the correctness of your search terms.

4. Merging hits from different databases

In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use reference management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an  overview of current reference management programs  . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).

Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.

When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.

Exporting data from literature databases

Exporting from literature databases is very easy. In  Business Source Complete  , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.

Exporting data from the literature database  EconBiz  is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.

After merging all hits from the various databases, duplicate entries (duplicates) are deleted.

5. Applying inclusion and exclusion criteria

All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with. 

The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.

You can create tag words for the inclusion and exclusion in your literature management tool to keep an overview.

In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools

The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.

Forward and backward search

Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.

6. Perform the review

As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.

Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).

Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.

Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.

Example concept matrix (Becker 2013) on the topic Business Process Management

7. Synthesizing results

Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:

• historical development of topics (histogram, time series: when, and how frequently, did publications on the research topic appear?);
• overview of journals, authors or specialist disciplines dealing with the topic;
• comparison of applied statistical methods;
• topics covered by research;
• identifying research gaps;
• developing a reference framework;
• developing constructs;
• performing a meta-analysis: comparison of the correlations of the results of different empirical studies (see for example Fink 2014, p. 203 on conducting meta-analyses)

Publications about the method

Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support.  Communications of the Association for Information Systems . 34(8), 154-204.

Booth, A., Papaioannou, D., and Sutton, A. (2012)  Systematic approaches to a successful literature review.  London: Sage.

Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.),  The Oxford Handbook of Evidenence Based Management . (S. 112-129). Oxford: Oxford University Press.

Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review . International Journal of Physical Distribution & Logistic Management , 46 (1/2), 118-137. doi:  https://doi.org/10.1108/IJPDLM-05-2013-0133

Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review.  English in Today's Research World 2.  Ann Arbor: University of Michigan Press. doi:  10.3998/mpub.309338

Fink, A. (2014).  Conducting Research Literature Reviews: From the Internet to Paper  (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research.  Management Review Quarterly,  68, 103–106 (2018).  doi.org/10.1007/s11301-018-0142-x

Guba, B. (2008). Systematische Literaturrecherche.  Wiener Medizinische Wochenschrift , 158 (1-2), S. 62-69. doi:  doi.org/10.1007/s10354-007-0500-0  Hart, C.  Doing a literature review: releasing the social science research imagination.  London: Sage.

Jesson, J. K., Metheson, L. & Lacey, F. (2011).  Doing your Literature Review - traditional and Systematic Techniques . Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

Petticrew, M. and Roberts, H. (2006).  Systematic Reviews in the Social Sciences: A Practical Guide . Oxford:Blackwell. Ridley, D. (2012).  The literature review: A step-by-step guide . 2nd edn. London: Sage. 

Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis,  Journal of Marketing , American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.

Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review.  British Journal of Management , 14 (3), S. 207-222. doi:  https://doi.org/10.1111/1467-8551.00375

Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review.  Management Information Systems Quarterly , 26(2), xiii-xxiii.  http://www.jstor.org/stable/4132319

Durach, C. F., Wieland, A. & Machuca, Jose. A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistics Management, 45(1/2), 118 – 137.

What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.

On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.

Bos-Nehles, A., Renkema, M. & Janssen, M. (2017). HRM and innovative work behaviour: a systematic literature review. Personnel Review, 46(7), pp. 1228-1253

• Only very specific keywords used
• No precise information on how the review process was carried out (who reviewed articles?)
• Only journals with impact factor (publication bias)

Jia, F., Orzes, G., Sartor, M. & Nassimbeni, G. (2017). Global sourcing strategy and structure: towards a conceptual framework. International Journal of Operations & Production Management, 37(7), 840-864

• Research questions are explicitly presented
• Search string very detailed
• Exact description of the review process
• 2 persons conducted the review independently of each other

Franziska Klatt

[email protected]

+49 30 314-29778

Privacy notice: The TU Berlin offers a chat information service. If you enable it, your IP address and chat messages will be transmitted to external EU servers. more information

The chat is currently unavailable.

Please use our alternative contact options.

• A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

• Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
• View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
• Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
• First published as a Review in Advance on August 08, 2018
• Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

• APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
• Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
• Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
• Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
• Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
• Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
• Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
• Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
• Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
• Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
• Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
• Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
• Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
• Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
• Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
• Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
• Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
• Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
• Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
• Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
• Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
• Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
• Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
• John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
• Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
• Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
• Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
• Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
• Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
• Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
• Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
• Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
• Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
• Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
• Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
• Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
• Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
• Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
• Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
• Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
• Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
• Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
• Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
• Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
• Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
• Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
• Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
• Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
• Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
• Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
• Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

• UNC Libraries
• HSL Academic Process
• Systematic Reviews

Systematic Reviews: Home

Created by health science librarians.

• Systematic review resources

What is a Systematic Review?

A simplified process map, how can the library help, publications by hsl librarians, systematic reviews in non-health disciplines, resources for performing systematic reviews.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol
• Step 3: Conduct Literature Searches
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias , provide reliable findings , and inform decision-making.  ¹  

There are many types of literature reviews.

Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares a few different types of reviews to help you decide which is best for you. 

• Scoping Review Guide For more information about scoping reviews, refer to the UNC HSL Scoping Review Guide.

• UNC HSL's Simplified, Step-by-Step Process Map A PDF file of the HSL's Systematic Review Process Map.
• Text-Only: UNC HSL's Systematic Reviews - A Simplified, Step-by-Step Process A text-only PDF file of HSL's Systematic Review Process Map.

The average systematic review takes 1,168 hours to complete. ¹   A librarian can help you speed up the process.

Systematic reviews follow established guidelines and best practices to produce high-quality research. Librarian involvement in systematic reviews is based on two levels. In Tier 1, your research team can consult with the librarian as needed. The librarian will answer questions and give you recommendations for tools to use. In Tier 2, the librarian will be an active member of your research team and co-author on your review. Roles and expectations of librarians vary based on the level of involvement desired. Examples of these differences are outlined in the table below.

• Request a systematic or scoping review consultation

The following are systematic and scoping reviews co-authored by HSL librarians.

Only the most recent 15 results are listed. Click the website link at the bottom of the list to see all reviews co-authored by HSL librarians in PubMed

Researchers conduct systematic reviews in a variety of disciplines.  If your focus is on a topic outside of the health sciences, you may want to also consult the resources below to learn how systematic reviews may vary in your field.  You can also contact a librarian for your discipline with questions.

• EPPI-Centre methods for conducting systematic reviews The EPPI-Centre develops methods and tools for conducting systematic reviews, including reviews for education, public and social policy.

Environmental Topics

• Collaboration for Environmental Evidence (CEE) CEE seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice as a public service

Social Sciences

• Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan 4;70:747-770. doi: 10.1146/annurev-psych-010418-102803. A resource for psychology systematic reviews, which also covers qualitative meta-syntheses or meta-ethnographies
• The Campbell Collaboration

Social Work

Software engineering

• Guidelines for Performing Systematic Literature Reviews in Software Engineering The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students.

Sport, Exercise, & Nutrition

• Application of systematic review methodology to the field of nutrition by Tufts Evidence-based Practice Center Publication Date: 2009
• Systematic Reviews and Meta-Analysis — Open & Free (Open Learning Initiative) The course follows guidelines and standards developed by the Campbell Collaboration, based on empirical evidence about how to produce the most comprehensive and accurate reviews of research

• Systematic Reviews by David Gough, Sandy Oliver & James Thomas Publication Date: 2020

Updating reviews

• Updating systematic reviews by University of Ottawa Evidence-based Practice Center Publication Date: 2007
• Next: Step 1: Complete Pre-Review Tasks >>
• Last Updated: May 16, 2024 3:24 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills
• Systematic reviews

What are systematic reviews?

• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Describing and appraising studies
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

Systematic reviews are a type of literature review of research which require equivalent standards of rigour as primary research. They have a clear, logical rationale that is reported to the reader of the review. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews particularly in the following elements of conduct and reporting.

Systematic reviews: 

• use explicit and transparent methods
• are a piece of research following a standard set of stages
• are accountable, replicable and updateable
• involve users to ensure a review is relevant and useful.

For example, systematic reviews (like all research) should have a clear research question, and the perspective of the authors in their approach to addressing the question is described. There are clearly described methods on how each study in a review was identified, how that study was appraised for quality and relevance and how it is combined with other studies in order to address the review question. A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings.

Research protocols for systematic reviews may be peer-reviewed and published or registered in a suitable repository to help avoid duplication of reviews and for comparisons to be made with the final review and the planned review.

• History of systematic reviews to inform policy (EPPI-Centre)
• Six reasons why it is important to be systematic (EPPI-Centre)
• Evidence Synthesis International (ESI): Position Statement Describes the issues, principles and goals in synthesising research evidence to inform policy, practice and decisions

On this page

Should all literature reviews be 'systematic reviews', different methods for systematic reviews, reporting standards for systematic reviews.

Literature reviews provide a more complete picture of research knowledge than is possible from individual pieces of research. This can be used to: clarify what is known from research, provide new perspectives, build theory, test theory, identify research gaps or inform research agendas.

A systematic review requires a considerable amount of time and resources, and is one type of literature review.

If the purpose of a review is to make justifiable evidence claims, then it should be systematic, as a systematic review uses rigorous explicit methods. The methods used can depend on the purpose of the review, and the time and resources available.

A 'non-systematic review' might use some of the same methods as systematic reviews, such as systematic approaches to identify studies or quality appraise the literature. There may be times when this approach can be useful. In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of research, there may also be a need to obtain a quick and not necessarily thorough overview of a literature to inform some other work (including a systematic review). Another example, is where policymakers, or other people using research findings, want to make quick decisions and there is no systematic review available to help them. They have a choice of gaining a rapid overview of the research literature or not having any research evidence to help their decision-making. 

Just like any other piece of research, the methods used to undertake any literature review should be carefully planned to justify the conclusions made. 

Finding out about different types of systematic reviews and the methods used for systematic reviews, and reading both systematic and other types of review will help to understand some of the differences. 

Typically, a systematic review addresses a focussed, structured research question in order to inform understanding and decisions on an area. (see the  Formulating a research question  section for examples). 

Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review.  

Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over 2,000 papers, while others are much smaller. One purpose of a map is to help choose a sub-set of studies to explore more fully in a synthesis. There are also other purposes of maps: see the box on  systematic evidence maps  for further information. 

Reporting standards specify minimum elements that need to go into the reporting of a review. The reporting standards refer mainly to methodological issues but they are not as detailed or specific as critical appraisal for the methodological standards of conduct of a review.

A number of organisations have developed specific guidelines and standards for both the conducting and reporting on systematic reviews in different topic areas.  

• PRISMA PRISMA is a reporting standard and is an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The Key Documents section of the PRISMA website links to a checklist, flow diagram and explanatory notes. PRISMA is less useful for certain types of reviews, including those that are iterative.
• eMERGe eMERGe is a reporting standard that has been developed for meta-ethnographies, a qualitative synthesis method.
• ROSES: RepOrting standards for Systematic Evidence Syntheses Reporting standards, including forms and flow diagram, designed specifically for systematic reviews and maps in the field of conservation and environmental management.

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Systematic reviews: crd's guidance for undertaking reviews in health care.

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Meta-analysis and research synthesis.

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Literature reviews.

• What is a literature review?
• Why are literature reviews important?
• << Previous: Systematic reviews
• Next: Types of systematic reviews >>
• Last Updated: May 30, 2024 4:38 PM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability

• Research Process
• Manuscript Preparation
• Manuscript Review
• Publication Process
• Publication Recognition
• Language Editing Services
• Translation Services

Systematic Literature Review or Literature Review?

• 3 minute read
• 51.4K views

Table of Contents

As a researcher, you may be required to conduct a literature review. But what kind of review do you need to complete? Is it a systematic literature review or a standard literature review? In this article, we’ll outline the purpose of a systematic literature review, the difference between literature review and systematic review, and other important aspects of systematic literature reviews.

What is a Systematic Literature Review?

The purpose of systematic literature reviews is simple. Essentially, it is to provide a high-level of a particular research question. This question, in and of itself, is highly focused to match the review of the literature related to the topic at hand. For example, a focused question related to medical or clinical outcomes.

The components of a systematic literature review are quite different from the standard literature review research theses that most of us are used to (more on this below). And because of the specificity of the research question, typically a systematic literature review involves more than one primary author. There’s more work related to a systematic literature review, so it makes sense to divide the work among two or three (or even more) researchers.

Your systematic literature review will follow very clear and defined protocols that are decided on prior to any review. This involves extensive planning, and a deliberately designed search strategy that is in tune with the specific research question. Every aspect of a systematic literature review, including the research protocols, which databases are used, and dates of each search, must be transparent so that other researchers can be assured that the systematic literature review is comprehensive and focused.

Most systematic literature reviews originated in the world of medicine science. Now, they also include any evidence-based research questions. In addition to the focus and transparency of these types of reviews, additional aspects of a quality systematic literature review includes:

• Clear and concise review and summary
• Comprehensive coverage of the topic
• Accessibility and equality of the research reviewed

Systematic Review vs Literature Review

The difference between literature review and systematic review comes back to the initial research question. Whereas the systematic review is very specific and focused, the standard literature review is much more general. The components of a literature review, for example, are similar to any other research paper. That is, it includes an introduction, description of the methods used, a discussion and conclusion, as well as a reference list or bibliography.

A systematic review, however, includes entirely different components that reflect the specificity of its research question, and the requirement for transparency and inclusion. For instance, the systematic review will include:

• Eligibility criteria for included research
• A description of the systematic research search strategy
• An assessment of the validity of reviewed research
• Interpretations of the results of research included in the review

As you can see, contrary to the general overview or summary of a topic, the systematic literature review includes much more detail and work to compile than a standard literature review. Indeed, it can take years to conduct and write a systematic literature review. But the information that practitioners and other researchers can glean from a systematic literature review is, by its very nature, exceptionally valuable.

This is not to diminish the value of the standard literature review. The importance of literature reviews in research writing is discussed in this article . It’s just that the two types of research reviews answer different questions, and, therefore, have different purposes and roles in the world of research and evidence-based writing.

Systematic Literature Review vs Meta Analysis

It would be understandable to think that a systematic literature review is similar to a meta analysis. But, whereas a systematic review can include several research studies to answer a specific question, typically a meta analysis includes a comparison of different studies to suss out any inconsistencies or discrepancies. For more about this topic, check out Systematic Review VS Meta-Analysis article.

Language Editing Plus

With Elsevier’s Language Editing Plus services , you can relax with our complete language review of your systematic literature review or literature review, or any other type of manuscript or scientific presentation. Our editors are PhD or PhD candidates, who are native-English speakers. Language Editing Plus includes checking the logic and flow of your manuscript, reference checks, formatting in accordance to your chosen journal and even a custom cover letter. Our most comprehensive editing package, Language Editing Plus also includes any English-editing needs for up to 180 days.

How to Make a PowerPoint Presentation of Your Research Paper

What is and How to Write a Good Hypothesis in Research?

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

Choosing the Right Research Methodology: A Guide for Researchers

Why is data validation important in research?

Writing a good review article

Input your search keywords and press Enter.

University Libraries      University of Nevada, Reno

• Skill Guides
• Subject Guides

Systematic, Scoping, and Other Literature Reviews: Overview

• Project Planning

What Is a Systematic Review?

Regular literature reviews are simply summaries of the literature on a particular topic. A systematic review, however, is a comprehensive literature review conducted to answer a specific research question. Authors of a systematic review aim to find, code, appraise, and synthesize all of the previous research on their question in an unbiased and well-documented manner. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) outline the minimum amount of information that needs to be reported at the conclusion of a systematic review project. 

Other types of what are known as "evidence syntheses," such as scoping, rapid, and integrative reviews, have varying methodologies. While systematic reviews originated with and continue to be a popular publication type in medicine and other health sciences fields, more and more researchers in other disciplines are choosing to conduct evidence syntheses. 

This guide will walk you through the major steps of a systematic review and point you to key resources including Covidence, a systematic review project management tool. For help with systematic reviews and other major literature review projects, please send us an email at  [email protected] .

Getting Help with Reviews

Organization such as the Institute of Medicine recommend that you consult a librarian when conducting a systematic review. Librarians at the University of Nevada, Reno can help you:

• Understand best practices for conducting systematic reviews and other evidence syntheses in your discipline
• Choose and formulate a research question
• Decide which review type (e.g., systematic, scoping, rapid, etc.) is the best fit for your project
• Determine what to include and where to register a systematic review protocol
• Select search terms and develop a search strategy
• Identify databases and platforms to search
• Find the full text of articles and other sources
• Become familiar with free citation management (e.g., EndNote, Zotero)
• Get access to you and help using Covidence, a systematic review project management tool

Doing a Systematic Review

• Plan - This is the project planning stage. You and your team will need to develop a good research question, determine the type of review you will conduct (systematic, scoping, rapid, etc.), and establish the inclusion and exclusion criteria (e.g., you're only going to look at studies that use a certain methodology). All of this information needs to be included in your protocol. You'll also need to ensure that the project is viable - has someone already done a systematic review on this topic? Do some searches and check the various protocol registries to find out. 
• Identify - Next, a comprehensive search of the literature is undertaken to ensure all studies that meet the predetermined criteria are identified. Each research question is different, so the number and types of databases you'll search - as well as other online publication venues - will vary. Some standards and guidelines specify that certain databases (e.g., MEDLINE, EMBASE) should be searched regardless. Your subject librarian can help you select appropriate databases to search and develop search strings for each of those databases.  
• Evaluate - In this step, retrieved articles are screened and sorted using the predetermined inclusion and exclusion criteria. The risk of bias for each included study is also assessed around this time. It's best if you import search results into a citation management tool (see below) to clean up the citations and remove any duplicates. You can then use a tool like Rayyan (see below) to screen the results. You should begin by screening titles and abstracts only, and then you'll examine the full text of any remaining articles. Each study should be reviewed by a minimum of two people on the project team. 
• Collect - Each included study is coded and the quantitative or qualitative data contained in these studies is then synthesized. You'll have to either find or develop a coding strategy or form that meets your needs. 
• Explain - The synthesized results are articulated and contextualized. What do the results mean? How have they answered your research question?
• Summarize - The final report provides a complete description of the methods and results in a clear, transparent fashion. 

Adapted from

Types of reviews, systematic review.

These types of studies employ a systematic method to analyze and synthesize the results of numerous studies. "Systematic" in this case means following a strict set of steps - as outlined by entities like PRISMA and the Institute of Medicine - so as to make the review more reproducible and less biased. Consistent, thorough documentation is also key. Reviews of this type are not meant to be conducted by an individual but rather a (small) team of researchers. Systematic reviews are widely used in the health sciences, often to find a generalized conclusion from multiple evidence-based studies. 

Meta-Analysis

A systematic method that uses statistics to analyze the data from numerous studies. The researchers combine the data from studies with similar data types and analyze them as a single, expanded dataset. Meta-analyses are a type of systematic review.

Scoping Review

A scoping review employs the systematic review methodology to explore a broader topic or question rather than a specific and answerable one, as is generally the case with a systematic review. Authors of these types of reviews seek to collect and categorize the existing literature so as to identify any gaps.

Rapid Review

Rapid reviews are systematic reviews conducted under a time constraint. Researchers make use of workarounds to complete the review quickly (e.g., only looking at English-language publications), which can lead to a less thorough and more biased review. 

Narrative Review

A traditional literature review that summarizes and synthesizes the findings of numerous original research articles. The purpose and scope of narrative literature reviews vary widely and do not follow a set protocol. Most literature reviews are narrative reviews. 

Umbrella Review

Umbrella reviews are, essentially, systematic reviews of systematic reviews. These compile evidence from multiple review studies into one usable document. 

Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal , vol. 26, no. 2, 2009, pp. 91-108. doi: 10.1111/j.1471-1842.2009.00848.x .

• Next: Project Planning >>

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

To read this content please select one of the options below:

Please note you do not have access to teaching notes, a contemporary systematic literature review of equestrian tourism: emerging advancements and future insights.

Journal of Hospitality and Tourism Insights

ISSN : 2514-9792

Article publication date: 2 July 2024

Horse-based tourism stands at the intersection of cultural heritage, leisure activities, and eco-friendly travel, captivating enthusiasts and researchers alike with its diverse facets and impacts. This study examines the horse-based tourism literature to provide an overview of horse-based tourism publications.

Design/methodology/approach

Using a systematic literature review (SLR) method, pertinent journal articles published over the past 3 decades were retrieved and analyzed. Based on the review process, 44 papers were identified and analyzed by publication year, journal distribution, research method, and lead author. Using Leximancer software, a thematic analysis was undertaken to determine the major themes of horse-based tourism.

The findings revealed a rising trend of horse-based tourism articles and the appearance of an increasing number of studies in tourism-oriented journals. In addition, it was discovered that the majority of available studies are qualitative, whereas quantitative research is few and limited.

Research limitations/implications

Our research establishes a foundational resource for future studies and scholarly discourse on the multifaceted contributions of horse-based tourism.

Practical implications

This study can assist decision-makers in understanding the potential of horse-based tourism in the sustainable development of destinations. Moreover, it provides clear direction on implementing appropriate strategies to manage horse-based tourism.

Originality/value

This study distinguishes itself as the inaugural comprehensive literature review encompassing the breadth of horse-based tourism publications and research domains. By pioneering this endeavor, we not only contribute a unique perspective to the existing body of knowledge in the field but also emphasize the vital role of horse-based tourism in fostering economic and social sustainability for the countries involved.

• Horse-based tourism
• Equestrian tourism
• Systematic literature review
• Research domains
• Thematic analysis

Rezapouraghdam, H. , Saydam, M.B. , Altun, O. , Roudi, S. and Nosrati, S. (2024), "A contemporary systematic literature review of equestrian tourism: emerging advancements and future insights", Journal of Hospitality and Tourism Insights , Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JHTI-01-2024-0046

Emerald Publishing Limited

Copyright © 2024, Emerald Publishing Limited

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Now Published: Systematic literature review on religious leader well-being, burnout, and trauma

Our second publication from the Helping the Helpers project is a systematic literature review of 82 empirical studies that look at burnout, trauma impacts, and/or well-being among religious leaders. We were able to highlight relational, systemic/organizational, and diversity issues that are crucial for gaining a more holistic understanding of these issues. The citation and abstract are below.

Hydinger, K. R., Wu, X., Captari, L. E., & Sandage, S. (2024). Burnout, Trauma Impacts, and Well-Being Among Clergy and Chaplains: A Systematic Review and Recommendations to Guide Best Practice. Pastoral Psychology . Advanced online publication. https://doi.org/10.1007/s11089-024-01150-x

Religious leaders (i.e., clergy and chaplains) face unique, ongoing stressors that can increase risks for psychosocial and vocational vulnerabilities. Emerging evidence indicates concerning prevalence rates of distress and attrition among these professionals, particularly since the COVID-19 pandemic. To date, most empirical work has focused on compromised functioning among religious leaders. Utilizing a more holistic approach, this systematic review explores individual, relational, and organizational factors associated with diverse outcomes. Following the PRISMA methodology, we identified 82 empirical articles investigating (a) risk and protective factors related to burnout, trauma impacts, spiritual distress, and other occupational hazards and/or (b) factors associated with well-being and flourishing, over and above distress reduction. We summarize the state of the available evidence, distinguishing between  risk increasers ,  protective factors , and  well-being enhancers . Attention is given to three domains:  individual  (e.g., demographics, personality factors, virtue development, coping and formation practices),  relational  (e.g., peer, family, and collegial supports; navigation of conflicts and polarized issues in one’s community of care), and  institutional  (e.g., role ambiguity or clarity, resource availability, systemic expectations and demands). We identify notable gaps to be addressed in future research; for example, most studies are cross-sectional, lack diversity in religion, gender, and geography, and operationalize well-being as the absence of symptoms rather than the presence of positive states and functioning. Considering the available evidence, we present best practices to guide psychological practitioners, denominational bodies, and others involved in religious leaders’ formation.

Where we stand on chronic wasting disease: A systematic literature review of its prevalence patterns, impacts, and management interventions

• Bhattarai, Sushma
• Grala, Robert K.
• Poudyal, Neelam C.
• Tanger, Shaun M.
• Adhikari, Ram K.

With high fatality and no cure, chronic wasting disease (CWD) has infected cervids in multiple regions, including the United States, Canada, Europe, and South Korea. Despite the rapid growth of literature on CWD, the full scope of its ecological, social, and economic impacts and the most effective and socially acceptable management strategies to mitigate the disease is unclear. Of 3008 initially identified published peer-reviewed papers, 134 were included in a final systematic literature review to synthesize the current knowledge on CWD transmission patterns, impacts, and the effectiveness of management interventions. The number of publications on CWD has increased steadily since 2000 with an average of six papers per year. Most papers were related to CWD prevalence (39 %), human behavior (33 %), CWD impacts (31 %), and management interventions (16 %). Environmental factors such as soil, water, and plants were identified as the most common transmission medium, with a higher prevalence rate among adult male cervids than females. Hunters showed a higher risk perception and were more likely to change hunting behavior due to CWD detection than non-hunters. Ecological impacts included the decreased survival rate accompanied by lower population growth, eventually leading to the decline of cervid populations. Culling was found to be an effective and widely implemented management strategy across countries, although it often was associated with public resistance. Despite potentially high negative economic impacts anticipated due to CWD, studies on this subject were limited. Sustained surveillance, ongoing research, and engagement of affected stakeholders will be essential for future disease control and management.

• Human behavior;
• Ecological impacts;
• Economic impacts;
• Systematic literature review

Help | Advanced Search

Computer Science > Software Engineering

Title: systematic literature review on application of learning-based approaches in continuous integration.

Abstract: Context: Machine learning (ML) and deep learning (DL) analyze raw data to extract valuable insights in specific phases. The rise of continuous practices in software projects emphasizes automating Continuous Integration (CI) with these learning-based methods, while the growing adoption of such approaches underscores the need for systematizing knowledge. Objective: Our objective is to comprehensively review and analyze existing literature concerning learning-based methods within the CI domain. We endeavour to identify and analyse various techniques documented in the literature, emphasizing the fundamental attributes of training phases within learning-based solutions in the context of CI. Method: We conducted a Systematic Literature Review (SLR) involving 52 primary studies. Through statistical and thematic analyses, we explored the correlations between CI tasks and the training phases of learning-based methodologies across the selected studies, encompassing a spectrum from data engineering techniques to evaluation metrics. Results: This paper presents an analysis of the automation of CI tasks utilizing learning-based methods. We identify and analyze nine types of data sources, four steps in data preparation, four feature types, nine subsets of data features, five approaches for hyperparameter selection and tuning, and fifteen evaluation metrics. Furthermore, we discuss the latest techniques employed, existing gaps in CI task automation, and the characteristics of the utilized learning-based techniques. Conclusion: This study provides a comprehensive overview of learning-based methods in CI, offering valuable insights for researchers and practitioners developing CI task automation. It also highlights the need for further research to advance these methods in CI.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Search Menu
• Sign in through your institution
• Advance Articles
• Editor's Choice
• Collections
• Supplements
• InSight Papers
• BSR Registers Papers
• Virtual Roundtables
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• About Rheumatology
• About the British Society for Rheumatology
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Dispatch Dates
• Terms and Conditions
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, supplementary material, data availability, acknowledgements.

• < Previous

Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis

• Article contents
• Figures & tables
• Supplementary Data

Philip J Mease, Dafna D Gladman, Joseph F Merola, Peter Nash, Stacy Grieve, Victor Laliman-Khara, Damon Willems, Vanessa Taieb, Adam R Prickett, Laura C Coates, Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis, Rheumatology , Volume 63, Issue 7, July 2024, Pages 1779–1789, https://doi.org/10.1093/rheumatology/kead705

• Permissions Icon Permissions

To understand the relative efficacy and safety of bimekizumab, a selective inhibitor of IL-17F in addition to IL-17A, vs other biologic and targeted synthetic DMARDs (b/tsDMARDs) for PsA using network meta-analysis (NMA).

A systematic literature review (most recent update conducted on 1 January 2023) identified randomized controlled trials (RCTs) of b/tsDMARDs in PsA. Bayesian NMAs were conducted for efficacy outcomes at Weeks 12–24 for b/tsDMARD-naïve and TNF inhibitor (TNFi)-experienced patients. Safety at Weeks 12–24 was analysed in a mixed population. Odds ratios (ORs) and differences of mean change with the associated 95% credible interval (CrI) were calculated for the best-fitting models, and the surface under the cumulative ranking curve (SUCRA) values were calculated to determine relative rank.

The NMA included 41 RCTs for 22 b/tsDMARDs. For minimal disease activity (MDA), bimekizumab ranked 1st in b/tsDMARD-naïve patients and 2nd in TNFi-experienced patients. In b/tsDMARD-naïve patients, bimekizumab ranked 6th, 5th and 3rd for ACR response ACR20/50/70, respectively. In TNFi-experienced patients, bimekizumab ranked 1st, 2nd and 1st for ACR20/50/70, respectively. For Psoriasis Area and Severity Index 90/100, bimekizumab ranked 2nd and 1st in b/tsDMARD-naïve patients, respectively, and 1st and 2nd in TNFi-experienced patients, respectively. Bimekizumab was comparable to b/tsDMARDs for serious adverse events.

Bimekizumab ranked favourably among b/tsDMARDs for efficacy on joint, skin and MDA outcomes, and showed comparable safety, suggesting it may be a beneficial treatment option for patients with PsA.

For joint efficacy, bimekizumab ranked highly among approved biologic/targeted synthetic DMARDs (b/tsDMARDs).

Bimekizumab provides better skin efficacy (Psoriasis Area and Severity Index, PASI100 and PASI90) than many other available treatments in PsA.

For minimal disease activity, bimekizumab ranked highest of all available b/tsDMARDs in b/tsDMARD-naïve and TNF inhibitor–experienced patients.

PsA is a chronic, systemic, inflammatory disease in which patients experience a high burden of illness [ 1–3 ]. PsA has multiple articular and extra-articular disease manifestations including peripheral arthritis, axial disease, enthesitis, dactylitis, skin psoriasis (PSO) and psoriatic nail disease [ 4 , 5 ]. Patients with PsA can also suffer from related inflammatory conditions, uveitis and IBD [ 4 , 5 ]. Approximately one fifth of all PSO patients, increasing to one quarter of patients with moderate to severe PSO, will develop PsA over time [ 6 , 7 ].

The goal of treatment is to control inflammation and prevent structural damage to minimize disease burden, normalize function and social participation, and maximize the quality of life of patients [ 1 , 4 ]. As PsA is a heterogeneous disease, the choice of treatment is guided by individual patient characteristics, efficacy against the broad spectrum of skin and joint symptoms, and varying contraindications to treatments [ 1 , 4 ]. There are a number of current treatments classed as conventional DMARDs such as MTX, SSZ, LEF; biologic (b) DMARDs such as TNF inhibitors (TNFi), IL inhibitors and cytotoxic T lymphocyte antigen 4 (CTLA4)-immunoglobulin; and targeted synthetic (ts) DMARDs which include phosphodiesterase-4 (PDE4) and Janus kinase (JAK) inhibitors [ 1 , 8 ].

Despite the number of available treatment options, the majority of patients with PsA report that they do not achieve remission and additional therapeutic options are needed [ 9 , 10 ]. Thus, the treatment landscape for PsA continues to evolve and treatment decisions increase in complexity, especially as direct comparative data are limited [ 2 ].

Bimekizumab is a monoclonal IgG1 antibody that selectively inhibits IL-17F in addition to IL-17A, which is approved for the treatment of adults with active PsA in Europe [ 11 , 12 ]. Both IL-17A and IL-17F are pro-inflammatory cytokines implicated in PsA [ 11 , 13 ]. IL-17F is structurally similar to IL-17A and expressed by the same immune cells; however, the mechanisms that regulate expression and kinetics differ [ 13 , 14 ]. IL-17A and IL-17F are expressed as homodimers and as IL-17A–IL-17F heterodimers that bind to and signal via the same IL-17 receptor A/C complex [ 13 , 15 ].

In vitro studies have demonstrated that the dual inhibition of both IL-17A and IL-17F with bimekizumab was more effective at suppressing PsA inflammatory genes and T cell and neutrophil migration, and periosteal new bone formation, than blocking IL-17A alone [ 11 , 14 , 16 , 17 ]. Furthermore, IL-17A and IL-17F protein levels are elevated in psoriatic lesions and the superiority of bimekizumab 320 mg every 4 weeks (Q4W) or every 8 weeks (Q8W) over the IL-17A inhibitor, secukinumab, in complete clearance of psoriatic skin was demonstrated in a head-to-head trial in PSO [ 16 , 18 ]. Collectively, this evidence suggests that neutralizing both IL-17F and IL-17A may provide more potent abrogation of IL-17-mediated inflammation than IL-17A alone.

Bimekizumab 160 mg Q4W demonstrated significant improvements in efficacy outcomes compared with placebo, and an acceptable safety profile in adults with PsA in the phase 3 RCTs BE OPTIMAL (NCT03895203) (b/tsDMARD-naïve patients) and BE COMPLETE (NCT03896581) (TNFi inadequate responders) [ 19 , 20 ].

The objective of this study was to establish the comparative efficacy and safety of bimekizumab 160 mg Q4W vs other available PsA treatments, using network meta-analysis (NMA).

Search strategy

A systematic literature review (SLR) was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines [ 21 ] and adhered to the principles outlined in the Cochrane Handbook for Systematic Reviews of Interventions, Centre for Reviews and Dissemination’s Guidance for Undertaking Reviews in Healthcare, and Methods for the Development of National Institute of Health and Care Excellence (NICE) Public Health Guidance [ 22–24 ]. The SLR of English-language publications was originally conducted on 3 December 2015, with updates on 7 January 2020, 2 May 2022 and 1 January 2023 in Medical Literature Analysis and Retrieval System Online (MEDLINE ® ), Excerpta Medica Database (Embase ® ) and the Cochrane Central Register of Controlled Trials (CENTRAL) for literature published from January 1991 onward using the Ovid platform. Additionally, bibliographies of SLRs and meta-analyses identified through database searches were reviewed to ensure any publications not identified in the initial search were included in this SLR. Key clinical conference proceedings not indexed in Ovid (from October 2019 to current) and ClinicalTrials.gov were also manually searched. The search strategy is presented in Supplementary Table S1 (available at Rheumatology online).

Study inclusion

Identified records were screened independently and in duplicate by two reviewers and any discrepancies were reconciled via discussion or a third reviewer. The SLR inclusion criteria were defined by the Patient populations, Interventions, Comparators, Outcome measures, and Study designs (PICOS) Statement ( Supplementary Table S2 , available at Rheumatology online). The SLR included published studies assessing approved therapies for the treatment of PsA. Collected data included study and patient population characteristics, interventions, comparators, and reported clinical and patient-reported outcomes relevant to PsA. For efficacy outcomes, pre-crossover data were extracted in studies where crossover occurred. All publications included in the analysis were evaluated according to the Cochrane risk-of-bias tool for randomized trials as described in the Cochrane Handbook [ 25 ].

Network meta-analysis methods

NMA is the quantitative assessment of relative treatment effects and associated uncertainty of two or more interventions [ 26 , 27 ]. It is used frequently in health technology assessment, guideline development and to inform treatment decision making in clinical practice [ 26 ].

Bimekizumab 160 mg Q4W was compared with current b/tsDMARDs at regulatory-approved doses ( Table 1 ) by NMA. All comparators were selected on the basis they were relevant to clinical practice, i.e. recommended by key clinical guidelines, licensed by key regulatory bodies and/or routinely used.

NMA intervention and comparators

See Supplementary Table S4 , available at Rheumatology online for additional dosing schedules used in included studies. BID: twice daily; CTLA4-Ig: cytotoxic T lymphocyte antigen 4-immunoglobulin; IL-17A/17Fi: IL-17A/17F inhibitor; IL-17Ai: IL-17A inhibitor; IL-12/23i: IL-12/23 inhibitor; IL-23i: IL-23 inhibitor; JAKi: Janus kinase inhibitor; NMA: network meta-analysis; PDE-4i: phosphodiesterase-4 inhibitor; Q1W: once weekly; Q2W: every 2 weeks; Q4W: every 4 weeks; Q8W: every 8 weeks; Q12W: every 12 weeks; QD: once daily; TNFi: TNF inhibitor.

Two sets of primary analyses were conducted, one for a b/tsDMARD-naïve PsA population and one for a TNFi-experienced PsA population. Prior treatment with TNFis has been shown to impact the response to subsequent bDMARD treatments [ 28 ]. In addition, most trials involving b/tsDMARDs for the treatment of PsA (including bimekizumab) report separate data on both b/tsDMARD-naïve and TNFi-experienced subgroups, making NMA in each of these patient populations feasible.

For each population the following outcomes were analysed: American College of Rheumatology response (ACR20/50/70), Psoriasis Area and Severity Index (PASI90/100), and minimal disease activity (MDA). The analysis of serious adverse events (SAE) was conducted using a mixed population (i.e. b/tsDMARD-naïve, TNFi-experienced and mixed population data all were included) as patients’ previous TNFI exposure was not anticipated to impact safety outcomes following discussions with clinicians. The NMA included studies for which data were available at week 16, if 16-week data were not available (or earlier crossover occurred), data available at weeks 12, 14 or 24 were included. Pre-crossover data were included in the analyses for efficacy outcomes to avoid intercurrent events.

Heterogeneity between studies for age, sex, ethnicity, mean time since diagnosis, concomitant MTX, NSAIDs or steroid use was assessed using Grubb’s test, also called the extreme Studentized deviate method, to identify outlier studies.

All univariate analyses involved a 10 000 run-in iteration phase and a 10 000-iteration phase for parameter estimation. All calculations were performed using the R2JAGS package to run Just Another Gibbs Sampler (JAGS) 3.2.3 and the code reported in NICE Decision Support Unit (DSU) Technical Support Document Series [ 29–33 ]. Convergence was confirmed through inspection of the ratios of Monte-Carlo error to the standard deviations of the posteriors; values >5% are strong signs of convergence issues [ 31 ]. In some cases, trials reported outcome results of zero (ACR70, PASI100, SAE) in one or more arms for which a continuity correction was applied to mitigate the issue, as without the correction most models were not convergent or provided a large posterior distribution making little clinical sense [ 31 ].

Four NMA models [fixed effects (FE) unadjusted, FE baseline risk-adjusted, random effects (RE) unadjusted and RE baseline risk-adjusted] were assessed and the best-fit models were chosen using methods described in NICE DSU Technical Support Document 2 [ 31 ]. Odds ratios (ORs) and differences of mean change (MC) with the associated 95% credible intervals (CrIs) were calculated for each treatment comparison in the evidence network for the best fitting models and presented in league tables and forest plots. In addition, the probability of bimekizumab 160 mg Q4W being better than other treatments was calculated using surface under the cumulative ranking curve (SUCRA) to determine relative rank. Conclusions (i.e. better/worse or comparable) for bimekizumab 160 mg Q4W vs comparators were based on whether the pairwise 95% CrIs of the ORs/difference of MC include 1 (dichotomous outcomes), 0 (continuous outcomes) or not. In the case where the 95% CrI included 1 or 0, then bimekizumab 160 mg Q4W and the comparator were considered comparable. If the 95% CrI did not include 1 or 0, then bimekizumab 160 mg Q4W was considered either better or worse depending on the direction of the effect.

Compliance with ethics guidelines

This article is based on previously conducted studies and does not contain any new studies with human participants or animals performed by any of the authors.

Study and patient characteristics

The SLR identified 4576 records through databases and 214 records through grey literature, of which 3143 were included for abstract review. Following the exclusion of a further 1609 records, a total of 1534 records were selected for full-text review. A total of 66 primary studies from 246 records were selected for data extraction. No trial was identified as having a moderate or high risk of bias ( Supplementary Table S3 , available at Rheumatology online).

Of the 66 studies identified in the SLR, 41 studies reported outcomes at weeks 12, 16 or 24 and met the criteria for inclusion in the NMA in either a b/tsDMARD-naïve population ( n  = 20), a TNFi-experienced population ( n  = 5), a mixed population with subgroups ( n  = 13) or a mixed PsA population without subgroups reported ( n  = 3). The PRISMA diagram is presented in Fig. 1 . Included and excluded studies are presented in Supplementary Tables S4 and S5 , respectively (available at Rheumatology online).

PRISMA flow diagram. The PRISMA flow diagram for the SLR conducted to identify published studies assessing approved treatments for the treatment of PsA. cDMARD: conventional DMARD; NMA: network meta-analysis; NR: not reported; PD: pharmacodynamic; PK: pharmacokinetic; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT: randomized controlled trial; SLR: systematic literature review

The baseline study and patient characteristics (where reported) are presented in Supplementary Table S6 (available at Rheumatology online). There were 20–483 patients included in treatment arms. The median age of patients was 48.9 years, the median percentage of males was 50.3% and a median of 92.3% of patients were Caucasian. Patients had a mean time since diagnosis of 7.6 years and a mean PASI score of 8.7. The mean (range) use of concomitant MTX, NSAIDs and steroids were 53.9% (29.1% to 84.0%), 72.4% (33.3% to 100.0%) and 16.8% (9.2% to 30.0%), respectively. Heterogeneity was generally low across studies except for the concomitant use of MTX, NSAIDs and steroids. Using an approach consistent with established NMA methods in PsA [ 34–36 ], a meta-regression model using JAGS code reported in NICE DSU Technical Support Document 3 [ 33 ] was used to account for variation in placebo responses when model-fit statistics suggested that baseline risk-adjusted models provided a better fit to the data.

NMA results

The network diagrams for ACR50 in b/tsDMARD-naïve and TNFi-experienced patients are presented in Fig. 2A and B with network diagrams for other outcomes presented in Supplementary Fig. S1 (available at Rheumatology online). The networks for ACR response were larger, in terms of both number of studies and patients included, than the networks for PASI. Similarly, the networks for b/tsDMARD-naïve patients were larger than TNFi-experienced patients across all outcomes analysed. Placebo was used as a common comparator in all networks and there were a few studies that included more than two arms (OPAL-Broaden, Select-PsA-1, SPIRIT-P1 and BE OPTIMAL) that included adalimumab as the reference arm in b/tsDMARD-naïve patients. Lastly, networks included studies where the primary outcome was evaluated at time points longer than 16 weeks (e.g. EXCEED study at 52 weeks) but as per the methods, 16-week data formed the network.

Network of evidence for ACR50. ( A ) b/tsDMARD-naïve patients. ( B ) TNFi-experienced patients. The size of the circle representing each intervention is proportional to the number of patients included in the analysis. The line width is proportional to the number of studies connecting the interventions. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CZP: certolizumab pegol; ETA: etanercept; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RIS: risankizumab; SEC: secukinumab; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

The best-fit model is noted for each outcome with full model fit statistics for all outcomes presented in Supplementary Table S7 (available at Rheumatology online). Forest plots for ACR50 and PASI100 are presented in Figs 3 and 4 , with forest plots for other outcomes, along with the league tables in Supplementary Fig. S2 and Table S8 , respectively (available at Rheumatology online).

ACR50. The results for the NMA on ACR50 at week 16. ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 469.59. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 205.33. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg Q4W is considered either better or worse depending on the direction of the effect. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CrI: credible interval; CZP: certolizumab pegol; DIC: deviance information criterion; ETA: etanercept; FE: fixed effects; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; NMA: network meta-analysis; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RE: random effects; RIS: risankizumab; SEC: secukinumab; SUCRA: surface under the cumulative ranking curve; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

PASI100. The results for the NMA on PASI100 at week 16: ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 150.27. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 81.76. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg 4W is considered better. ADA: adalimumab; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ, bimekizumab; CrI, credible interval; CZP, certolizumab pegol; DIC, deviance information criterion; FE, fixed effects; GOL, golimumab; GUS, guselkumab; IXE, ixekizumab; NMA, network meta-analysis; PASI, Psoriasis Area and Severity Index; PBO, placebo; Q4W, every 4 weeks; Q8W, every 8 weeks; RE, random effects; SEC, secukinumab; SUCRA, surface under the cumulative ranking curve; TNFi-experienced, TNF inhibitor–experienced; UPA, upadacitinib

Joint outcomes

For ACR50 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

b/tsDMARD-naïve patients

Bimekizumab 160 mg Q4W ranked 6th for ACR20 (SUCRA = 0.75), 5th for ACR50 (SUCRA = 0.74) ( Fig. 3A ) and 3rd for ACR70 (SUCRA = 0.80) among 21 treatments. For ACR50, bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, guselkumab 100 mg Q4W, ustekinumab 45 mg, risankizumab 150 mg, guselkumab 100 mg Q8W and ustekinumab 90 mg; worse than golimumab 2 mg i.v.; and comparable to the remaining treatments in the network ( Fig. 3A ).

TNFi-experienced patients

Bimekizumab 160 mg Q4W ranked 1st among 16 treatments for ACR20 (SUCRA = 0.96), 2nd among 15 treatments for ACR50 (SUCRA = 0.84) ( Fig. 3B ) and 1st among 16 treatments for ACR70 (SUCRA = 0.83). Bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, secukinumab 150 mg without loading dose, tofacitinib 5 mg and secukinumab 150 mg; and comparable to the remaining treatments in the network on ACR50 ( Fig. 3B ).

Skin outcomes

For PASI100 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 2nd among 15 treatments (SUCRA = 0.89) for PASI90 and 1st among 11 treatments (SUCRA = 0.95) for PASI100 ( Fig. 4A ). Bimekizumab 160 mg Q4W was better than placebo, certolizumab pegol pooled, golimumab 2 mg i.v., secukinumab 150 mg, adalimumab 40 mg, upadacitinib 15 mg, secukinumab 300 mg and ixekizumab 80 mg Q4W; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4A ).

Bimekizumab 160 mg Q4W ranked 1st among 10 treatments (SUCRA = 0.85) for PASI90 and 2nd among 7 treatments (SUCRA = 0.79) for PASI100 ( Fig. 4B ). Bimekizumab 160 mg Q4W was better than placebo, ixekizumab 80 mg Q4W and upadacitinib 15 mg; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4B ).

For MDA, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 1st among 13 treatments (SUCRA = 0.91) and was better than placebo [OR (95% CrI) 6.31 (4.61–8.20)], guselkumab 100 mg Q4W [2.06 (1.29–3.10)], guselkumab 100 mg Q8W [1.76 (1.09–2.69)], risankizumab 150 mg [1.99 (1.40–2.76)] and adalimumab 40 mg [1.41 (1.01–1.93)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2G , available at Rheumatology online).

Bimekizumab 160 mg Q4W ranked 1st among 11 treatments (SUCRA = 0.83) and was better than placebo [12.10 (5.31–28.19)] and tofacitinib 5 mg [6.81 (2.14–21.35)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2H , available at Rheumatology online).

The network for SAEs for a mixed population included 23 treatments and the best-fit model was an RE-unadjusted model (due to study populations and time point reporting heterogeneity). Bimekizumab 160 mg Q4W showed comparable safety to all treatments in the network ( Supplementary Fig. S2I , available at Rheumatology online).

The treatment landscape for PsA is complex, with numerous treatment options and limited direct comparative evidence. Bimekizumab 160 mg Q4W has recently been approved for the treatment of active PsA by the European Medicines Agency and recommended by NICE in the UK, and the published phase 3 results warrant comparison with existing therapies by NMA.

This NMA included 41 studies evaluating 22 b/tsDMARDs including the novel IL-17F and IL-17A inhibitor, bimekizumab. Overall, bimekizumab 160 mg Q4W ranked favourably among b/tsDMARDS for efficacy in joint, skin and disease activity outcomes in PsA across both b/tsDMARD-naïve and TNFi-experienced populations. The safety of bimekizumab 160 mg Q4W was similar to the other b/tsDMARDs.

The Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA) and EULAR provide evidence-based recommendations for the treatment of PsA [ 1 , 2 ]. To treat peripheral arthritis symptoms in PsA, efficacy across the classes of current b/tsDMARDs are considered similar by both GRAPPA and EULAR, in part due to a lack of data comparing licensed therapies in a head-to-head trial setting [ 1 , 2 ]. EULAR recommends the use of JAK inhibitors in the case of inadequate response, intolerance or when a bDMARD is not appropriate [ 1 ]. This recommendation was made when tofacitinib was the only available JAK inhibitor, but reflects current marketing authorizations for tofacitinib and upadacitinib which indicate use in patients with an inadequate response or prior intolerance to TNFis (USA) or bDMARDs (Europe) [ 37–40 ]. This NMA suggests that bimekizumab 160 mg Q4W may have an advantage over current treatments, including IL-23 inhibitors in b/tsDMARD naïve patients, and secukinumab 150 mg and tofacitinib in TNFi-experienced patients, as evidenced by our analysis of ACR50 for which the pairwise comparisons were significantly in favour of bimekizumab 160 mg Q4W.

For the treatment of skin symptoms in PsA, IL-23, IL-12/23 and IL-17A inhibitors are currently recommended due to their greater efficacy compared with TNFis [ 1 , 4 ]. GRAPPA also suggests considering efficacy demonstrated in direct comparative studies in PSO when selecting a treatment for PsA skin symptoms [ 2 ]. In our analysis of complete skin clearance as measured by PASI100, bimekizumab 160 mg Q4W demonstrated the likelihood of significantly greater efficacy than IL-17A, JAK inhibitors and TNFis in b/tsDMARD-naïve patients and IL-17A and JAK inhibitors in TNFi-experienced patients. Furthermore, the NMA results for skin clearance in PsA are in alignment with previous studies in PSO that demonstrated superiority of bimekizumab 320 mg Q4W or Q8W vs secukinumab, ustekinumab and adalimumab ( P  < 0.001) (note that the dosing of bimekizumab in PSO differs from that in PsA) [ 12 , 18 , 41 , 42 ].

There are similarities between our results and other recently published NMAs of b/tsDMARDs in PsA, although methodological heterogeneity across all NMAs makes comparisons challenging [ 34–36 , 43–45 ]. Among recent NMAs, the largest evaluated 21 treatments [ 34 ] and only four considered subgroups of b/tsDMARD-naïve and TNFi-experienced patients or those with inadequate response [ 35 , 36 , 43 , 45 ]. Furthermore, different or pooled levels of response were evaluated for ACR and PASI outcomes.

Previous NMAs also support IL-17, IL-12/23 and IL-23 inhibitors having greater efficacy for skin symptoms than TNFis [ 35 , 36 ]. In an overall PsA population, McInnes et al. demonstrated that secukinumab 300 mg, ixekizumab 80 mg Q4W, and ustekinumab 45 mg and 90 mg were likely more efficacious than TNFis for PASI90 [ 35 ]. In another NMA by Ruyssen-Witrand et al. , results suggested that ixekizumab 80 mg Q4W had significantly greater efficacy than adalimumab, certolizumab pegol pooled, and etanercept 25 mg twice weekly/50 mg once weekly for any PASI score (50%, 75%, 90% and 100% reduction) in bDMARD-naïve patients [ 36 ].

For joint outcomes, Mease et al. compared guselkumab Q4W and Q8W with other b/tsDMARDs in a network of 21 treatments in an overall PsA population for ACR50 [ 34 ]. Both guselkumab dosing schedules were better than abatacept and apremilast, but golimumab 2 mg i.v. had a higher likelihood of ACR50 response than guselkumab Q8W [ 34 ]. Despite MDA being assessed in clinical trials for bDMARD therapies and a treatment target in PsA [ 46 ], evidence for comparative efficacy for this outcome is limited. None of the most recent NMAs before this one included an analysis of MDA [ 34–36 ]. With regard to safety outcomes, previous NMAs evaluating SAEs also resulted in either no difference between b/tsDMARDs vs placebo or other b/tsDMARDs [ 34 , 36 , 44 , 45 ].

This study has a number of strengths. To our knowledge this NMA represents the most comprehensive and in-depth comparative efficacy analysis of approved treatments in PsA to date. The evidence was derived from a recent SLR, ensuring that new RCTs and updated results from previously published RCTs were included. It is also the first NMA to include the phase 3 BE COMPLETE and BE OPTIMAL trials of bimekizumab [ 19 , 20 ]. Our NMA used robust methods and accounted for variation in placebo response through network meta-regression in accordance with NICE DSU Technical Support Documents [ 31–33 ]. As an acknowledgement of the evolution of treatment advances, separate analyses of b/tsDMARD-naïve and TNFi-experienced subgroups were conducted with the intent to assist healthcare decision-making in different clinical settings. In addition, a panel of clinical experts were consulted from project inception and are authors of this paper, ensuring inclusion of a comprehensive set of clinically meaningful outcomes, including the composite, treat-to-target outcome of MDA.

Despite the robust evidence base and methodology, this NMA has limitations. Indirect treatment comparisons such as this NMA are not a substitute for head-to-head trials. There was heterogeneity in the endpoints and reporting in the included studies. Fewer studies reporting PASI outcomes resulted in smaller networks compared with the network of studies evaluating ACR response criteria. Not all trials reported outcomes at the same timepoint, thereby reducing the comparability of trial results, which has been transparently addressed by noting where week 24 data were used vs week 12, 14 or 16 data. The analyses for the TNFi-experienced population were limited by potential heterogeneity, especially in the analyses where fewer studies were included in the networks, as this group could include patients who had an inadequate response to TNFi or discontinued TNFi treatment due to other reasons (e.g. lost access). Also, in the analyses for the TNFi-experienced population, very low patient numbers for some treatments resulted in less statistical power. Additionally, the data included in the analysis were derived exclusively from RCTs, for which the study populations may not reflect a typical patient population seen in real-world practice. For example, trial results may be different in patients with oligoarthritis who are not well-represented in clinical trials.

Over the years covering our SLR, we acknowledge that patient populations and the PsA treatment landscape have evolved. After a thorough review of baseline patient characteristics, no significant differences were observed across the studies included in the NMA. To further mitigate uncertainty, baseline regression was used to actively correct for changes in the placebo rate over time ensuring a consistent and fair comparison across all included treatments. In addition, our analyses were conducted in separate b/tsDMARD-naïve and TNFi-experienced populations that reflect the evolving PsA patient population over time. Radiographic progression was not within the purview of this NMA because the NMA focused on a shorter timeframe than the 52-week duration typically recommended by the literature for investigating radiographic progression. Furthermore, there is existing literature on this topic, as exemplified by the work of Wang et al. in 2022 [ 47 ]. Nevertheless, the comprehensive and current evidence base, examination of multiple endpoints, and consistency with previous reported NMAs lend credence to our results.

Overall, the results of this NMA demonstrated the favourable relative efficacy and safety of bimekizumab 160 mg Q4W vs all approved treatments for PsA. Bimekizumab ranked high in terms of efficacy on joint, skin and MDA outcomes in both b/tsDMARD-naïve and TNFi-experienced patient populations, and showed comparable safety to other treatments. In the evolving PsA treatment landscape, bimekizumab 160 mg Q4W is a potentially beneficial treatment option for patients with PsA.

Supplementary material is available at Rheumatology online.

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

This study was funded in full by UCB Pharma.

Disclosure statement : P.J.M.: has received research grants from AbbVie, Amgen, BMS, Eli Lilly, Gilead, Janssen, Novartis, Pfizer, Sun Pharma and UCB Pharma; consultancy fees from AbbVie, Acelyrin, Aclaris, Amgen, BMS, Boehringer Ingelheim, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Moonlake Pharma, Novartis, Pfizer, Sun Pharma and UCB Pharma; and speakers’ bureau for AbbVie, Amgen, Eli Lilly, Janssen, Novartis, Pfizer and UCB Pharma. L.C.C.: received grants/research support from AbbVie, Amgen, Celgene, Eli Lilly, Janssen, Novartis, Pfizer and UCB; worked as a paid consultant for AbbVie, Amgen, Bristol Myers Squibb, Celgene, Eli Lilly, Gilead, Galapagos, Janssen, Moonlake, Novartis, Pfizer and UCB; and has been paid as a speaker for AbbVie, Amgen, Biogen, Celgene, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Medac, Novartis, Pfizer and UCB. D.D.G.: consultant and/or received grant support from Abbvie, Amgen, BMS, Celgene, Eli Lilly, Galapagos, Gilead, Janssen, Novartis, Pfizer and UCB. J.F.M.: consultant and/or investigator for AbbVie, Amgen, Biogen, BMS, Dermavant, Eli Lilly, Janssen, LEO Pharma, Novartis, Pfizer, Regeneron, Sanofi, Sun Pharma and UCB Pharma. P.N.: research grants, clinical trials and honoraria for advice and lectures on behalf of AbbVie, Boehringer Ingelheim, BMS, Eli Lilly, Galapagos/Gilead, GSK, Janssen, Novartis, Pfizer, Samsung, Sanofi and UCB Pharma. S.G. and V.L.-K.: employees of Cytel, Inc. which served as a consultant on the project. A.R.P., D.W. and V.T.: employees and stockholders of UCB Pharma.

The authors acknowledge Leah Wiltshire of Cytel for medical writing and editorial assistance based on the authors’ input and direction, Heather Edens (UCB Pharma, Smyrna, GA, USA) for publication coordination and Costello Medical for review management, which were funded by UCB Pharma. This analysis was funded by UCB Pharma in accordance with Good Publication Practice (GPP 2022) guidelines ( http://www.ismpp.org/gpp-2022 ). Data were previously presented at ISPOR-US 2023 (Boston, MA, USA, 7–10 May 2023).

Gossec L , Baraliakos X , Kerschbaumer A et al.  EULAR recommendations for the management of psoriatic arthritis with pharmacological therapies: 2019 update . Ann Rheum Dis 2020 ; 79 : 700 – 12 .

Google Scholar

Coates LC , Soriano ER , Corp N et al. ; GRAPPA Treatment Recommendations domain subcommittees . Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 18 : 465 – 79 .

Fitzgerald O , Ogdie A , Chandran V et al.  Psoriatic arthritis . Nat Rev Dis Primers 2021 ; 7 : 59 .

Coates LC , Soriano ER , Corp N et al.  Treatment recommendations domain subcommittees. Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 27 : 1 – 15 .

Najm A , Goodyear CS , McInnes IB , Siebert S. Phenotypic heterogeneity in psoriatic arthritis: towards tissue pathology-based therapy . Nat Rev Rheumatol 2023 ; 19 : 153 – 65 .

Ogdie A , Weiss P. The epidemiology of psoriatic arthritis . Rheum Dis Clin North Am 2015 ; 41 : 545 – 68 .

Alinaghi F , Calov M , Kristensen LE et al.  Prevalence of psoriatic arthritis in patients with psoriasis: a systematic review and meta-analysis of observational and clinical studies . J Am Acad Dermatol 2019 ; 80 : 251 – 65.e19 .

Singh JA , Guyatt G , Ogdie A et al.  Special Article: 2018 American college of rheumatology/national psoriasis foundation guideline for the treatment of psoriatic arthritis . Arthritis Rheumatol 2019 ; 71 : 5 – 32 .

Coates LC , Robinson DE , Orbai AM et al.  What influences patients' opinion of remission and low disease activity in psoriatic arthritis? Principal component analysis of an international study . Rheumatology (Oxford) 2021 ; 60 : 5292 – 9 .

Gondo G , Mosca M , Hong J et al.  Demographic and clinical factors associated with patient-reported remission in psoriatic arthritis . Dermatol Ther (Heidelb) 2022 ; 12 : 1885 – 95 .

Glatt S , Baeten D , Baker T et al.  Dual IL-17A and IL-17F neutralisation by bimekizumab in psoriatic arthritis: evidence from preclinical experiments and a randomised placebo-controlled clinical trial that IL-17F contributes to human chronic tissue inflammation . Ann Rheum Dis 2018 ; 77 : 523 – 32 .

UCB Pharma S.A . Bimzelx ® (bimekizumab): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/bimzelx (26 June 2023, date last accessed).

Adams R , Maroof A , Baker T et al.  Bimekizumab, a novel humanized IgG1 antibody that neutralizes both IL-17A and IL-17F . Front Immunol 2020 ; 11 : 1894 .

Burns LA , Maroof A , Marshall D et al.  Presence, function, and regulation of IL-17F-expressing human CD4(+) T cells . Eur J Immunol 2020 ; 50 : 568 – 80 .

Kuestner R , Taft D , Haran A et al.  Identification of the IL-17 receptor related molecule IL-17RC as the receptor for IL-17F . J Immunol 2007 ; 179 : 5462 – 73 .

Johansen C , Usher PA , Kjellerup RB et al.  Characterization of the interleukin-17 isoforms and receptors in lesional psoriatic skin . Br J Dermatol 2009 ; 160 : 319 – 24 .

Shah M , Maroof A , Gikas P et al.  Dual neutralisation of IL-17F and IL-17A with bimekizumab blocks inflammation-driven osteogenic differentiation of human periosteal cells . RMD Open 2020 ; 6 : e001306 .

Reich K , Warren RB , Lebwohl M et al.  Bimekizumab versus Secukinumab in Plaque Psoriasis . New Engl J Med 2021 ; 385 : 142 – 52 .

McInnes IB , Asahina A , Coates LC et al.  Bimekizumab in patients with psoriatic arthritis, naive to biologic treatment: a randomised, double-blind, placebo-controlled, phase 3 trial (BE OPTIMAL) . Lancet 2023 ; 401 : 25 – 37 .

Merola JF , Landewe R , McInnes IB et al.  Bimekizumab in patients with active psoriatic arthritis and previous inadequate response or intolerance to tumour necrosis factor-alpha inhibitors: a randomised, double-blind, placebo-controlled, phase 3 trial (BE COMPLETE) . Lancet 2023 ; 401 : 38 – 48 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Higgins JP. Cochrane handbook for systematic reviews of interventions. Vol. 2. 2nd edn. Chichester, UK: John Wiley & Sons, 2019 .

National Institute for Health and Care Excellence . The guidelines manual: Process and methods [PMG6]. 2012 . https://www.nice.org.uk/process/pmg6/chapter/introduction (3 March 2023, date last accessed).

Booth AM , Wright KE , Outhwaite H. Centre for Reviews and Dissemination databases: value, content, and developments . Int J Technol Assess Health Care 2010 ; 26 : 470 – 2 .

Sterne JAC , Savovic J , Page MJ et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials . BMJ 2019 ; 366 : l4898 .

Daly C , Dias S , Welton N , Anwer S , Ades A. NICE Guidelines Technical Support Unit. Meta-Analysis: Guideline Methodology Document 1 (Version 1). 2021 . http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/mpes/nice/guideline-methodology-documents-gmds/ (1 March 2023, date last accessed).

Dias S , Caldwell DM. Network meta-analysis explained . Archi Dis Childhood Fetal Neonatal Ed 2019 ; 104 : F8 – F12 .

Merola JF , Lockshin B , Mody EA. Switching biologics in the treatment of psoriatic arthritis . Semin Arthritis Rheum 2017 ; 47 : 29 – 37 .

Openbugs (website) 2014 . http://www.openbugs.net/w/FrontPage (6 April 2023, date last accessed).

Lunn D , Spiegelhalter D , Thomas A , Best N. The BUGS project: evolution, critique and future directions . Stat Med 2009 ; 28 : 3049 – 67 .

Dias S , Welton NJ , Sutton AJ , Ades AE. NICE DSU technical support document 2: a generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. London. 2016 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Welton N , Sutton AJ , Caldwell DM , Lu G , Ades AE. NICE DSU technical support document 4: inconsistency in networks of evidence based on randomised controlled trials. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Sutton AJ , Welton N , Ades AE. NICE DSU Technical support document 3: heterogeneity: subgroups, meta-regression, bias and bias-adjustment. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Mease PJ , McInnes IB , Tam LS et al.  Comparative effectiveness of guselkumab in psoriatic arthritis: results from systematic literature review and network meta-analysis . Rheumatology (Oxford) 2021 ; 60 : 2109 – 21 .

McInnes IB , Sawyer LM , Markus K et al.  Targeted systemic therapies for psoriatic arthritis: a systematic review and comparative synthesis of short-term articular, dermatological, enthesitis and dactylitis outcomes . RMD Open 2022 ; 8 : e002074 .

Ruyssen-Witrand A , Perry R , Watkins C et al.  Efficacy and safety of biologics in psoriatic arthritis: a systematic literature review and network meta-analysis . RMD Open 2020 ; 6 : e001117 .

Pfizer Inc . XELJANZ ® (tofacitinib): Summary of Product Characteristics. 2022 . https://www.ema.europa.eu/en/medicines/human/EPAR/xeljanz (4 May 2023, date last accessed).

Pfizer Inc . XELJANZ ® (tofacitinib): US Prescribing Information. 2022 . https://labeling.pfizer.com/ShowLabeling.aspx?id=959 (4 May 2023, date last accessed).

Abbvie Inc . RINVOQ ® (upadacitinib) extended-release tablets, for oral use: US Prescribing Information. 2023 . https://www.rxabbvie.com/pdf/rinvoq_pi.pdf (4 May 2023, date last accessed).

AbbVie Deutschland GmbH & Co . KG. RINVOQ ® (upadacitinib): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/rinvoq (4 May 2023, date last accessed).

Warren RB , Blauvelt A , Bagel J et al.  Bimekizumab versus Adalimumab in Plaque Psoriasis . N Engl J Med 2021 ; 385 : 130 – 41 .

Reich K , Papp KA , Blauvelt A et al.  Bimekizumab versus ustekinumab for the treatment of moderate to severe plaque psoriasis (BE VIVID): efficacy and safety from a 52-week, multicentre, double-blind, active comparator and placebo controlled phase 3 trial . Lancet 2021 ; 397 : 487 – 98 .

Gladman DD , Orbai AM , Gomez-Reino J et al.  Network meta-analysis of tofacitinib, biologic disease-modifying antirheumatic drugs, and apremilast for the treatment of psoriatic arthritis . Curr Ther Res Clin Exp 2020 ; 93 : 100601 .

Qiu M , Xu Z , Gao W et al.  Fourteen small molecule and biological agents for psoriatic arthritis: a network meta-analysis of randomized controlled trials . Medicine (Baltimore) 2020 ; 99 : e21447 .

Kawalec P , Holko P , Mocko P , Pilc A. Comparative effectiveness of abatacept, apremilast, secukinumab and ustekinumab treatment of psoriatic arthritis: a systematic review and network meta-analysis . Rheumatol Int 2018 ; 38 : 189 – 201 .

Gossec L , McGonagle D , Korotaeva T et al.  Minimal disease activity as a treatment target in psoriatic arthritis: a review of the literature . J Rheumatol 2018 ; 45 : 6 – 13 .

Wang SH , Yu CL , Wang TY , Yang CH , Chi CC. Biologic disease-modifying antirheumatic drugs for preventing radiographic progression in psoriatic arthritis: a systematic review and network meta-analysis . Pharmaceutics 2022 ; 14 .

Supplementary data

Email alerts

Citing articles via.

• Rheumatology Twitter
• BSR Twitter
• BSR Facebook
• Recommend to Your Librarian

Affiliations

• Online ISSN 1462-0332
• Print ISSN 1462-0324
• Copyright © 2024 British Society for Rheumatology
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Bilothorax: A Case Report and Systematic Literature Review of the Rare Entity

Affiliations.

• 1 Division of Pulmonary and Critical Care Medicine Virginia Tech Carilion School of Medicine, Roanoke, VA, USA.
• 2 Department of Family Nurse Practitioner Frontier Nursing University, Versailles, KY, USA.
• 3 Division of Pulmonary and Critical Care Medicine University of Kentucky, Bowling Green, KY, USA.
• 4 Department of Internal Medicine Mount Sinai Chicago, Chicago, IL, USA.
• 5 Department of Emergency Medicine Grande International Hospital, Kathmandu, Nepal.
• 6 Department of Internal Medicine Universal College of Medical Sciences, Siddharthanagar, Nepal.
• PMID: 38947176
• PMCID: PMC11213635
• DOI: 10.1155/2024/3973056

Background: Bilothorax is defined as the presence of bile in the pleural space. It is a rare condition, and diagnosis is confirmed with a pleural fluid-to-serum bilirubin ratio of >1.

Methods: The PubMed, Embase, Google Scholar, and CINAHL databases were searched using predetermined Boolean parameters. The systematic literature review was done per PRISMA guidelines. Retrospective studies, case series, case reports, and conference abstracts were included. The patients with reported pleural fluid analyses were pooled for fluid parameter data analysis.

Results: Of 838 articles identified through the inclusion criteria and removing 105 duplicates, 732 articles were screened with abstracts, and 285 were screened for full article review. After this, 123 studies qualified for further detailed review, and of these, 115 were pooled for data analysis. The mean pleural fluid and serum bilirubin levels were 72 mg/dL and 61 mg/dL, respectively, with a mean pleural fluid-to-serum bilirubin ratio of 3.47. In most cases, the bilothorax was reported as a subacute or remote complication of hepatobiliary surgery or procedure, and traumatic injury to the chest or abdomen was the second most common cause. Tube thoracostomy was the main treatment modality (73.83%), followed by serial thoracentesis. Fifty-two patients (51.30%) had associated bronchopleural fistulas. The mortality was considerable, with 18/115 (15.65%) reported death. Most of the patients with mortality had advanced hepatobiliary cancer and were noted to die of complications not related to bilothorax.

Conclusion: Bilothorax should be suspected in patients presenting with pleural effusion following surgical manipulation of hepatobiliary structures or a traumatic injury to the chest. This review is registered with CRD42023438426.

Copyright © 2024 Roshan Acharya et al.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no conflicts of interest.

PRISMA diagram of included studies.

• Austin A., Fox N., Huggins J. T., Chopra A. The green pleural effusion: a comprehensive review of the bilothorax with case series. The Official Journal of the International Society of Pleural Diseases . 2017;4:21–31.
• Aneja A., Schimmel M., Nostrand K. V. Bilothorax: a rare occurrence after laparoscopic gastric sleeve resection. Chest . 2020;158(4):p. A1277. doi: 10.1016/j.chest.2020.08.1164. - DOI
• Austin A., Chopra A. Loculated bilothorax: a rare sequalae of obstructive jaundice without biliopleural fistulization. Chest . 2016;150(4):p. 596A. doi: 10.1016/j.chest.2016.08.685. - DOI
• Bilal M., Chong J., Lega M. American Thoracic Society International Conference Abstracts . American Thoracic Society; 2015. Bilothorax: it can happen. In: B39 Do You Want to Know a Secret? Case Reports in Pleural Disease; pp. A2957–A2957. - DOI
• Celis C. A., Tobon M., Ramirez R., Garcia M., Suarez J. B39 Do You Want to Know a Secret? Case Reports in Pleural Disease . American Thoracic Society; 2015. Cholethorax secondary to percutaneous transhepatic gallbladder drainage: a case report; pp. A2978–A2978. American Thoracic Society International Conference Abstracts. - DOI

Publication types

• Search in MeSH

Related information

• PubChem Compound (MeSH Keyword)

LinkOut - more resources

Full text sources.

• Hindawi Limited
• PubMed Central
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Factors, prediction, and explainability of vehicle accident risk due to driving behavior through machine learning: a systematic literature review, 2013–2023.

Share and Cite

Lacherre, J.; Castillo-Sequera, J.L.; Mauricio, D. Factors, Prediction, and Explainability of Vehicle Accident Risk Due to Driving Behavior through Machine Learning: A Systematic Literature Review, 2013–2023. Computation 2024 , 12 , 131. https://doi.org/10.3390/computation12070131

Lacherre J, Castillo-Sequera JL, Mauricio D. Factors, Prediction, and Explainability of Vehicle Accident Risk Due to Driving Behavior through Machine Learning: A Systematic Literature Review, 2013–2023. Computation . 2024; 12(7):131. https://doi.org/10.3390/computation12070131

Lacherre, Javier, José Luis Castillo-Sequera, and David Mauricio. 2024. "Factors, Prediction, and Explainability of Vehicle Accident Risk Due to Driving Behavior through Machine Learning: A Systematic Literature Review, 2013–2023" Computation 12, no. 7: 131. https://doi.org/10.3390/computation12070131

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals