traditional and systematic review of literature

• En español – ExME
• Em português – EME

Traditional reviews vs. systematic reviews

Posted on 3rd February 2016 by Weyinmi Demeyin

Millions of articles are published yearly (1) , making it difficult for clinicians to keep abreast of the literature. Reviews of literature are necessary in order to provide clinicians with accurate, up to date information to ensure appropriate management of their patients. Reviews usually involve summaries and synthesis of primary research findings on a particular topic of interest and can be grouped into 2 main categories; the ‘traditional’ review and the ‘systematic’ review with major differences between them.

Traditional reviews provide a broad overview of a research topic with no clear methodological approach (2) . Information is collected and interpreted unsystematically with subjective summaries of findings. Authors aim to describe and discuss the literature from a contextual or theoretical point of view. Although the reviews may be conducted by topic experts, due to preconceived ideas or conclusions, they could be subject to bias.

Systematic reviews are overviews of the literature undertaken by identifying, critically appraising and synthesising results of primary research studies using an explicit, methodological approach(3). They aim to summarise the best available evidence on a particular research topic.

The main differences between traditional reviews and systematic reviews are summarised below in terms of the following characteristics: Authors, Study protocol, Research question, Search strategy, Sources of literature, Selection criteria, Critical appraisal, Synthesis, Conclusions, Reproducibility, and Update.

Traditional reviews

• Authors: One or more authors usually experts in the topic of interest
• Study protocol: No study protocol
• Research question: Broad to specific question, hypothesis not stated
• Search strategy: No detailed search strategy, search is probably conducted using keywords
• Sources of literature: Not usually stated and non-exhaustive, usually well-known articles. Prone to publication bias
• Selection criteria: No specific selection criteria, usually subjective. Prone to selection bias
• Critical appraisal: Variable evaluation of study quality or method
• Synthesis: Often qualitative synthesis of evidence
• Conclusions: Sometimes evidence based but can be influenced by author’s personal belief
• Reproducibility: Findings cannot be reproduced independently as conclusions may be subjective
• Update: Cannot be continuously updated

Systematic reviews

• Authors: Two or more authors are involved in good quality systematic reviews, may comprise experts in the different stages of the review
• Study protocol: Written study protocol which includes details of the methods to be used
• Research question: Specific question which may have all or some of PICO components (Population, Intervention, Comparator, and Outcome). Hypothesis is stated
• Search strategy: Detailed and comprehensive search strategy is developed
• Sources of literature: List of databases, websites and other sources of included studies are listed. Both published and unpublished literature are considered
• Selection criteria: Specific inclusion and exclusion criteria
• Critical appraisal: Rigorous appraisal of study quality
• Synthesis: Narrative, quantitative or qualitative synthesis
• Conclusions: Conclusions drawn are evidence based
• Reproducibility: Accurate documentation of method means results can be reproduced
• Update: Systematic reviews can be periodically updated to include new evidence

Decisions and health policies about patient care should be evidence based in order to provide the best treatment for patients. Systematic reviews provide a means of systematically identifying and synthesising the evidence, making it easier for policy makers and practitioners to assess such relevant information and hopefully improve patient outcomes.

• Fletcher RH, Fletcher SW. Evidence-Based Approach to the Medical Literature. Journal of General Internal Medicine. 1997; 12(Suppl 2):S5-S14. doi:10.1046/j.1525-1497.12.s2.1.x. Available from:  http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1497222/
• Rother ET. Systematic literature review X narrative review. Acta paul. enferm. [Internet]. 2007 June [cited 2015 Dec 25]; 20(2): v-vi. Available from: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0103-21002007000200001&lng=en. http://dx.doi.org/10.1590/S0103-21002007000200001
• Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J. Undertaking systematic reviews of research on effectiveness: CRD’s guidance for carrying out or commissioning reviews. NHS Centre for Reviews and Dissemination; 2001.

Weyinmi Demeyin

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on Traditional reviews vs. systematic reviews

THE INFORMATION IS VERY MUCH VALUABLE, A LOT IS INDEED EXPECTED IN ORDER TO MASTER SYSTEMATIC REVIEW

Thank you very much for the information here. My question is : Is it possible for me to do a systematic review which is not directed toward patients but just a specific population? To be specific can I do a systematic review on the mental health needs of students?

Hi Rosemary, I wonder whether it would be useful for you to look at Module 1 of the Cochrane Interactive Learning modules. This is a free module, open to everyone (you will just need to register for a Cochrane account if you don’t already have one). This guides you through conducting a systematic review, with a section specifically around defining your research question, which I feel will help you in understanding your question further. Head to this link for more details: https://training.cochrane.org/interactivelearning

I wonder if you have had a search on the Cochrane Library as yet, to see what Cochrane systematic reviews already exist? There is one review, titled “Psychological interventions to foster resilience in healthcare students” which may be of interest: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013684/full You can run searches on the library by the population and intervention you are interested in.

I hope these help you start in your investigations. Best wishes. Emma.

La revisión sistemática vale si hay solo un autor?

HI Alex, so sorry for the delay in replying to you. Yes, that is a very good point. I have copied a paragraph from the Cochrane Handbook, here, which does say that for a Cochrane Review, you should have more than one author.

“Cochrane Reviews should be undertaken by more than one person. In putting together a team, authors should consider the need for clinical and methodological expertise for the review, as well as the perspectives of stakeholders. Cochrane author teams are encouraged to seek and incorporate the views of users, including consumers, clinicians and those from varying regions and settings to develop protocols and reviews. Author teams for reviews relevant to particular settings (e.g. neglected tropical diseases) should involve contributors experienced in those settings”.

Thank you for the discussion point, much appreciated.

Hello, I’d like to ask you a question: what’s the difference between systematic review and systematized review? In addition, if the screening process of the review was made by only one author, is still a systematic or is a systematized review? Thanks

Hi. This article from Grant & Booth is a really good one to look at explaining different types of reviews: https://onlinelibrary.wiley.com/doi/10.1111/j.1471-1842.2009.00848.x It includes Systematic Reviews and Systematized Reviews. In answer to your second question, have a look at this Chapter from the Cochrane handbook. It covers the question about ‘Who should do a systematic review’. https://training.cochrane.org/handbook/current/chapter-01

A really relevant part of this chapter is this: “Systematic reviews should be undertaken by a team. Indeed, Cochrane will not publish a review that is proposed to be undertaken by a single person. Working as a team not only spreads the effort, but ensures that tasks such as the selection of studies for eligibility, data extraction and rating the certainty of the evidence will be performed by at least two people independently, minimizing the likelihood of errors.”

I hope this helps with the question. Best wishes. Emma.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

What do trialists do about participants who are ‘lost to follow-up’?

Participants in clinical trials may exit the study prior to having their results collated; in this case, what do we do with their results?

Family Therapy approaches for Anorexia Nervosa

Is Family Therapy effective in the treatment of Anorexia Nervosa? Emily summarises a recent Cochrane Review in this blog and examines the evidence.

Antihypertensive drugs for primary prevention – at what blood pressure do we start treatment?

In this blog, Giorgio Karam examines the evidence on antihypertensive drugs for primary prevention – when do we start treatment?

• Research Process
• Manuscript Preparation
• Manuscript Review
• Publication Process
• Publication Recognition
• Language Editing Services
• Translation Services

Systematic Literature Review or Literature Review?

• 3 minute read
• 51.5K views

Table of Contents

As a researcher, you may be required to conduct a literature review. But what kind of review do you need to complete? Is it a systematic literature review or a standard literature review? In this article, we’ll outline the purpose of a systematic literature review, the difference between literature review and systematic review, and other important aspects of systematic literature reviews.

What is a Systematic Literature Review?

The purpose of systematic literature reviews is simple. Essentially, it is to provide a high-level of a particular research question. This question, in and of itself, is highly focused to match the review of the literature related to the topic at hand. For example, a focused question related to medical or clinical outcomes.

The components of a systematic literature review are quite different from the standard literature review research theses that most of us are used to (more on this below). And because of the specificity of the research question, typically a systematic literature review involves more than one primary author. There’s more work related to a systematic literature review, so it makes sense to divide the work among two or three (or even more) researchers.

Your systematic literature review will follow very clear and defined protocols that are decided on prior to any review. This involves extensive planning, and a deliberately designed search strategy that is in tune with the specific research question. Every aspect of a systematic literature review, including the research protocols, which databases are used, and dates of each search, must be transparent so that other researchers can be assured that the systematic literature review is comprehensive and focused.

Most systematic literature reviews originated in the world of medicine science. Now, they also include any evidence-based research questions. In addition to the focus and transparency of these types of reviews, additional aspects of a quality systematic literature review includes:

• Clear and concise review and summary
• Comprehensive coverage of the topic
• Accessibility and equality of the research reviewed

Systematic Review vs Literature Review

The difference between literature review and systematic review comes back to the initial research question. Whereas the systematic review is very specific and focused, the standard literature review is much more general. The components of a literature review, for example, are similar to any other research paper. That is, it includes an introduction, description of the methods used, a discussion and conclusion, as well as a reference list or bibliography.

A systematic review, however, includes entirely different components that reflect the specificity of its research question, and the requirement for transparency and inclusion. For instance, the systematic review will include:

• Eligibility criteria for included research
• A description of the systematic research search strategy
• An assessment of the validity of reviewed research
• Interpretations of the results of research included in the review

As you can see, contrary to the general overview or summary of a topic, the systematic literature review includes much more detail and work to compile than a standard literature review. Indeed, it can take years to conduct and write a systematic literature review. But the information that practitioners and other researchers can glean from a systematic literature review is, by its very nature, exceptionally valuable.

This is not to diminish the value of the standard literature review. The importance of literature reviews in research writing is discussed in this article . It’s just that the two types of research reviews answer different questions, and, therefore, have different purposes and roles in the world of research and evidence-based writing.

Systematic Literature Review vs Meta Analysis

It would be understandable to think that a systematic literature review is similar to a meta analysis. But, whereas a systematic review can include several research studies to answer a specific question, typically a meta analysis includes a comparison of different studies to suss out any inconsistencies or discrepancies. For more about this topic, check out Systematic Review VS Meta-Analysis article.

Language Editing Plus

With Elsevier’s Language Editing Plus services , you can relax with our complete language review of your systematic literature review or literature review, or any other type of manuscript or scientific presentation. Our editors are PhD or PhD candidates, who are native-English speakers. Language Editing Plus includes checking the logic and flow of your manuscript, reference checks, formatting in accordance to your chosen journal and even a custom cover letter. Our most comprehensive editing package, Language Editing Plus also includes any English-editing needs for up to 180 days.

How to Make a PowerPoint Presentation of Your Research Paper

What is and How to Write a Good Hypothesis in Research?

You may also like.

Descriptive Research Design and Its Myriad Uses

Five Common Mistakes to Avoid When Writing a Biomedical Research Paper

Making Technical Writing in Environmental Engineering Accessible

To Err is Not Human: The Dangers of AI-assisted Academic Writing

When Data Speak, Listen: Importance of Data Collection and Analysis Methods

Choosing the Right Research Methodology: A Guide for Researchers

Why is data validation important in research?

Writing a good review article

Input your search keywords and press Enter.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved July 5, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Literature Review: Traditional or narrative literature reviews

Traditional or narrative literature reviews.

• Scoping Reviews
• Systematic literature reviews
• Annotated bibliography
• Keeping up to date with literature
• Finding a thesis
• Evaluating sources and critical appraisal of literature
• Managing and analysing your literature
• Further reading and resources

A narrative or traditional literature review is a comprehensive, critical and objective analysis of the current knowledge on a topic. They are an essential part of the research process and help to establish a theoretical framework and focus or context for your research. A literature review will help you to identify patterns and trends in the literature so that you can identify gaps or inconsistencies in a body of knowledge. This should lead you to a sufficiently focused research question that justifies your research.

Onwuegbuzie and Frels (pp 24-25, 2016) define four common types of narrative reviews:

• General literature review that provides a review of the most important and critical aspects of the current knowledge of the topic. This general literature review forms the introduction to a thesis or dissertation and must be defined by the research objective, underlying hypothesis or problem or the reviewer's argumentative thesis.
• Theoretical literature review which examines how theory shapes or frames research
• Methodological literature review where the research methods and design are described. These methodological reviews outline the strengths and weaknesses of the methods used and provide future direction
• Historical literature review which focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

References and additional resources

Baker, J. D. (2016) The purpose, process and methods of writing a literature review: Editorial . Association of Operating Room Nurses. AORN Journal, 103 (3), 265-269. doi:10.1016/j.aorn.2016.01.016

• << Previous: Types of literature reviews
• Next: Scoping Reviews >>
• Last Updated: Jun 3, 2024 9:25 AM
• URL: https://libguides.csu.edu.au/review

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Types of Literature Reviews : Home

• Health Science Information Consortium
• University Health Network - New
• Types of Literature Reviews

Need More Help?

• Knowledge Synthesis Support
• Literature Search Request
• The Right Review for You - Workshop Recording (YouTube) 27min video From UHN Libraries Recorded Nov 2021
• The Screening Phase for Reviews Tutorial This tutorial presents information on the screening process for systematic reviews or other knowledge syntheses, and contains a variety of resources for successfully preparing to complete this important research stage.
• Workshops Find more UHN Libraries workshops, live and on-demand, and other learning opportunities helpful for knowledge synthesis projects.

How to Choose Your Review Method

TREAD* Lightly and Consider...

• Available T ime for conducting your review
• Any R esource constraints within which you must deliver your review
• Any requirements for specialist E xpertise in order to complete the review
• The requirements of the A udience for your review and its intended purpose
• The richness, thickness and availability of D ata within included studies

* Booth A, Sutton A, Papaioannou D.  Systematic approaches to a successful literature review.  2nd edition.  Los Angeles, CA:  Sage, 2016.  (p.36)

How do I write a Review Protocol?

• What is a Protocol? (UofT)
• Guidance on Registering a Review with PROSPERO

Writing Resources

• Advice on Academic Writing (University of Toronto)
• How to write a great research paper using reporting guidelines (EQUATOR Network)
• Instructions to Authors in the Health Sciences
• Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals (ICMJE)
• Writing resources guide (BMC)

What is a Literature Review?

A literature review provides an overview of what's been written about a specific topic. It is a generic term. There are many different types of literature reviews which can cover a wide range of subjects at various levels of completeness and comprehensiveness. Choosing the type of review you wish to conduct will depend on the purpose of your review, and the time and resources you have available.

This page will provide definitions of some of the most common review types in the health sciences and links to relevant reporting guidelines or methodological papers.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies . Health Information & Libraries Journal . 2009 Jun 1;26(2):91-108. 

• Summary of Five Types of Reviews Table summarizing the characteristics, guidelines etc. of 5 common types of review article.

Traditional (Narrative) Review

Traditional (narrative) literature reviews provide a broad overview of a research topic with no clear methodological approach. Information is collected and interpreted unsystematically with subjective summaries of findings. Authors aim to describe and discuss the literature from a contextual or theoretical point of view. Although the reviews may be conducted by topic experts, due to preconceived ideas or conclusions, they could be subject to bias. This sort of literature review can be appropriate if you have a broad topic area, are working on your own, or have time constraints.

Agarwal S, Charlesworth M, Elrakhawy M. How to write a narrative review . Anaesthesia . 2023;78(9):1162-1166. doi:10.1111/anae.16016

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade . Journal of Chiropractic Medicine . 2006;5(3):101-117. doi:10.1016/S0899-3467(07)60142-6.

Ferrari R. Writing narrative style literature reviews . Medical Writing. 2015 Dec 1;24(4):230-5.

Greenhalgh T, Thorne S, Malterud K. Time to challenge the spurious hierarchy of systematic over narrative reviews ? European journal of clinical investigation . 2018;48:e12931.

Knowledge Synthesis

• What is Knowledge Synthesis?
• Learn more!

CIHR Definition of Knowledge Syntheses:

“The contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods.”  - A Guide to Knowledge Synthesis, CIHR

Grimshaw J. A Guide to Knowledge Synthesis [Internet]. CIHR. Canadian Institutes of Health Research; 2010.

Canadian Institutes of Health Research. Synthesis Resources [Internet]. CIHR. Canadian Institutes of Health Research; 2013.

Booth A, Noyes J, Flemming K, Gerhardus A, Wahlster P, van der Wilt, Gert Jan, et al.  Structured methodology review identified seven (RETREAT) criteria for selecting qualitative evidence synthesis approaches . Journal of clinical epidemiology. 2018;99:41-52.

Kastner M, Tricco AC, Soobiah C, et al. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review . BMC Medical Research Methodology . 2012;12:114. doi:10.1186/1471-2288-12-114.

Kastner M, Antony J, Soobiah C, Straus SE, Tricco AC. Conceptual recommendations for selecting the most appropriate knowledge synthesis method to answer research questions related to complex evidence . Journal of Clinical Epidemiology . 2016;73:43-49.

Knowledge Synthesis Support at UHN

Common Types of Knowledge Syntheses

• Systematic Reviews
• Meta-Analysis
• Scoping Reviews
• Rapid or Restricted Reviews
• Clinical Practice Guidelines
• Realist Reviews
• Mixed Methods Reviews
• Qualitative Synthesis
• Narrative Synthesis

A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question. Researchers conducting systematic reviews use explicit methods aimed at minimizing bias, in order to produce more reliable findings that can be used to inform decision making. (See Section 1.2 in the Cochrane Handbook for Systematic Reviews of Interventions .)

A systematic review is not the same as a traditional (narrative) review or a literature review. Unlike other kinds of reviews, systematic reviews must be as thorough and unbiased as possible, and must also make explicit how the search was conducted. Systematic reviews may or may not include a meta-analysis.

On average, a systematic review project takes a year. If your timelines are shorter, you may wish to consider other types of synthesis projects or a traditional (narrative) review. See suggested timelines for a Cochrane Review for reference.

Systematic Review Overview (UHN)

Systematic Review Overview workshop recording (UHN)

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

Greyson D, Rafferty E, Slater L, et al. Systematic review searches must be systematic, comprehensive, and transparent: a critique of Perman et al. BMC public health . 2019;19:153.

Ioannidis J. P. (2016). The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses . The Milbank quarterly , 94 (3), 485-514.

A subset of systematic reviews. Meta-analysis is a technique that statistically combines the results of quantitative studies to provide a more precise effect of the results.

"..a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting and synthesizing existing knowledge." (Colquhoun, HL et al., 2014)

Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework .   International Journal of Social Research Methodology: Theory and Practice , 8 (1), 19-32. doi:10.1080/1364557032000119616.

Levac, D., Colquhoun, H. & O'Brien, K.K. Scoping studies: advancing the methodology . Implementation Sci 5 , 69 (2010). https://doi.org/10.1186/1748-5908-5-69

Colquhoun, H. L., Levac, D., O'Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., . . . Moher, D. (2014). Scoping reviews: Time for clarity in definition, methods, and reporting . Journal of Clinical Epidemiology , 67(12), 1291-1294. doi:10.1016/j.jclinepi.2014.03.013.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews . Int.J.Evid Based.Healthc . 2015 Sep;13(3):141-146.

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis , JBI, 2020.

Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation . Ann Intern Med . [Epub ahead of print ] doi: 10.7326/M18-0850.

“…a type of knowledge synthesis in which systematic review processes are accelerated and methods are streamlined to complete the review more quickly than is the case for typical systematic reviews. Rapid reviews take an average of 5–12 weeks to complete, thus providing evidence within a shorter time frame required for some health policy and systems decisions.” (Tricco AC et al., 2017)

Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews . Implementation Science : IS . 2010;5:56. doi:10.1186/1748-5908-5-56.

Langlois EV, Straus SE, Antony J, King VJ, Tricco AC. Using rapid reviews to strengthen health policy and systems and progress towards universal health coverage . BMJ Global Health . 2019;4:e001178.

Tricco AC, Langlois EV, Straus SE, editors. Rapid reviews to strengthen health policy and systems: a practical guide . Geneva: World Health Organization; 2017. Licence: CC BY-NC-SA 3.0 IGO.

Watt A, Cameron A, Sturm L, Lathlean T , Babidge W, Blamey S, et al. Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment . International Journal of Technology Assessment in Health Care . 2008;24(2):133-9.

“Clinical practice guidelines are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.” Source: Institute of Medicine. (1990). Clinical Practice Guidelines: Directions for a New Program, M.J. Field and K.N. Lohr (eds.) Washington, DC: National Academy Press. Page 38.

–Disclosure of any author conflicts of interest

AGREE Reporting Checklist

Alonso-Coello, P., Oxman, A. D., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., ... & Guyatt, G. H. (2016). GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines . BMJ , 353 , i2089.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review - a new method of systematic review designed for complex policy interventions . Journal of Health Services Research & Policy . 2005;10:21-34.

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research . Implementation science : IS . 2012;7:33.

Wong G, Greenhalgh T, Westhorp G, Pawson R. Realist methods in medical education research: what are they and what can they contribute? Medical Education . 2012;46(1):89-96.

"Mixed-methods systematic reviews can be defined as combining the findings of qualitative and quantitative studies within a single systematic review to address the same overlapping or complementary review questions." (Harden A, 2010)

Harden A.   Mixed-Methods Systematic Reviews: Integrating quantitative and qualitative findings .   NCDDR:FOCUS. 2010.

Lizarondo L, Stern C, Apostolo J, et al. Five common pitfalls in mixed methods systematic reviews: lessons learned . J Clin Epidemiol . 2022;148:178-183. doi:10.1016/j.jclinepi.2022.03.014

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews . Annual review of public health . 2014;35:29-45.

Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews . International journal of evidence-based healthcare . 2015;13:121-131.

The Joanna Briggs Institute 2014 Reviewers Manual: Methodology for JBI Mixed Methods Systematic Reviews .

There are various methods for integrating the results from qualitative studies. "Systematic reviews of qualitative research have an important role in informing the delivery of evidence-based healthcare. Qualitative systematic reviews have investigated the culture of communities, exploring how consumers experience, perceive and manage their health and journey through the health system, and can evaluate components and activities of health services such as health promotion and community development." (Lockwood C et al., 2015)

Booth A, Noyes J, Flemming K, Gerhardus A, Wahlster P, van der Wilt, Gert Jan, et al. Structured methodology review identified seven (RETREAT) criteria for selecting qualitative evidence synthesis approaches . Journal of clinical epidemiology. 2018;99:41-52.

Ring N, Jepson R, Ritchie K. Methods of synthesizing qualitative research studies for health technology assessment . International Journal of Technology Assessment in Health Care . 2011;27:384-390.

Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation . International journal of evidence-based healthcare . 2015;13:179-187.

France EF, Cunningham M, Ring N, et al. Improving reporting of meta-ethnography: The eMERGe reporting guidance . Journal of advanced nursing . 2019.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review . BMC Med Res Methodol . 2009;9:59. Published 2009 Aug 11. doi:10.1186/1471-2288-9-59.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews . BMC Med Res Methodol . 2008;8:45. Published 2008 Jul 10. doi:10.1186/1471-2288-8-45

Lewin S, Booth A, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series . Implementation science : IS . 2018;13:2.

"Narrative synthesis refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis. Whilst narrative synthesis can involve the manipulation of statistical data, the defining characteristic is that it adopts a textual approach to the process of synthesis to ‘tell the story’ of the findings from the included studies." (Popay J, 2006)

Tricco AC, Soobiah C, Antony J, et al. A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method . Journal of Clinical Epidemiology . 2016;73:19-28.

Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews . Lancaster: ESRC Research Methods Programme; 2006.

Snilstveit B, Oliver S, Vojtkova M. Narrative approaches to systematic review and synthesis of evidence for international development policy and practice . Journal of development effectiveness . 2012 Sep 1;4(3):409-29. 

Lucas PJ, Baird J, Arai L, Law C, Roberts HM. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews . BMC medical research methodology . 2007 Dec;7(1):4.

Ryan R. Cochrane Consumer sand Communication Review Group. Cochrane Consumers and Communication Review Group: data synthesis and analysis . June 2013.

• Last Updated: May 8, 2024 10:29 AM
• URL: https://guides.hsict.library.utoronto.ca/c.php?g=705263

We acknowledge this sacred land on which the University Health Network operates. For thousands of years it has been the traditional territory of the Huron-Wendat, the Haudenosaunee, and most recently, the Mississaugas of the Credit River. This territory was the subject of the Dish With One Spoon Wampum Belt Covenant, an agreement between the Haudenosaunee Confederacy and the Confederacy of the Ojibwe and allied nations to peaceably share and care for the resources around the Great Lakes. Today, the meeting place of Toronto is still the home to many Indigenous people from across Turtle Island and we are grateful to have the opportunity to work and learn on this territory

UHN Library and Information Services

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

The difference between a systematic review and a literature review

• Best Practice

Home | Blog | Best Practice | The difference between a systematic review and a literature review

Covidence takes a look at the difference between the two

Most of us are familiar with the terms systematic review and literature review. Both review types synthesise evidence and provide summary information. So what are the differences? What does systematic mean? And which approach is best 🤔 ?

‘ Systematic ‘ describes the review’s methods. It means that they are transparent, reproducible and defined before the search gets underway. That’s important because it helps to minimise the bias that would result from cherry-picking studies in a non-systematic way. 

This brings us to literature reviews. Literature reviews don’t usually apply the same rigour in their methods. That’s because, unlike systematic reviews, they don’t aim to produce an answer to a clinical question. Literature reviews can provide context or background information for a new piece of research. They can also stand alone as a general guide to what is already known about a particular topic. 

Interest in systematic reviews has grown in recent years and the frequency of ‘systematic reviews’ in Google books has overtaken ‘literature reviews’ (with all the usual Ngram Viewer warnings – it searches around 6% of all books, no journals). 

Let’s take a look at the two review types in more detail to highlight some key similarities and differences 👀.

🙋🏾‍♂️ What is a systematic review?

Systematic reviews ask a specific question about the effectiveness of a treatment and answer it by summarising evidence that meets a set of pre-specified criteria. 

The process starts with a research question and a protocol or research plan. A review team searches for studies to answer the question using a highly sensitive search strategy. The retrieved studies are then screened for eligibility using the inclusion and exclusion criteria (this is done by at least two people working independently). Next, the reviewers extract the relevant data and assess the quality of the included studies. Finally, the review team synthesises the extracted study data and presents the results. The process is shown in figure 2 .

The results of a systematic review can be presented in many ways and the choice will depend on factors such as the type of data. Some reviews use meta-analysis to produce a statistical summary of effect estimates. Other reviews use narrative synthesis to present a textual summary.

Covidence accelerates the screening, data extraction, and quality assessment stages of your systematic review. It provides simple workflows and easy collaboration with colleagues around the world.

When is it appropriate to do a systematic review?

If you have a clinical question about the effectiveness of a particular treatment or treatments, you could answer it by conducting a systematic review. Systematic reviews in clinical medicine often follow the PICO framework, which stands for:

👦 Population (or patients)

💊 Intervention

💊 Comparison

Here’s a typical example of a systematic review title that uses the PICO framework: Alarms [intervention] versus drug treatments [comparison] for the prevention of nocturnal enuresis [outcome] in children [population]

Key attributes

• Systematic reviews follow prespecified methods
• The methods are explicit and replicable
• The review team assesses the quality of the evidence and attempts to minimise bias
• Results and conclusions are based on the evidence

🙋🏻‍♀️ What is a literature review?

Literature reviews provide an overview of what is known about a particular topic. They evaluate the material, rather than simply restating it, but the methods used to do this are not usually prespecified and they are not described in detail in the review. The search might be comprehensive but it does not aim to be exhaustive. Literature reviews are also referred to as narrative reviews.

Literature reviews use a topical approach and often take the form of a discussion. Precision and replicability are not the focus, rather the author seeks to demonstrate their understanding and perhaps also present their work in the context of what has come before. Often, this sort of synthesis does not attempt to control for the author’s own bias. The results or conclusion of a literature review is likely to be presented using words rather than statistical methods.

When is it appropriate to do a literature review?

We’ve all written some form of literature review: they are a central part of academic research ✍🏾. Literature reviews often form the introduction to a piece of writing, to provide the context. They can also be used to identify gaps in the literature and the need to fill them with new research 📚.

• Literature reviews take a thematic approach
• They do not specify inclusion or exclusion criteria
• They do not answer a clinical question
• The conclusions might be influenced by the author’s own views

🙋🏽 Ok, but what is a systematic literature review?

A quick internet search retrieves a cool 200 million hits for ‘systematic literature review’. What strange hybrid is this 🤯🤯 ?

Systematic review methodology has its roots in evidence-based medicine but it quickly gained traction in other areas – the social sciences for example – where researchers recognise the value of being methodical and minimising bias. Systematic review methods are increasingly applied to the more traditional types of review, including literature reviews, hence the proliferation of terms like ‘systematic literature review’ and many more.

Beware of the labels 🚨. The terminology used to describe review types can vary by discipline and changes over time. To really understand how any review was done you will need to examine the methods critically and make your own assessment of the quality and reliability of each synthesis 🤓.

Review methods are evolving constantly as researchers find new ways to meet the challenge of synthesising the evidence. Systematic review methods have influenced many other review types, including the traditional literature review. 

Covidence is a web-based tool that saves you time at the screening, selection, data extraction and quality assessment stages of your systematic review. It supports easy collaboration across teams and provides a clear overview of task status.

Get a glimpse inside Covidence and how it works

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Data Extraction Tip 5: Communicate Regularly

The Covidence Global Scholarship recipients are putting evidence-based research into practice. We caught up with some of the winners to discover the impact of their work and find out more about their experiences.

Data Extraction Tip 4: Extract the Right Amount of Data

Data Extraction Tip 3: Pilot the Template

Better systematic review management, head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers: 

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic Review of the Literature: Best Practices

Affiliations.

• 1 Department of Radiology and Imaging, Rush University Medical Center, 3833, 1653 W Congress Pkwy, Chicago, IL 60612. Electronic address: [email protected].
• 2 Cardiothoracic Imaging, Department of Radiology, UT Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX, USA.
• 3 Department of Radiology, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL 32224.
• 4 Chief, Cardiothoracic and Emergency Radiology, Department of Radiology, Medical College of Wisconsin, Milwaukee, WI, USA.
• 5 Department of Diagnostic Radiology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd. Unit 1478, Houston, TX, 77030.
• 6 Department of Medical Imaging, University of Toronto, 263 McCaul Street, 4th Floor, Toronto, ON, M5T 1W7.
• 7 Department of Radiology, Medical University of South Carolina, Charleston, South Carolina, Department of Radiology and Radiological Sciences, Medical University of South Carolina, 96 Jonathan Lucas St. MSC 323, Charleston, SC 29425.
• 8 Department of Radiology and Medical Imaging, University of Virginia Medical Center, 1215 Lee St., Charlottesville, VA 22903.
• PMID: 30442379
• DOI: 10.1016/j.acra.2018.04.025

Reviews of published scientific literature are a valuable resource that can underline best practices in medicine and clarify clinical controversies. Among the various types of reviews, the systematic review of the literature is ranked as the most rigorous since it is a high-level summary of existing evidence focused on answering a precise question. Systematic reviews employ a pre-defined protocol to identify relevant and trustworthy literature. Such reviews can accomplish several critical goals that are not easily achievable with typical empirical studies by allowing identification and discussion of best evidence, contradictory findings, and gaps in the literature. The Association of University Radiologists Radiology Research Alliance Systematic Review Task Force convened to explore the methodology and practical considerations involved in performing a systematic review. This article provides a detailed and practical guide for performing a systematic review and discusses its applications in radiology.

Keywords: effective systematic review; radiology review; research methodology; systematic review.

Copyright © 2018. Published by Elsevier Inc.

PubMed Disclaimer

Similar articles

• How Systematic Review Can Shape Clinical Practice in Radiology. Kim TH, Woo S. Kim TH, et al. AJR Am J Roentgenol. 2024 Jan;222(1):e2329603. doi: 10.2214/AJR.23.29603. Epub 2023 Jul 26. AJR Am J Roentgenol. 2024. PMID: 37493323
• Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Aromataris E, et al. Int J Evid Based Healthc. 2015 Sep;13(3):132-40. doi: 10.1097/XEB.0000000000000055. Int J Evid Based Healthc. 2015. PMID: 26360830
• Cervical spondylotic myelopathy: methodological approaches to evaluate the literature and establish best evidence. Skelly AC, Hashimoto RE, Norvell DC, Dettori JR, Fischer DJ, Wilson JR, Tetreault LA, Fehlings MG. Skelly AC, et al. Spine (Phila Pa 1976). 2013 Oct 15;38(22 Suppl 1):S9-18. doi: 10.1097/BRS.0b013e3182a7ebbf. Spine (Phila Pa 1976). 2013. PMID: 24026148
• Decision and Simulation Modeling in Systematic Reviews [Internet]. Kuntz K, Sainfort F, Butler M, Taylor B, Kulasingam S, Gregory S, Mann E, Anderson JM, Kane RL. Kuntz K, et al. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Feb. Report No.: 11(13)-EHC037-EF. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Feb. Report No.: 11(13)-EHC037-EF. PMID: 23534078 Free Books & Documents. Review.
• Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Manchikanti L. Manchikanti L. Pain Physician. 2008 Mar-Apr;11(2):161-86. Pain Physician. 2008. PMID: 18354710 Review.
• Identifying and ranking employer brand improvement strategies in post-COVID 19 tourism and hospitality. Bagheri M, Baum T, Mobasheri AA, Nikbakht A. Bagheri M, et al. Tour Hosp Res. 2023 Jul;23(3):391-405. doi: 10.1177/14673584221112607. Epub 2022 Jul 4. Tour Hosp Res. 2023. PMID: 37350846 Free PMC article.
• A bibliometric analysis of systematic reviews and meta-analyses in ophthalmology. Fu Y, Mao Y, Jiang S, Luo S, Chen X, Xiao W. Fu Y, et al. Front Med (Lausanne). 2023 Mar 2;10:1135592. doi: 10.3389/fmed.2023.1135592. eCollection 2023. Front Med (Lausanne). 2023. PMID: 36936241 Free PMC article.
• Developing pre-service teachers' computational thinking: a systematic literature review. Dong W, Li Y, Sun L, Liu Y. Dong W, et al. Int J Technol Des Educ. 2023 Feb 13:1-37. doi: 10.1007/s10798-023-09811-3. Online ahead of print. Int J Technol Des Educ. 2023. PMID: 36816094 Free PMC article.
• Machine learning computational tools to assist the performance of systematic reviews: A mapping review. Cierco Jimenez R, Lee T, Rosillo N, Cordova R, Cree IA, Gonzalez A, Indave Ruiz BI. Cierco Jimenez R, et al. BMC Med Res Methodol. 2022 Dec 16;22(1):322. doi: 10.1186/s12874-022-01805-4. BMC Med Res Methodol. 2022. PMID: 36522637 Free PMC article. Review.
• Review of Microgels for Enhanced Oil Recovery: Properties and Cases of Application. Rozhkova YA, Burin DA, Galkin SV, Yang H. Rozhkova YA, et al. Gels. 2022 Feb 11;8(2):112. doi: 10.3390/gels8020112. Gels. 2022. PMID: 35200492 Free PMC article. Review.
• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Elsevier Science
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Penn State University Libraries

• Home-Articles and Databases
• Asking the clinical question
• PICO & Finding Evidence
• Evaluating the Evidence
• Systematic Review vs. Literature Review
• Ethical & Legal Issues for Nurses
• Nursing Library Instruction Course
• Data Management Toolkit This link opens in a new window
• Useful Nursing Resources
• Writing Resources
• LionSearch and Finding Articles
• The Catalog and Finding Books

Know the Difference! Systematic Review vs. Literature Review

It is common to confuse systematic and literature reviews as both are used to provide a summary of the existent literature or research on a specific topic.  Even with this common ground, both types vary significantly.  Please review the following chart (and its corresponding poster linked below) for the detailed explanation of each as well as the differences between each type of review.

• What's in a name? The difference between a Systematic Review and a Literature Review, and why it matters by Lynn Kysh, MLIS, University of Southern California - Norris Medical Library
• << Previous: Evaluating the Evidence
• Next: Ethical & Legal Issues for Nurses >>
• Last Updated: Mar 1, 2024 11:54 AM
• URL: https://guides.libraries.psu.edu/nursing

Communicative Sciences and Disorders

• Online Learners: Quick Links
• ASHA Journals
• Research Tip 1: Define the Research Question
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images/Streaming Video
• Database Tutorials
• Crafting a Search
• Cited Reference Searching
• Research Tip 4: Find Grey Literature
• Research Tip 5: Save Your Work
• Cite and Manage Your Sources
• Critical Appraisal
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Point of Care Tools (Mobile Apps)

Choosing a Review Type

For guidance related to choosing a review type, see:

• "What Type of Review is Right for You?" - Decision Tree (PDF) This decision tree, from Cornell University Library, highlights key difference between narrative, systematic, umbrella, scoping and rapid reviews.
• Reviewing the literature: choosing a review design Noble, H., & Smith, J. (2018). Reviewing the literature: Choosing a review design. Evidence Based Nursing, 21(2), 39–41. https://doi.org/10.1136/eb-2018-102895
• What synthesis methodology should I use? A review and analysis of approaches to research synthesis Schick-Makaroff, K., MacDonald, M., Plummer, M., Burgess, J., & Neander, W. (2016). What synthesis methodology should I use? A review and analysis of approaches to research synthesis. AIMS Public Health, 3 (1), 172-215. doi:10.3934/publichealth.2016.1.172 More information less... ABSTRACT: Our purpose is to present a comprehensive overview and assessment of the main approaches to research synthesis. We use "research synthesis" as a broad overarching term to describe various approaches to combining, integrating, and synthesizing research findings.
• Right Review - Decision Support Tool Not sure of the most suitable review method? Answer a few questions and be guided to suitable knowledge synthesis methods. Updated in 2022 and featured in the Journal of Clinical Epidemiology 10.1016/j.jclinepi.2022.03.004

Types of Evidence Synthesis / Literature Reviews

Literature reviews are comprehensive summaries and syntheses of the previous research on a given topic.  While narrative reviews are common across all academic disciplines, reviews that focus on appraising and synthesizing research evidence are increasingly important in the health and social sciences.  

Most evidence synthesis methods use formal and explicit methods to identify, select and combine results from multiple studies, making evidence synthesis a form of meta-research.  

The review purpose, methods used and the results produced vary among different kinds of literature reviews; some of the common types of literature review are detailed below.

Common Types of Literature Reviews 1

Narrative (literature) review.

• A broad term referring to reviews with a wide scope and non-standardized methodology
• Search strategies, comprehensiveness of literature search, time range covered and method of synthesis will vary and do not follow an established protocol

Integrative Review

• A type of literature review based on a systematic, structured literature search
• Often has a broadly defined purpose or review question
• Seeks to generate or refine and theory or hypothesis and/or develop a holistic understanding of a topic of interest
• Relies on diverse sources of data (e.g. empirical, theoretical or methodological literature; qualitative or quantitative studies)

Systematic Review

• Systematically and transparently collects and categorize existing evidence on a question of scientific, policy or management importance
• Follows a research protocol that is established a priori
• Some sub-types of systematic reviews include: SRs of intervention effectiveness, diagnosis, prognosis, etiology, qualitative evidence, economic evidence, and more.
• Time-intensive and often takes months to a year or more to complete 
• The most commonly referred to type of evidence synthesis; sometimes confused as a blanket term for other types of reviews

Meta-Analysis

• Statistical technique for combining the findings from disparate quantitative studies
• Uses statistical methods to objectively evaluate, synthesize, and summarize results
• Often conducted as part of a systematic review

Scoping Review

• Systematically and transparently collects and categorizes existing evidence on a broad question of scientific, policy or management importance
• Seeks to identify research gaps, identify key concepts and characteristics of the literature and/or examine how research is conducted on a topic of interest
• Useful when the complexity or heterogeneity of the body of literature does not lend itself to a precise systematic review
• Useful if authors do not have a single, precise review question
• May critically evaluate existing evidence, but does not attempt to synthesize the results in the way a systematic review would 
• May take longer than a systematic review

Rapid Review

• Applies a systematic review methodology within a time-constrained setting
• Employs methodological "shortcuts" (e.g., limiting search terms and the scope of the literature search), at the risk of introducing bias
• Useful for addressing issues requiring quick decisions, such as developing policy recommendations

Umbrella Review

• Reviews other systematic reviews on a topic
• Often defines a broader question than is typical of a traditional systematic review
• Most useful when there are competing interventions to consider

1. Adapted from:

Eldermire, E. (2021, November 15). A guide to evidence synthesis: Types of evidence synthesis. Cornell University LibGuides. https://guides.library.cornell.edu/evidence-synthesis/types

Nolfi, D. (2021, October 6). Integrative Review: Systematic vs. Scoping vs. Integrative. Duquesne University LibGuides. https://guides.library.duq.edu/c.php?g=1055475&p=7725920

Delaney, L. (2021, November 24). Systematic reviews: Other review types. UniSA LibGuides. https://guides.library.unisa.edu.au/SystematicReviews/OtherReviewTypes

Further Reading: Exploring Different Types of Literature Reviews

• A typology of reviews: An analysis of 14 review types and associated methodologies Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26 (2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x More information less... ABSTRACT: The expansion of evidence-based practice across sectors has lead to an increasing variety of review types. However, the diversity of terminology used means that the full potential of these review types may be lost amongst a confusion of indistinct and misapplied terms. The objective of this study is to provide descriptive insight into the most common types of reviews, with illustrative examples from health and health information domains.
• Clarifying differences between review designs and methods Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1 , 28. doi:10.1186/2046-4053-1-28 More information less... ABSTRACT: This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews....It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation.
• Are we talking the same paradigm? Considering methodological choices in health education systematic review Gordon, M. (2016). Are we talking the same paradigm? Considering methodological choices in health education systematic review. Medical Teacher, 38 (7), 746-750. doi:10.3109/0142159X.2016.1147536 More information less... ABSTRACT: Key items discussed are the positivist synthesis methods meta-analysis and content analysis to address questions in the form of "whether and what" education is effective. These can be juxtaposed with the constructivist aligned thematic analysis and meta-ethnography to address questions in the form of "why." The concept of the realist review is also considered. It is proposed that authors of such work should describe their research alignment and the link between question, alignment and evidence synthesis method selected.
• Meeting the review family: Exploring review types and associated information retrieval requirements Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information & Libraries Journal, 36(3), 202–222. doi: 10.1111/hir.12276

Integrative Reviews

"The integrative review method is an approach that allows for the inclusion of diverse methodologies (i.e. experimental and non-experimental research)." (Whittemore & Knafl, 2005, p. 547).

• The integrative review: Updated methodology Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52 (5), 546–553. doi:10.1111/j.1365-2648.2005.03621.x More information less... ABSTRACT: The aim of this paper is to distinguish the integrative review method from other review methods and to propose methodological strategies specific to the integrative review method to enhance the rigour of the process....An integrative review is a specific review method that summarizes past empirical or theoretical literature to provide a more comprehensive understanding of a particular phenomenon or healthcare problem....Well-done integrative reviews present the state of the science, contribute to theory development, and have direct applicability to practice and policy.

• Conducting integrative reviews: A guide for novice nursing researchers Dhollande, S., Taylor, A., Meyer, S., & Scott, M. (2021). Conducting integrative reviews: A guide for novice nursing researchers. Journal of Research in Nursing, 26(5), 427–438. https://doi.org/10.1177/1744987121997907
• Rigour in integrative reviews Whittemore, R. (2007). Rigour in integrative reviews. In C. Webb & B. Roe (Eds.), Reviewing Research Evidence for Nursing Practice (pp. 149–156). John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470692127.ch11

Scoping Reviews

Scoping reviews are evidence syntheses that are conducted systematically, but begin with a broader scope of question than traditional systematic reviews, allowing the research to 'map' the relevant literature on a given topic.

• Scoping studies: Towards a methodological framework Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19-32. doi:10.1080/1364557032000119616 More information less... ABSTRACT: We distinguish between different types of scoping studies and indicate where these stand in relation to full systematic reviews. We outline a framework for conducting a scoping study based on our recent experiences of reviewing the literature on services for carers for people with mental health problems.
• Scoping studies: Advancing the methodology Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 69. doi:10.1186/1748-5908-5-69 More information less... ABSTRACT: We build upon our experiences conducting three scoping studies using the Arksey and O'Malley methodology to propose recommendations that clarify and enhance each stage of the framework.
• Methodology for JBI scoping reviews Peters, M. D. J., Godfrey, C. M., McInerney, P., Baldini Soares, C., Khalil, H., & Parker, D. (2015). The Joanna Briggs Institute reviewers’ manual: Methodology for JBI scoping reviews [PDF]. Retrieved from The Joanna Briggs Institute website: http://joannabriggs.org/assets/docs/sumari/Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf More information less... ABSTRACT: Unlike other reviews that address relatively precise questions, such as a systematic review of the effectiveness of a particular intervention based on a precise set of outcomes, scoping reviews can be used to map the key concepts underpinning a research area as well as to clarify working definitions, and/or the conceptual boundaries of a topic. A scoping review may focus on one of these aims or all of them as a set.

Systematic vs. Scoping Reviews: What's the Difference? 

YouTube Video 4 minutes, 45 seconds

Rapid Reviews

Rapid reviews are systematic reviews that are undertaken under a tighter timeframe than traditional systematic reviews. 

• Evidence summaries: The evolution of a rapid review approach Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: The evolution of a rapid review approach. Systematic Reviews, 1 (1), 10. doi:10.1186/2046-4053-1-10 More information less... ABSTRACT: Rapid reviews have emerged as a streamlined approach to synthesizing evidence - typically for informing emergent decisions faced by decision makers in health care settings. Although there is growing use of rapid review "methods," and proliferation of rapid review products, there is a dearth of published literature on rapid review methodology. This paper outlines our experience with rapidly producing, publishing and disseminating evidence summaries in the context of our Knowledge to Action (KTA) research program.
• What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments Harker, J., & Kleijnen, J. (2012). What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. International Journal of Evidence‐Based Healthcare, 10 (4), 397-410. doi:10.1111/j.1744-1609.2012.00290.x More information less... ABSTRACT: In recent years, there has been an emergence of "rapid reviews" within Health Technology Assessments; however, there is no known published guidance or agreed methodology within recognised systematic review or Health Technology Assessment guidelines. In order to answer the research question "What is a rapid review and is methodology consistent in rapid reviews of Health Technology Assessments?", a study was undertaken in a sample of rapid review Health Technology Assessments from the Health Technology Assessment database within the Cochrane Library and other specialised Health Technology Assessment databases to investigate similarities and/or differences in rapid review methodology utilised.
• Rapid Review Guidebook Dobbins, M. (2017). Rapid review guidebook. Hamilton, ON: National Collaborating Centre for Methods and Tools.
• NCCMT Summary and Tool for Dobbins' Rapid Review Guidebook National Collaborating Centre for Methods and Tools. (2017). Rapid review guidebook. Hamilton, ON: McMaster University. Retrieved from http://www.nccmt.ca/knowledge-repositories/search/308
• << Previous: Literature Reviews
• Next: Conducting & Reporting Systematic Reviews >>
• Last Updated: Jun 26, 2024 3:00 PM
• URL: https://guides.nyu.edu/speech

How to Conduct a Literature Review: A Guide for Graduate Students

• Let's Get Started!

Traditional or Narrative Reviews

• Systematic Reviews
• Typology of Reviews
• Literature Review Resources
• Developing a Search Strategy
• What Literature to Search
• Where to Search: Indexes and Databases
• Finding articles: Libkey Nomad
• Finding Dissertations and Theses
• Extending Your Searching with Citation Chains
• Forward Citation Chains - Cited Reference Searching
• Keeping up with the Literature
• Managing Your References
• Need More Information?

A narrative or traditional literature review is a comprehensive, critical and objective analysis of the current knowledge on a topic. They are an essential part of the research process and help to establish a theoretical framework and focus or context for your research. A literature review will help you to identify patterns and trends in the literature so that you can identify gaps or inconsistencies in a body of knowledge. This should lead you to a sufficiently focused research question that justifies your research.

Onwuegbuzie and Frels (pp 24-25, 2016) define four common types of narrative reviews:

• General literature review that provides a review of the most important and critical aspects of the current knowledge of the topic. This general literature review forms the introduction to a thesis or dissertation and must be defined by the research objective, underlying hypothesis or problem or the reviewer's argumentative thesis.
• Historical literature review which focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.
• Methodological literature review where the research methods and design are described. These methodological reviews outline the strengths and weaknesses of the methods used and provide future direction
• Theoretical literature review which examines how theory shapes or frames research

References and additional resources

Machi, Lawrence A. & Brenda T. McEvoy (2016), The Literature Review: Six steps to success . 3rd edition.; Thousand Oaks, CA: Corwin. Onwuegbuzie, A. J. & Frels, R. (2016) 7 steps to a comprehensive literature review: A multimodal & cultural approach . London: Sage Publications.

• << Previous: Types of Literature Reviews
• Next: Systematic Reviews >>

The library's collections and services are available to all ISU students, faculty, and staff and Parks Library is open to the public .

• Last Updated: Mar 14, 2024 12:15 PM
• URL: https://instr.iastate.libguides.com/gradlitrev

• Directories

Purpose of traditional literature reviews

Literature reviews come in many shapes and sizes, and may be placed in different areas in different theses. However, for most projects, the literature review usually has one overall purpose: to provide a rationale for your research in terms of what has gone before.

The purpose of a traditional literature review is to demonstrate a gap or problem in your field that your research seeks to address. The importance of addressing the gap for your field or discipline must not be assumed but persuasively demonstrated. Indeed, explaining why there is a need for filling the gap helps you to justify your work's value, originality and significance.

To establish your credibility as a scholar, your literature review will typically need to do at least some (if not all) of the following effectively.

• Demonstrate that your research is rigorous and up-to-date by enagaging with seminal and current work
• Summarise relevant bodies of work and evaluate their strengths and weaknesses to demonstrate your critical understanding of the literature.
• Point out gaps in the literature or identify problems / issues remaining to be solved.
• Justify the choice of topic/area of research, to establish the value of the research and the nature of the contribution you expect to make to the research field.
• Highlight key issues essential to your own research.
• Synthesise the main themes and arguments of a particular body of literature.
• Develop an argument in the process of reviewing the literature
• Produce a brief historical survey or other context information in order to situate your research.

The literature review helps you to establish the nature of your contribution to knoweldge. The type of contribution differs from project to project. The list below gives a variety of ways in which different projects can contribute to different fields. Does your project fit into any of these categories?

• Filling a significant gap in the research (could be almost anything).
• Challenging prevailing methodologies, theories or test procedures.
• Continuing an ongoing line of enquiry (common in the experimental sciences).
• Developing new experimental procedures or perhaps a model.
• Developing new products, software or technology (perhaps for industry).
• Using established techniques or approaches in a new context (common in area studies).
• Providing extensive new data (common in field research).
• Producing a portfolio of creative work with an accompanying exegesis.
• Presenting studio work or performance presentations, as in some arts.
• Writing a biography (perhaps in History or English).
• Setting up a comprehensive practical study for professional development purposes.
• Doing a comprehensive analysis of issues in new, and important, policy proposals.
• Documenting a language (perhaps in Linguistics).

Whichever type of contribution you're making, your literature review should establish the need for your contribution.

Placement of traditional literature reviews>>

Literature reviews

Placement of traditional literature reviews

Structuring a literature review

Language of the literature review

Systematic literature reviews

Grounded theory literature reviews

• ANU Library Academic Skills
• +61 2 6125 2972

• Advanced Search

Shedding light on the dark side – A systematic literature review of the issues in agile software development methodology use

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, recommendations, software process improvement in agile software development a systematic literature review.

It is recognized the relevance and importance that software process improvement (SPI) and agile development have gained in the field of software engineering. Both are approaches that increase the efficiency and effectiveness of a software development ...

Systematic literature review on agile practices in global software development

Developing software in distributed development environments exhibits coordination, control and communication challenges. Agile practices, which demand frequent communication and self-organization between remote sites, are increasingly ...

Challenges in Combining Agile Development and CMMI: A Systematic Literature Review

Recently, Agile Development has emerged as an alternative approach in software engineering. The Agile Software Development (ASD) process provides the ability to cope with ever-changing requirements. On the other hand, the Capability Maturity Model ...

Information

Published in.

Elsevier Science Inc.

United States

Publication History

Author tags.

• Agile software development methodologies
• Issues in ASD
• Dark side of ASD
• Information systems development agility
• Systematic Literature Review
• Research-article

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Digital Transformation and Flexible Performance Management: A Systematic Literature Review of the Evolution of Performance Measurement Systems

• REVIEW PAPERS
• Open access
• Published: 01 July 2024

Cite this article

You have full access to this open access article

• Marcello Cosa   ORCID: orcid.org/0000-0001-9554-9837 1 &
• Riccardo Torelli   ORCID: orcid.org/0000-0003-1956-3981 1  

51 Accesses

Explore all metrics

In the era of digital transformation, businesses must innovate and adapt to sustain a competitive edge. This dynamic environment compels a reevaluation of traditional management practices, highlighting the need for highly flexible systems. Flexibility, defined as the ability to adapt organizational resources, processes, and strategies in response to environmental changes such as rapid technological advancements, is crucial. Our systematic review of 47 studies investigates how digital transformation influences performance measurement systems across various industries and global contexts. We found that digital transformation fosters the dynamism and adaptability of these systems. This study integrates strategic, organizational, and information systems flexibility concepts that are essential for effective adaptation and resilience. Our findings underscore the shifts towards decision-making agility, inclusivity, and sustainability, stressing the significant role of human resources in adapting to digital imperatives. We advocate for a comprehensive approach that fosters digital literacy, upholds ethical standards, promotes continuous skill development, and enhances strategic adaptability. Practical implications suggest integrating digital technologies into performance strategies, utilizing real-time metrics for agile decision-making and emphasizing ethical and sustainable practices to improve transparency and stakeholder trust. These strategies are crucial for optimizing performance in the digital age.

Similar content being viewed by others

Dynamic capabilities and firm performance: the rise and fall of Charles Schwab

How management support systems affect job performance: a systematic literature review and research agenda

Barriers to Organizations to Adopt Digital Transformation for Driving Eco-Innovation and Sustainable Performance

Avoid common mistakes on your manuscript.

Introduction

Digital transformation (DT) is rapidly reshaping industries, requiring businesses to innovate and adapt quickly to remain competitive and meet evolving stakeholder demands (Alnoor et al., 2024 ; D’Adamo et al., 2023a , 2023b ). This environment challenges traditional performance measurement systems (PMSs), which often fail to fully leverage the benefits of emerging technologies, such as artificial intelligence (AI), big data, and the Internet of Things (IoT) (Aldoseri et al., 2024 ; Sardi et al., 2023 ). In response, flexible management has become essential, enabling organizations to adapt their operational, strategic, and measurement practices in real time to foster a culture of continuous improvement (Chowdhury et al., 2024 ; Enrique et al., 2022 ; Gao & Chen, 2021 ).

Flexibility is crucial for aligning with strategic objectives, responding to changes, and enhancing sustainability. For example, supply chain and tourism companies exemplify the rapid adaptation to new business models and unforeseen challenges (Agrawal et al., 2024 ; Pandey et al., 2024 ; Singh et al., 2023 ). Indeed, the evolving role of flexibility increasingly contributes to organizational and environmental adaptability (Singh et al., 2021 ).

The DT-driven evolution of business models, methods, and customer experiences necessitates a comprehensive understanding of how these changes impact PMSs (Sakhteh et al., 2023 ). Technical, cultural, organizational, and relational shifts underscore DTs’ role in enhancing performance and creating customer value (Mergel et al., 2019 ).

However, the broader implications for organizational flexibility and strategic alignment remain underexplored (Korsen & Ingvaldsen, 2022 ), as existing systematic reviews focus narrowly on specific technological impacts without considering pervasive organizational effects. For instance, Yadav and colleagues’ study ( 2022 ) within the agricultural food supply chain highlighted the need for PMSs that incorporate sustainability, spurred by rapid digitalization. Hidalgo Martins et al. ( 2022 ) noted challenges in performance measurement for SMEs in the manufacturing industry. Additionally, Miklosik and Evans ( 2020 ) addressed the issue of information overload in marketing organizations, a complication arising from disorganized data from digital sources. These studies detailed the technological integration within PMSs, yet seldom addressed the holistic transformation of organizational strategies.

Our study addresses this gap by examining how DT necessitates realignment within organizations, fostering a more interconnected and responsive business environment. We explore how DT enhances the flexibility and effectiveness of PMSs across diverse organizational contexts (Gong & Ribiere, 2021 ). Therefore, we propose the following research questions:

RQ1: How does DT influence PMSs in terms of organizational flexibility? This question seeks a deeper understanding of the relationship between DT and PMSs beyond technical aspects (Nadkarni & Prügl, 2021 ).

RQ2: How do digital technologies affect decision-making within PMSs? This inquiry is critical to obtain insights into maintaining innovation and agility in a fast-paced digital economy (Kamble & Gunasekaran, 2020 ).

RQ3: How do traditional measurement methods adapt in the digital era? Exploring this topic is imperative, as the pace of digital advancements threatens the relevance of conventional PMS tools, necessitating their evolution to accurately capture firm-created value (Bansal et al., 2023 ).

We conducted the first systematic review with a thematic analysis of the role of PMSs in digitally transformed environments, integrating 47 peer-reviewed articles. This analysis reveals the fundamental functions of PMSs, such as monitoring, attention focusing, strategic decision-making, and legitimacy (Henri, 2006 ), and provides a unified view that advances the understanding of modern organizations’ responsiveness to ongoing digitalizing environments.

Following this introduction, Section “ Methodology ” details our review methodology, describing the selection criteria and analysis techniques used to comprehensively examine the literature. Section “ Results ” presents our thematic findings, and Section “ Discussion ” explores the impact of DT on flexibility in decision-making within PMSs, linking to our research questions. Section “ Contributions and Implications ” concludes the paper by providing theoretical and practical implications and outlining future research directions, emphasizing the need for PMS alignment with digital advancements to boost organizational effectiveness.

Methodology

We conducted a systematic review adhering to the PRISMA guidelines of Liberati et al. ( 2009 ). This approach enhances research quality and reliability by offering a comprehensive, unbiased synthesis of both published and unpublished literature. By systematically identifying, evaluating, and integrating studies based on predefined criteria, our review ensures thorough coverage of the topic, promoting the reproducibility of findings and deepening understanding in this research area (Popay et al., 2006 ; Tranfield et al., 2003 ). Such rigour is crucial for identifying knowledge gaps and directing future research (Webster & Watson, 2002 ). It supports evidence-based practice by providing clear, synthesized outcomes that aid decision-making for practitioners and policy-makers (Schardt et al., 2007 ).

Conceptual Framework

Figure  1 illustrates our adopted conceptual framework, as defined by Henri ( 2006 ), which categorizes the flexible roles and capabilities of PMSs into four types: monitoring, attention focusing, strategic decision-making, and legitimization. Each category enhances organizational agility to adapt to DT.

Source: Adapted in part from Henri ( 2006 )

Conceptual framework: Uses of PMSs.

Henri’s framework is noted for its thorough examination of performance measurement complexities and has been successfully applied in fundamental studies (DeNisi & Smith, 2014 ; Franco-Santos et al., 2012 ; Ukko et al., 2019 ). Its focus on flexibility aligns with our examination of how DT reshapes PMSs to support more adaptive and dynamic organizational environments.

By applying Henri’s structured approach, we categorize the literature into four thematic areas:

Monitoring involves setting goals and using them as flexible diagnostic tools that adjust to new data and changing conditions. It is crucial for tracking progress and ensuring that firm performance aligns with stakeholder expectations.

Attention focusing enables leaders to dynamically highlight and communicate organizational priorities and critical success factors, fostering adaptability to strategic objectives.

Strategic decision-making assists in formulating long-term, adaptive decisions by providing insights into the evolving dynamics within organizational processes and facilitating strategic planning and execution.

Legitimization ensures that PMSs rationalize past decisions under changing conditions and bolster future planning efforts. Having flexible and accountable demonstrations enhances organizational credibility and secures societal support.

Henri’s framework integrates seamlessly with our research questions, coherently presenting findings and illustrating the dynamic interplay between PMSs and DT. It effectively addresses the identified gaps in the literature by offering nuanced insights into how organizations can leverage PMSs in digitally evolving business landscapes. Thus, this study provides a new perspective on the relationship between digital technologies and performance management tools crucial for strategic decision-making in modern organizational contexts.

Search Strategy

On August 7, 2023, we searched the Web of Science, Scopus, and ProQuest databases, focusing on DT and performance measurement. This search adhered to methodologies from previously validated systematic reviews, limiting inclusion to peer-reviewed studies published in international journals and excluding conference papers and book chapters.

For DT, we utilized search terms from recent literature (Gurzhii et al., 2022 ; Hanelt et al., 2021 ; Verhoef et al., 2021 ; Zhu et al., 2021 ): “digital transformation”, “digital strategy”, “digital disruption”, “digital business strategy”, “digitalize”, “digitize”, “IT transformation”, “IS transformation”, “business transformation”, and “emerging technologies.”

In terms of performance measurement, we derived keyword variants from earlier systematic reviews (Bititci et al., 2012 ; Franco-Santos et al., 2012 ; Rojas-Lema et al., 2021 ), including “performance measurement system*”, “performance measure*”, “management control*”, “performance measurement”, “performance management”, “performance indicators”, “strategic control”, “performance evaluation”, and “performance assessment.” To address the multidisciplinary nature of our study, we expanded our search to include terms such as “organi* performance”, “firm performance”, and “SME performance.”

We employed the wildcard “*” to capture plural forms and variants in our search terms. Table 1 outlines the search strings used for each database and the results obtained.

Our expansive keyword approach aimed to overcome the limitations of keyword-centric searches, acknowledging the lack of universally accepted definitions for DT and performance measurement (Vial, 2021 ).

We performed our searches across titles, abstracts, and keywords. The initial search yielded 3109 articles, from which we removed 930 duplicates using Zotero 6.0.27. The remaining 2179 articles were screened for eligibility and focused on adaptability, agility, and resilience, which are themes relevant to the impacts of DT on PMSs.

Inclusion/Exclusion Criteria

The next stage involved screening papers using the Population, Intervention, Comparison, Outcome, Time (PICOT) framework, as suggested by Echevarria and Walker ( 2014 ). Table 2 details the inclusion and exclusion criteria, providing clear guidelines for our systematic review process.

We adopted a focused selection strategy to ensure that our research on DT’s impact on PMSs was relevant and specific. Following Franco-Santos et al. ( 2007 ), we concentrated on studies with a precise unit of analysis, aiming for clear and in-depth research outcomes. Therefore, we included only peer-reviewed articles in English that were strictly related to PMSs rather than to performance measurement in general. This approach allowed us to delve deeply into how DT influences PMSs specifically. We also excluded studies assessing the performance and metrics for the different phases of DT, as these areas have been extensively reviewed elsewhere (Ochoa-Urrego & Peña-Reyes, 2021 ; Teichert, 2019 ).

The exclusion of studies from the public and nonprofit sectors was intentional due to their unique measurement standards and challenges. The public sector is subject to diverse, legally mandated measurement standards that vary significantly across different national contexts (Speklé & Verbeeten, 2014 ), introducing variables that could confound the analysis of DT’s impact on PMSs. Similarly, the nonprofit sector’s nascent stage in adopting PMSs (Treinta et al., 2020 ) suggests that its inclusion might not offer the mature perspective necessary for our investigation.

We set our timeframe for the included studies from January 2000 to the present, following Verhoef et al. ( 2021 ). This period is significant because the internet bubble burst when tech giants such as Google, Amazon, and eBay not only survived but also began significantly shaping our understanding of DT. We did not restrict our search to journal rankings or research fields to maintain broad coverage across disciplines.

After screening the titles and abstracts, 2,132 papers were excluded, leaving 804 for full-text review. Ultimately, 47 papers met our criteria and were included in our systematic review. Figure  2 depicts the PRISMA flowchart of our screening process.

PRISMA flowchart for screening and inclusion

Our stringent selection criteria might limit the scope of the study. However, this specificity is crucial for ensuring the integrity and applicability of our findings, particularly regarding DT’s impact on PMSs within business organizations. Our focused approach strengthens the foundation for future research and enhances the precision and relevance of our contributions to discussions on DTs and PMSs.

Data Extraction

We extracted essential information from each paper, including title, authors, abstract, and publication year. To mitigate potential biases, we also gathered detailed data, such as country of origin, research questions, study design, sample size, demographic information, and main findings.

We employed thematic analysis to systematically categorize and interpret the data. This method is particularly effective for exploring varied research questions, from subjective experiences to objective performance assessments (Clarke & Braun, 2013 ). Analysing the data this way provided deeper insights into the underlying themes and patterns that emerged.

Our focus was on PMS roles (monitoring, attention focusing, strategic decision-making, and legitimization), guided by Henri’s ( 2006 ) conceptual framework, which links specific PMS functions to their capabilities, as observed in the literature (Pinheiro de Lima et al., 2008 ).

Adhering to the PRISMA checklist and applying stringent selection criteria ensured that our review was comprehensive and sharply focused. This meticulous approach enhances the credibility of our findings and supports their applicability across diverse contexts. Section “ Results ” will delve into how these findings illustrate PMSs’ adaptive responses to DT. We explore significant themes, such as the strategic implications of these adaptations across various industries, demonstrating the practical impact of digitalization on performance management practices.

Overview of Results

Our final sample included 47 studies, as detailed in Table  3 . The methodologies used varied and included quantitative (26 studies), qualitative (3), mixed-method (3), conceptual (6), and case studies (9). Geographically, the studies were conducted across Europe (14), Asia (12), the Americas (4), multiple countries (4), and Africa (1). Six articles did not specify a location, and the six conceptual papers inherently lacked geographical data.

Figure  3 illustrates the temporal distribution of the studies. There has been a noticeable increase in publications, with over 85% of publications released since 2019, indicating a growing academic interest in this area.

The number of studies published per year. Notes In 2023, we found 16 studies published in Web of Science, Scopus, and ProQuest as of 07/08/2023

The majority of the articles focused on strategic decision-making. Only four studies explored attention focusing, a critical element in digitally transforming environments. Figure  4 illustrates the distribution based on Henri’s ( 2006 ) categories.

Roles of PMSs in included articles

In the following sections, we will further analyse each category to understand the impact of DT on PMSs, decision-making processes, and the adaptation of traditional tools within digital contexts.

Monitoring ( n  = 15)

Fifteen studies focused on observing and assessing organizational activities using PMSs. These studies emphasized the importance of tracking performance metrics to align operations with strategic objectives and promptly detect deviations, underscoring the critical role of monitoring in flexible management.

We identified five clusters within this theme:

Digital tools and techniques

Methodological approaches

Context of digitalization

Challenges and opportunities

Illustrative case studies

Digital Tools and Techniques ( n  = 4)

Four studies highlighted how innovative digital tools designed for monitoring are revolutionizing industry practices. For instance, Ahmad and Qiu ( 2009 ) utilized a comprehensive dataset covering 1500 firms from 1993 to 2005 to develop an integrated model for manufacturing SMEs. This model underscores the critical role of human resources in technology adoption, especially amid widespread talent scarcity. This insight emphasizes the necessity of human capital in maximizing the benefits of digital tools. Similarly, studies by Bonci et al. ( 2019 ), Litavniece et al. ( 2023 ), and Fischer et al. ( 2021 ) demonstrated how integrating computer algorithms with physical processes enhances efficiency and sustainability across various sectors. These studies indicate that effective use of digital tools depends on integrating skilled human resources.

Methodological Approaches ( n  = 3)

Three studies explored structured methodologies for digital monitoring, offering a broader perspective on the applications of such tools. Aibinu and Papadonikolaki ( 2020 ) expanded the utility of building information modelling by introducing an “effort distribution analysis” methodology. This approach aims to enhance organizational learning and innovation, illustrating the potential of structured methods to foster significant advancements in company practices. In Hong Kong, Ng et al. ( 2017 ) highlighted how adaptable performance analysis methodologies incorporating financial and nonfinancial indicators are crucial for innovative firms. Papiorek and Hiebl ( 2023 ) further supported this view by demonstrating the importance of high-quality information for effective management control systems, requiring robust IT capabilities and external expertise. These insights underscore that flexible performance analysis requires high-quality information systems. Companies must integrate these elements skilfully to optimize results, highlighting the value of technological innovation and methodological adaptability.

Context of Digitalization ( n  = 4)

This cluster examines the interface between digital transformation and performance tracking. Studies such as those by Scalco and Simske ( 2023 ) and AlMujaini et al. ( 2021 ) revealed that successful DT involves more than merely implementing new technologies. It requires a strategic alignment integrating technology with insightful human management and organizational vision. Similarly, AL-Khatib ( 2022 ) demonstrated that intellectual capital coupled with big data analytics (BDA) significantly enhances innovation performance in 333 Jordanian banks. In addition, Park et al. ( 2022 ) advocated for global collaboration to leverage disruptive technologies for improved sustainability and energy efficiency in their conceptual study on appliance and equipment systems. These findings challenge traditional views of technology adoption, advocating for a more nuanced approach that leverages human insights alongside digital advancements.

Challenges and Opportunities ( n  = 2)

The digital era introduces challenges and opportunities for monitoring, as seen in studies by Baral et al. ( 2023 ) and Globerson ( 2024 ). Baral et al. ( 2023 ) highlighted how the COVID-19 pandemic underscored the vulnerability of global supply chains, prompting SMEs to develop resilient strategic plans. This adaptation involves not only automation but also a comprehensive rethinking of performance monitoring systems to ensure resilience and real-time adaptability (Globerson, 2024 ). These studies illustrate that the digital era reshaped performance monitoring paradigms. Digital disruptions and global uncertainties highlight the vulnerabilities of traditional PMSs. The emerging digital landscape demands not only automation but also strategic rethinking of performance monitoring, ensuring resilience and real-time flexibility.

Illustrative Case Studies ( n  = 2)

Case studies by Chhabra et al. ( 2022 ) and Quille et al. ( 2023 ) provided practical insights into how digital tools can transform traditional monitoring practices. These studies showcased the application of the IoT, BDA, global positioning system, and robotic process automation in enhancing monitoring efficiency, sustainability, and customer satisfaction. They exemplified how leveraging cutting-edge technologies can revolutionize traditional practices, offering a blueprint for future innovations in performance monitoring. Together, these case studies underscore the evolving nature of monitoring in digitalization. By leveraging such emerging technologies, businesses can dramatically boost operational flexibility, sustainability, and customer satisfaction.

Attention Focusing ( n  = 4)

Four studies explored the mechanisms organizations and individuals use to prioritize specific areas, issues, or metrics crucial for swiftly addressing vital aspects. These studies range in scope from broad organizational strategies to targeted tactical actions. Strategic considerations set the overarching corporate direction, while tactical measures focus on immediate, actionable steps.

Therefore, we classified the articles as follows:

Digital strategy shifts

Tactical performance signals

Digital Strategy Shifts ( n  = 2)

Exploring the transition from traditional to digital marketing, Homburg and Wielgos ( 2022 ) analysed responses from 382 German-speaking senior managers and financial data from 273 global companies. Their findings emphasize the necessity of a customer-centric approach and internal processes aligned with this perspective, essential for maintaining relevance in the face of rapid technological changes. Similarly, Reinking et al. ( 2020 ) interviewed 27 managers across various industries to examine how visual performance measurement tools influence managerial focus on specific metrics. Their research introduced the concept of “strategy surrogation”, where managers may prefer simpler tactics over complex strategies to align decisions with broader goals. This approach was critical for effectively diffusing corporate strategy, with PMSs providing real-time feedback that positively influences managerial behaviour. Both studies highlight that DT requires a strategic reorientation. Organizations should enhance their PMSs to focus on key areas such as employee digital marketing skills and strategic internal knowledge dissemination.

Tactical Performance Signals ( n  = 2)

The role of performance appraisal systems in signalling organizational priorities was the focus of a study by Curzi and colleagues ( 2019 ), which surveyed 865 employees from multinational firms in Italy. Their findings suggest that appraisals aimed at specific performance outcomes or new competencies can foster innovative work behaviour. However, they noted that overly formalized systems, such as standardized yearly evaluations, may inhibit innovation. In a related study, Čizmić and Ahmić ( 2021 ) explored the impact of robust human resource practices on organizational success by studying 97 managers in Bosnia and Herzegovina. Their research showed that talent identification and skills development are crucial for boosting profitability and sales growth, emphasizing the significance of tactical measures in steering organizational direction. These studies reveal that while balanced PMSs can drive innovation and effectively signal organizational priorities, excessive formalization in appraisal systems might stifle creativity, underscoring the need for a delicate balance between individual autonomy and strategic alignment.

Strategic Decision-Making ( n  = 21)

Twenty-one studies investigate how PMSs and performance metrics guide strategic decisions and align with organizational directions, priorities, and visions. We categorized the papers into three distinct clusters:

DT in business strategies

Technological tools in decision-making

Strategic AI and digital shifts

DT in Business Strategies ( n  = 9)

Nine studies examined how DT influences strategic business decisions. The significance of digital tools and a skilled workforce was highlighted by Teng and colleagues ( 2022 ) in their study of 335 Chinese SMEs. Research on Finnish SMEs offered contradictory insights. Holopainen et al. ( 2023 ) observed that technological understanding alone does not increase PMS usage. However, Joensuu-Salo and Matalamäki ( 2023 ) found that mastery of digital technologies significantly boosts performance and growth. This dichotomy underscores the diverse effects of technology on strategy depending on contextual factors. Further supporting the critical role of technology, studies by Moumtzidis et al. ( 2022 ), Hung et al. ( 2023 ), and Opazo-Basáez et al. ( 2023 ) documented the positive impacts of BDA, IoT, and cloud platforms on strategic decision-making, enhancing production efficiency and customer satisfaction.

From an innovation perspective, Trequattrini et al. ( 2022 ) conducted a case study of Soundreef S.p.A., an Italian copyright management company, showing how technologies bolster PMS transparency and accuracy. Adding services to products (servitization) and digitalization interact to create value in 828 Spanish firms (Martín-Peña et al., 2020 ). However, despite recent advances, many executives still rely on outdated indicators (Wengler et al., 2021 ).

In sum, while digital technologies are reshaping business strategies, a dichotomy exists in the perceived utility of PMS usage. Firms must align technologies with updated and relevant performance metrics to gain a competitive advantage.

Technological Decision-Making Tools ( n  = 8)

This cluster focuses on strategically integrating AI and other technological tools across various industries. From an operations management lens, Ng ( 2009 ) analysed the strategic advantage of R&D activities in 12 US technology companies, showing how investment in intangible assets boosts market value. Transitioning to optimization tools, Moretti and Re Cecconi ( 2019 ) introduced a decision support system (DSS) applied to an Italian office building to predict maintenance needs and optimize operations. Building on performance measurement frameworks, Wang and Chien ( 2016 ) employed the balanced scorecard approach in 23 Taiwanese LED companies, proposing the integration of financial and nonfinancial indicators to gain a holistic view of performance.

In supply chain management, Szymczak et al. ( 2018 ) reported the cautious adoption of new technologies such as cloud computing and data mining among 200 Polish companies, while El Kihel et al. ( 2023 ) highlighted the transformative potential of BDA and AI in Stellantis car manufacturing operations in Morocco. Nudurupati et al. ( 2021 ) noted an evolution in performance measures over 17 months, incorporating broader value-creation networks.

Finally, Bititci ( 2007 ) and Meagher ( 2002 ) proposed conceptual frameworks emphasizing the seamless integration of leadership, strategy, processes, and performance metrics for business evolution, advocating a shift to evidence-based decision-making over mere intuition.

These studies underscore that PMSs streamline resource allocation and refine decision-making across industries. Embracing a comprehensive approach can enable organizations to make more informed decisions.

Strategic AI and Digital Shifts ( n  = 4)

Four studies specifically emphasized the transformative impact of AI on business operations. Joshi and colleagues ( 2022 ) surveyed 881 global firms and introduced the concept of the IT governance process capability. It refers to a company’s ability to choose the right tech resources, make decisions, plan, update systems, deliver services, and monitor them effectively. The authors found that such capability significantly enhances technological and financial performance. Building on this technological momentum, Olan et al. ( 2022 ) demonstrated the synergistic benefits of AI integration with knowledge-sharing tools, noting improvements in organizational efficiency.

Diving deeper into AI’s potential, Samarghandi et al. ( 2023 ) applied deep learning techniques in an Iranian audit organization to predict human actions in an accounting information system (AIS), identifying key predictors of effective AIS usage. Finally, Colombo and Beuren ( 2023 ) surveyed 298 employees in a Brazilian shared services centre and found that an innovation culture and an interactive PMS significantly boost accounting process automation.

These studies emphasise that AI, data mining, and cloud systems are transforming strategic decision-making. Effective implementation requires strong governance, expert human oversight, and a proactive approach to technological innovation.

Legitimizing ( n  = 7)

This section reviews seven studies that analyse how organizations leverage PMSs to enhance credibility and justify decisions within societal norms and expectations. We divided these articles into two primary clusters:

Stakeholder engagement and legitimacy

Ethics and sustainability management

Stakeholder Engagement and Legitimacy ( n  = 2)

Two articles explored the impact of DT on organizational performance and legitimization. As part of their study on incorporating stakeholder feedback in organizational decision-making, Hristov and Appolloni ( 2022 ) conducted semistructured interviews with 183 managers, surveyed 637 stakeholders from 61 Italian organizations, and analysed internal reports. Their findings underscore the importance of incorporating stakeholders’ insights into the decision-making process, identifying four key integration dimensions: sustainable development, organizational drivers, digital transformation, and cultural context. According to Vărzaru’s ( 2022 ) study, using BDA and cloud computing significantly improves sustainability reporting across 21 European Union countries. This enhances transparency in communicating sustainable development strategies to stakeholders. These studies illustrate that strategic stakeholder engagement through advanced technologies can boost organizational legitimacy and performance.

Ethics and Sustainability Management ( n  = 5)

Five studies underscored how digital capabilities influence corporate sustainability practices and performance metrics. Shin et al. ( 2023 ) demonstrated that leadership proficient in digital technologies can significantly enhance corporate performance in South Korea, with a supportive technology adoption culture and digital skills among employees amplifying this effect. In the same country, the study by Kim ( 2021 ) indicated industry-specific variations in factors driving sustainable growth among SMEs in the IT/software sectors, emphasizing the impact of business technology skills. A study of 319 Indian SMEs by Vrontis and colleagues ( 2022 ) highlighted how digital tools such as social media apps, AI, BDA, the IoT, and blockchain contribute to economic growth and societal benefits.

To further emphasize the role of technology in sustainability, Lavorato and Piedepalumbo ( 2023 ) presented a case study of an innovative Italian high-tech startup, illustrating how smart technologies such as the IoT and cloud solutions enhance sustainability measures and align with sustainable development goals. Nandi and associates ( 2023 ) proposed a CE performance measurement model that integrates digital technologies with alternative pricing valuation methods, enabling firms to effectively assess sustainability performance and CE benefits.

The reviewed studies affirm that tailored digital tools are crucial for enhancing corporate sustainability and ethical management. They drive SME growth and align business practices with broader societal values, underscoring the critical role of digital expertise in achieving sustainable development goals. The integration of digital solutions into PMSs not only boosts performance but also significantly contributes to societal progress and sustainable development.

This study explored the intricate relationship between DTs and PMSs across various industries and global regions. We aimed to address critical gaps in the literature, specifically the underexplored dynamics of how DT enhances PMS dynamism and adaptability, the influence of digital technologies on decision-making processes within these systems, and the evolution of traditional PMS tools in response to digital advancements.

Impact of Digital Transformation on PMSs

In addressing the first research question, we found that DT profoundly impacts PMSs by enhancing their operational dynamics through increased adaptability, agility, and resilience. These flexible management practices, enabled by digital tools such as AI and big data, deepen the integration of technology with human resources, which is essential for effective operation. This integration entails adopting new technologies and transforming decision-making cultures within organizations to be more dynamic and responsive.

Digital technologies facilitate new operational capabilities and transform organizational decision-making cultures to be more dynamic and responsive. The necessity for skilled human intervention underscores that while technology extends capabilities, human oversight ensures strategic alignment. Methods such as building information modelling (BIM) illustrate this synergy by merging advanced tools with human expertise to boost performance (Aibinu & Papadonikolaki, 2020 ; Ng et al., 2017 ; Papiorek & Hiebl, 2023 ). Moreover, the volatile digital era demands a synthesized approach integrating technology, human factors, and organizational strategy (AL-Khatib, 2022 ; Park et al., 2022 ).

Contrasting studies have shed light on the varying impacts of DT on PMS usage. While Teng et al. ( 2022 ) and Joensuu-Salo and Matalamäki ( 2023 ) emphasized the significance of managerial digital literacy, Holopainen et al. ( 2023 ) found no direct correlation between technological awareness and PMS application, suggesting that the benefits of digital tools may not be universally perceived. This can occur in sectors characterized by low competition (Soto Setzke et al., 2023 ). This diversity of findings enriches our understanding of digital tool integration across different competitive landscapes.

Our findings confirm that digitally enabled PMSs significantly enhance institutional flexibility across various sectors. For instance, using AI and BDA in the commercial agriculture industry facilitates real-time performance adjustments, supporting decisions responsive to changing market and environmental conditions (Abeysiriwardana et al., 2022 ). This capability is equally valuable in health care and manufacturing, where real-time data support enhances patient care and optimizes operational flexibility and responsiveness (Brandín & Abrishami, 2024 ; Dogra et al., 2023 ).

As we consider the enhanced operational dynamics facilitated by DT, it is also crucial to explore how these technologies specifically augment decision-making flexibility within organizations, a point we examine in the following section.

Flexibility Factors in PMSs and Organizational Decision-Making

Regarding the second research question, our investigation reveals that digital technologies fundamentally enhance the flexibility of decision-making processes within PMSs, democratizing and enriching this crucial organizational function. By implementing digital tools that enable dynamic and real-time metrics, PMSs have evolved from static, rigid systems into adaptable, responsive frameworks that facilitate participatory and inclusive decision-making processes (Lavorato & Piedepalumbo, 2023 ). This shift not only entails incorporating new tools but also transforming the decision-making culture within organizations (Shukla & Shankar, 2024 ).

Digital tools such as AI-driven analytics platforms enable systems to quickly integrate new information and adapt outputs to meet changing conditions, showcasing adaptability in sectors such as construction and manufacturing (Moretti & Re Cecconi, 2019 ; Szymczak et al., 2018 ). Furthermore, decision support systems leveraging the IoT and big data provide instant insights into operational efficiency, facilitating rapid responses to logistical or supply chain challenges (Joshi et al., 2022 ; Olan et al., 2022 ).

Resilience, another critical aspect of PMSs, involves maintaining functionality and quickly recovering from setbacks. Technologies such as cloud-based PMSs ensure data integrity and availability across multiple geographies, safeguarding against localized failures (Hung et al., 2023 ; Opazo-Basáez et al., 2023 ). This is crucial in industries such as health care, where downtime can have severe repercussions (Sharma et al., 2023 ).

Our findings align with seminal studies that underscore managerial cognition’s role in dynamically and creatively interpreting performance metrics (Ittner & Larcker, 2003 ; Malmi, 2001 ). These digital tools enable a shift towards more inclusive and innovative decision-making processes, as demonstrated by the strategic value of R&D activities guided by KPIs continually refined by AI and BDA (Moretti & Re Cecconi, 2019 ; Ng, 2009 ). Integrating these technologies enhances decision-making and underscores the importance of governance and human expertise in effectively leveraging these technologies (Chen et al., 2012 ; Kar et al., 2023 ).

After exploring how digital tools enhance both the adaptability and resilience of PMSs, we now focus on how traditional PMSs have undergone significant transformations due to digital technologies.

Evolution of Traditional PMSs in Digitally Transformed Settings

Responding to the third research question, we observe a significant evolution in the capabilities of traditional PMSs driven by the integration of advanced technologies such as AI, BDA, and the IoT. The shift from periodic, retrospective analysis to continuous, real-time monitoring has enhanced the accuracy of performance metrics and the speed of organizational response.

Real-time dashboards, AI-enhanced forecasting tools, and blockchain technology have each played a role in advancing PMS capabilities, aligning them more closely with modern organizational needs and stakeholder expectations (El Kihel et al., 2023 ; Fischer et al., 2021 ; Hristov & Appolloni, 2022 ; Nandi et al., 2023 ). These advancements underscore the transformative impact of DT on traditional systems, enhancing its utility and strategic value.

DT’s potential to foster sustainable communities is significant. By leveraging emerging technologies, organizations can enhance operational efficiency and effectively manage their environmental impacts (Feroz et al., 2021 ). This approach ensures compliance with environmental regulations and supports broader sustainability goals that benefit entire communities (Ciasullo et al., 2024 ). For instance, digitally enabled PMSs track resource consumption and energy efficiency, providing insights that can help reduce ecological footprints (Latifah & Soewarno, 2023 ). Integrating these systems into public sector initiatives amplifies their impact, contributing to more resilient community infrastructures (Ayoko, 2021 ).

This strategic alignment between digital advancements and community development emphasizes transforming technological capabilities into tangible societal benefits (Joy et al., 2023 ). Studies such as D’Adamo et al., ( 2023a , 2023b ) show how photovoltaic systems optimize energy consumption within community frameworks, enhancing local sustainability efforts. Additionally, higher education institutions play a crucial role in promoting sustainability through community engagement projects (Biancardi et al., 2023 ).

Henri’s ( 2006 ) framework remains relevant. However, our findings extend these principles by emphasizing the role of real-time analytics and digital tools in optimizing monitoring effectiveness. Our review also highlights the continuing relevance of human resources, particularly in SMEs where talent scarcity poses significant challenges (Ahmad & Qiu., 2009 ). The complexities introduced by digital tools necessitate an approach valorizing human capabilities. DT’s influence transcends mere tool adaptation, reshaping organizational attention management. We finally identified an evolved legitimizing role. Modern PMSs now serve as strategic assets, driving ethical sustainability (Kim, 2021 ; Shin et al., 2023 ), thus enhancing their legitimizing function in the DT context. By synthesizing these findings in alignment with our research questions, we see a clear trajectory of how digital technologies have intricately and profoundly reshaped PMSs, influencing their design, functionalities, and objectives.

Contributions and Implications

Theoretical contributions.

Our study enhances the understanding of PMSs by illustrating their evolution from traditional “rationalization machines” (Henri, 2006 , p. 81) to strategic assets within organizations (Nandi et al., 2023 ; Vrontis et al., 2022 ). This transition reflects a significant shift in PMS conceptualization, aligning with the principles of flexible systems management by integrating adaptability, strategic flexibility, and resilience into their core functions (Nayal et al., 2024 ).

We identified a symbiotic interplay between DTs and PMSs. Figure  5 illustrates this dynamic enrichment process, emphasizing how DT enhances PMS comprehensiveness and dynamism. This interaction underlines the importance of flexible management in increasing organizational resilience and adaptability during digital transitions, prompting a reevaluation of the discourse on the coevolution of DT and PMSs in modern organizations.

The ‘symbiotic interplay’ between DT and PMSs

Furthermore, our research broadens the application of flexible systems management by incorporating ethical and sustainable decision-making metrics into our PMS analysis. This contributes to the ongoing discourse on performance metrics (Kim, 2021 ; Shin et al., 2023 ; Vrontis et al., 2022 ) and challenges the prevailing narratives that overly emphasize technology at the expense of human involvement.

We emphasize the importance of integrating human expertise and technology to achieve the benefits of flexible systems management, highlighting that harmonization is essential for effective management flexibility. The interplay between technological capabilities and human resources marks a crucial expansion of flexible systems management. Insights from human resource management and information systems are vital to fully leverage the potential of PMSs in the digital age, suggesting a model where technology and human resource strategies are cohesively aligned.

Practical Implications and Policy Directions

Our findings offer actionable insights for managers, policy-makers, and organizations navigating the digital landscape. We propose several practical recommendations based on our thematic analysis to integrate DT into PMS strategies effectively:

Integrating DT into PMS Strategies : Organizations should view digital technologies as integral rather than supplementary components of performance measurement frameworks. By embedding these technologies directly into PMS strategies, organizations can adopt a more dynamic approach to performance measurement. Continuous training initiatives ensure that personnel develop the necessary digital literacy to utilize complex performance metrics effectively, thus empowering employees to leverage digital tools (Joensuu-Salo & Matalamäki, 2023 ; Teng et al., 2022 ).

Embracing Real-Time Metrics : With the increasing importance of timely data in decision-making, organizations must transition from traditional periodic reviews to dynamic, data-driven approaches. Investing in real-time analytics technologies and cultivating a culture that values data interpretation skills are essential. This shift enhances organizational agility and responsiveness by enabling quicker reactions to market changes and internal challenges (Fischer et al., 2021 ; Szymczak et al., 2018 ).

Prioritizing Ethical and Sustainable Measures : As sustainability becomes a critical performance indicator, organizations should employ digital tools to enhance the transparency of their sustainability efforts. Such transparency aids compliance with environmental standards and bolsters stakeholder trust and organizational credibility. Digital platforms that facilitate detailed tracking and reporting of sustainability metrics enable organizations to effectively communicate their efforts and impacts (Kim, 2021 ; Vrontis et al., 2022 ).

Continuous Training and Development : Adopting new technologies necessitates an ongoing commitment to training and development. Establishing continuous learning environments ensures that organizations remain current with technological advancements and that their workforce is proficient in the latest digital tools (Aibinu & Papadonikolaki, 2020 ). This commitment is vital for maintaining effective PMSs and for enabling employees to adapt to new tools and strategies as they emerge. A holistic approach that merges DTs’ technological capabilities with human resource expertise is essential for organizations aiming to enhance their performance in today’s digital era. The strategic governance of technological changes will position organizations to make informed, proactive decisions in a turbulent marketplace rather than engage in passive compliance (Cosa et al., 2024 ).

Policy Innovations for AI-driven PMSs : Integrating AI-driven PMSs with policy innovations is crucial for maintaining high integrity and aligning with evolving regulations. Implementing these systems requires policies that support continuous adaptation and learning, facilitate the seamless integration of new technologies into existing frameworks, and encourage the development of skills necessary to manage and optimize these systems. Such policies support a sustainable transition to more intelligent and responsive organizational practices.

Limitations

Our systematic review has inherent limitations. First, our focus was strictly on articles that addressed PMS as the unit of analysis, excluding studies that discussed performance measurement more broadly. This selection criterion limited our final sample to 47 papers. Although this number may appear small, this focused approach was intended to maintain sharp relevance to our research objectives. This methodological rigour allows for an in-depth exploration of PMS-specific insights, providing a targeted understanding of how DT reshapes PMSs despite potentially missing broader insights from the general performance measurement literature.

Second, the diverse nature of the articles posed classification challenges. We adhered to Henri’s ( 2006 ) framework for categorization and followed Massaro et al.’s ( 2016 ) recommendation to prioritize each paper’s most prominent research focus. This approach helped maintain clarity and coherence, even as we adopted a narrative style to highlight the complex web of research angles, enriching our discussion across multiple dimensions (Popay et al., 2006 ).

A third limitation involves our terminology. Despite subtle distinctions, we used “digital transformation” and “digital technologies” interchangeably. DT refers to integrating digital technologies across all business areas, while digital technologies are specific tools, systems, devices, and resources (Berman, 2012 ). We made this choice for coherence and clarity, yet it is important to recognize these terms’ nuances when interpreting our findings.

Additionally, we excluded studies that focused on metrics for different DT phases and did not include articles from the public or nonprofit sectors due to their unique measurement dynamics. Our temporal scope, focusing on post-2000 publications, aimed to capture insights from an era shaped by the survival of tech giants post-Internet bubble burst, possibly omitting foundational insights from earlier periods. Finally, our decision to include all studies, regardless of journal rankings or research fields, aimed to broaden the perspectives considered, counterbalancing our other exclusion criteria.

Future Research

Our study highlights significant opportunities for further investigation into the evolving landscape of digitalization and PMSs. Detailed research is needed on the specific advantages, challenges, and strategies for integrating individual digital tools in various industrial contexts. Such research could explore the most effective tools in the digital era and whether organizations are adopting innovative frameworks.

The rapid evolution of DT underscores the necessity for a comprehensive research agenda focused on emerging trends in performance measurement. This agenda would pinpoint critical areas for exploration, offering practitioners insights into the field’s trajectory and helping them adapt strategically.

Surprisingly, the attention-focusing role of PMSs is underrepresented in the literature despite the growing emphasis on employee well-being (Pradhan & Hati, 2022 ; Rasool et al., 2021 ). Future studies could investigate how a well-defined PMS can guide employees on what to prioritize, thereby reducing cognitive overload, uncertainty, and stress. Exploring the interplay between corporate wellness initiatives and performance measurement could yield valuable insights for enhancing organizational health in the digital era.

Another area for future exploration is human resistance and the acceptance of digital tools. Discrepancies in findings on managers’ technological understandings and their impact on PMS usage suggest that factors such as organizational culture, existing infrastructure, or the specific design and utility of PMS tools themselves might play more significant roles than previously thought (Holopainen et al., 2023 ; Joensuu-Salo & Matalamäki, 2023 ; Teng et al., 2022 ). A more nuanced exploration, potentially integrating qualitative methodologies, is necessary to fully understand these underlying dynamics.

Finally, several studies have addressed SMEs’ unique challenges and opportunities related to DT and performance measurement (Ahmad & Qiu, 2009 ; AlMujaini et al., 2021 ; Baral et al., 2023 ; Kim, 2021 ; Vrontis et al., 2022 ). Future research could focus specifically on the types of digital tools SMEs prioritize, the challenges they encounter in integrating these tools, and how their strategies differ from or converge with those of larger organizations. Such studies could help tailor performance management strategies to the needs of SMEs, fostering more effective and sustainable practices.

Key Questions for Further Reflection

In what ways can PMSs contribute to enhancing organizational resilience and sustainability in the digital age?

What role do human resources play in optimizing digital tools within PMSs to enhance organizational performance?

How can organizations balance the need for technological innovation with ethical standards and sustainability practices in their PMS frameworks?

Abeysiriwardana, P. C., Jayasinghe-Mudalige, U. K., Kodituwakku, S. R., & Madhushani, K. B. (2022). Intelligently driven performance management: an enabler of real-time research forecasting for innovative commercial agriculture. SN Social Sciences . https://doi.org/10.1007/s43545-022-00484-8

Article   Google Scholar  

Agrawal, N., Sharma, M., Raut, R. D., Mangla, S. K., & Arisian, S. (2024). Supply chain flexibility and post-pandemic Resilience. Global Journal of Flexible Systems Management, 24 (Suppl 1), 119-138. https://doi.org/10.1007/s40171-024-00375-2

Ahmad, N., & Qiu, R. G. (2009). Integrated model of operations effectiveness of small to medium-sized manufacturing enterprises. Journal of Intelligent Manufacturing, 20 , 79–89. https://doi.org/10.1007/s10845-008-0105-5

Aibinu, A. A., & Papadonikolaki, E. (2020). Conceptualizing and operationalizing team task interdependences: BIM implementation assessment using effort distribution analytics. Construction Management and Economics, 38 (5), 420–446. https://doi.org/10.1080/01446193.2019.1623409

Aldoseri, A., Al-Khalifa, K. N., & Hamouda, A. M. (2024). Methodological approach to assessing the current state of organizations for AI-based digital transformation. Applied System Innovation, 7 (1), 14. https://doi.org/10.3390/asi7010014

Al-Khatib, A. W. (2022). Intellectual capital and innovation performance: The moderating role of big data analytics: Evidence from the banking sector in Jordan. EuroMed Journal of Business, 17 (3), 391–423. https://doi.org/10.1108/EMJB-10-2021-0154

AlMujaini, H., Hilmi, M., Abudaqa, A., & Alzahmi, R. (2021). Corporate foresight organizational learning and performance: The moderating role of digital transformation and mediating role of innovativeness in SMEs. International Journal of Data and Network Science, 5 (4), 703–712. https://doi.org/10.5267/j.ijdns.2021.7.011

Alnoor, A., Atiyah, A. G., & Abbas, S. (2024). Unveiling the determinants of digital strategy from the perspective of entrepreneurial orientation theory: A two-stage SEM-ANN approach. Global Journal of Flexible Systems Management, 25 (2), 243–260. https://doi.org/10.1007/s40171-024-00385-0

Ayoko, O. B. (2021). Digital transformation, robotics, artificial intelligence, and innovation. Journal of Management & Organization, 27 (5), 831–835. https://doi.org/10.1017/jmo.2021.64

Bansal, A., Panchal, T., Jabeen, F., Mangla, S. K., & Singh, G. (2023). A study of human resource digital transformation (HRDT): A phenomenon of innovation capability led by digital and individual factors. Journal of Business Research, 157 , 113611. https://doi.org/10.1016/j.jbusres.2022.113611

Baral, M. M., Singh, R. K., & Kazançoğlu, Y. (2023). Analysis of factors impacting survivability of sustainable supply chain during COVID-19 pandemic: An empirical study in the context of SMEs. The International Journal of Logistics Management, 34 (4), 935–961. https://doi.org/10.1108/IJLM-04-2021-0198

Berman, S. J. (2012). Digital transformation: Opportunities to create new business models. Strategy & Leadership, 40 (2), 16–24. https://doi.org/10.1108/10878571211209314

Biancardi, A., Colasante, A., D’Adamo, I., Daraio, C., Gastaldi, M., & Uricchio, A. F. (2023). Strategies for developing sustainable communities in higher education institutions. Scientific Reports, 13 (1), 20596. https://doi.org/10.1038/s41598-023-48021-8

Bititci, U. S. (2007). An executive’s guide to business transformation. Business Strategy Series, 8 (3), 203–213. https://doi.org/10.1108/17515630710684204

Bititci, U. S., Garengo, P., Dörfler, V., & Nudurupati, S. (2012). Performance measurement: Challenges for tomorrow. International Journal of Management Reviews, 14 (3), 305–327. https://doi.org/10.1111/j.1468-2370.2011.00318.x

Bonci, A., Carbonari, A., Cucchiarelli, A., Messi, L., Pirani, M., & Vaccarini, M. (2019). A cyber-physical system approach for building efficiency monitoring. Automation in Construction, 102 , 68–85. https://doi.org/10.1016/j.autcon.2019.02.010

Brandín, R., & Abrishami, S. (2024). IoT-BIM and blockchain integration for enhanced data traceability in offsite manufacturing. Automation in Construction, 159 , 105266. https://doi.org/10.1016/j.autcon.2024.105266

Chen, H., Chiang, R. H., & Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS Quarterly, 36 (4), 1165–1188. https://doi.org/10.2307/41703503

Chhabra, D., Singh, R. K., & Kumar, V. (2022). Developing IT-enabled performance monitoring system for green logistics: A case study. International Journal of Productivity and Performance Management, 71 (3), 775–789. https://doi.org/10.1108/IJPPM-12-2020-0678

Chowdhury, M. M. H., Paul, S. K., Khan, E. A., & Shakil Mahmud, A. K. M. (2024). A decision support model for barriers and optimal strategy design in sustainable humanitarian supply chain management. Global Journal of Flexible Systems Management . https://doi.org/10.1007/s40171-024-00394-z

Ciasullo, M. V., Chiarini, A., & Palumbo, R. (2024). Mastering the interplay of organizational resilience and sustainability: Insights from a hybrid literature review. Business Strategy and the Environment, 33 (2), 1418–1446. https://doi.org/10.1002/bse.3530

Čizmić, E., & Ahmić, A. (2021). The influence of talent management on organisational performance in Bosnia & Herzegovina as a developing country. Management: Journal of Contemporary Management Issues, 26 (1), 129–147. https://doi.org/10.30924/mjcmi.26.1.8

Clarke, V., & Braun, V. (2013). Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning. The Psychologist, 26 (2), 120–123.

Google Scholar  

Colombo, V. L. B., & Beuren, I. M. (2023). Accountants robots in shared service centers: Effects of the culture for innovation, work engagement and performance measurement system. Journal of Business & Industrial Marketing, 38 (12), 2760–2771. https://doi.org/10.1108/JBIM-09-2022-0436

Cosa, M., Pedro, E., & Urban, B. (2024). How to assess the intellectual capital of firms in uncertain times: A systematic literature review and a proposed model for practical adoption. Journal of Intellectual Capital, 25 (7), 1–22. https://doi.org/10.1108/JIC-05-2023-0096

Curzi, Y., Fabbri, T., Scapolan, A. C., & Boscolo, S. (2019). Performance appraisal and innovative behavior in the digital era. Frontiers in Psychology, 10 , 1659. https://doi.org/10.3389/fpsyg.2019.01659

D’Adamo, I., Gastaldi, M., Piccioni, J., & Rosa, P. (2023a). The role of automotive flexibility in supporting the diffusion of sustainable mobility initiatives: A stakeholder attitudes assessment. Global Journal of Flexible Systems Management, 24 (3), 459–481. https://doi.org/10.1007/s40171-023-00349-w

D’Adamo, I., Mammetti, M., Ottaviani, D., & Ozturk, I. (2023b). Photovoltaic systems and sustainable communities: New social models for ecological transition. The impact of incentive policies in profitability analyses. Renewable Energy, 202 , 1291–1304. https://doi.org/10.1016/j.renene.2022.11.127

DeNisi, A., & Smith, C. E. (2014). Performance appraisal, performance management, and firm-level performance: A review, a proposed model, and new directions for future research. Academy of Management Annals, 8 (1), 127–179. https://doi.org/10.5465/19416520.2014.873178

Dogra, V., Thanoch, A., Rai, C. P., Kant, K., & Rumky, J. K. (2023). Real-Time health monitoring and management: leveraging the power of IoT and machine learning. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Kirtipur, Nepal (pp. 7–11). IEEE. https://doi.org/10.1109/I-SMAC58438.2023.10290596

Echevarria, I. M., & Walker, S. (2014). To make your case, start with a PICOT question. Nursing, 44 (2), 18–19. https://doi.org/10.1097/01.NURSE.0000442594.00242.f9

El Kihel, Y., Zouggar Amrani, A., Ducq, Y., Amegouz, D., & Lfakir, A. (2023). Methodology combining industry 4.0 technologies and KPI’s reliability for supply chain performance. International Journal of Computer Integrated Manufacturing, 36 (8), 1128–1152. https://doi.org/10.1080/0951192X.2022.2162605

Enrique, D. V., Lerman, L. V., de Sousa, P. R., Benitez, G. B., Santos, F. M. B. C., & Frank, A. G. (2022). Being digital and flexible to navigate the storm: How digital transformation enhances supply chain flexibility in turbulent environments. International Journal of Production Economics, 250 , 108668. https://doi.org/10.1016/j.ijpe.2022.108668

Feroz, A. K., Zo, H., & Chiravuri, A. (2021). Digital transformation and environmental sustainability: A review and research agenda. Sustainability, 13 (3), 1530. https://doi.org/10.3390/su13031530

Fischer, M., Hofmann, A., Imgrund, F., Janiesch, C., & Winkelmann, A. (2021). On the composition of the long tail of business processes: Implications from a process mining study. Information Systems, 97 , 101689. https://doi.org/10.1016/j.is.2020.101689

Franco-Santos, M., Kennerley, M., Micheli, P., Martinez, V., Mason, S., Marr, B., Gray, D., & Neely, A. (2007). Towards a definition of a business performance measurement system. International Journal of Operations & Production Management, 27 (8), 784–801. https://doi.org/10.1108/01443570710763778

Franco-Santos, M., Lucianetti, L., & Bourne, M. (2012). Contemporary performance measurement systems: A review of their consequences and a framework for research. Management Accounting Research, 23 (2), 79–119. https://doi.org/10.1016/j.mar.2012.04.001

Gao, B., & Chen, Z. (2021). Research on the strategy of flexible management in the operation and management of small and medium-sized enterprises under computer application. In Journal of Physics: Conference Series (Vol. 1744, p. 032046). https://doi.org/10.1088/1742-6596/1744/3/032046

Globerson, S. (2024). Revising a performance measurement and control system of an organization. International Journal of Organizational Analysis, 32 (3), 397–404. https://doi.org/10.1108/IJOA-01-2023-3593

Gong, C., & Ribiere, V. (2021). Developing a unified definition of digital transformation. Technovation, 102 , 102217. https://doi.org/10.1016/j.technovation.2020.102217

Gurzhii, A., Islam, A. N., Haque, A. B., & Marella, V. (2022). Blockchain enabled digital transformation: A systematic literature review. IEEE Access, 10 , 79584–79605. https://doi.org/10.1109/ACCESS.2022.3194004

Hanelt, A., Bohnsack, R., Marz, D., & Antunes Marante, C. (2021). A systematic review of the literature on digital transformation: Insights and implications for strategy and organizational change. Journal of Management Studies, 58 (5), 1159–1197. https://doi.org/10.1111/joms.12639

Henri, J. F. (2006). Organizational culture and performance measurement systems. Accounting, Organizations and Society, 31 (1), 77–103. https://doi.org/10.1016/j.aos.2004.10.003

Hidalgo Martins, G., Deschamps, F., Pereira Detro, S., & Valle, P. D. (2022). Performance measurement based on machines data: Systematic literature review. IET Collaborative Intelligent Manufacturing, 4 (2), 74–86. https://doi.org/10.1049/cim2.12051

Holopainen, M., Saunila, M., & Ukko, J. (2023). Facilitating performance measurement and management through digital business strategy. Measuring Business Excellence, 27 (2), 246–260. https://doi.org/10.1108/MBE-01-2022-0015

Homburg, C., & Wielgos, D. M. (2022). The value relevance of digital marketing capabilities to firm performance. Journal of the Academy of Marketing Science, 50 (4), 666–688. https://doi.org/10.1007/s11747-022-00858-7

Hristov, I., & Appolloni, A. (2022). Stakeholders’ engagement in the business strategy as a key driver to increase companies’ performance: Evidence from managerial and stakeholders’ practices. Business Strategy and the Environment, 31 (4), 1488–1503. https://doi.org/10.1002/bse.2965

Hung, B. Q., Hoa, T. A., Hoai, T. T., & Nguyen, N. P. (2023). Advancement of cloud-based accounting effectiveness, decision-making quality, and firm performance through digital transformation and digital leadership: Empirical evidence from Vietnam. Heliyon, 9 (6), e16929. https://doi.org/10.1016/j.heliyon.2023.e16929

Ittner, C. D., & Larcker, D. F. (2003). Coming up short on nonfinancial performance measurement. Harvard Business Review, 81 (11), 88–95.

Joensuu-Salo, S., & Matalamäki, M. (2023). The impact of digital capability on firm performance and growth in incumbent SMEs. Journal of Enterprising Culture, 31 (2), 1–22. https://doi.org/10.1142/S0218495823500073

Joshi, A., Benitez, J., Huygh, T., Ruiz, L., & De Haes, S. (2022). Impact of IT governance process capability on business performance: Theory and empirical evidence. Decision Support Systems, 153 , 113668. https://doi.org/10.1016/j.dss.2021.113668

Joy, A., Roberts, J., Grohmann, B., & Peña, C. (2023). Confronting climate crisis through corporate narratives: The fairy tale in LVMH’s 2020 and 2021 social and environmental responsibility reports. Luxury, 10 (1–2), 1–38. https://doi.org/10.1080/20511817.2023.2280321

Kamble, S. S., & Gunasekaran, A. (2020). Big data-driven supply chain performance measurement system: A review and framework for implementation. International Journal of Production Research, 58 (1), 65–86. https://doi.org/10.1080/00207543.2019.1630770

Kar, A. K., Varsha, P. S., & Rajan, S. (2023). Unravelling the impact of generative artificial intelligence (GAI) in industrial applications: A review of scientific and grey literature. Global Journal of Flexible Systems Management, 24 (4), 659–689. https://doi.org/10.1007/s40171-023-00356-x

Kim, S. S. (2021). Sustainable growth variables by industry sectors and their influence on changes in business models of SMEs in the era of digital transformation. Sustainability, 13 (13), 7114. https://doi.org/10.3390/su13137114

Korsen, E. B. H., & Ingvaldsen, J. A. (2022). Digitalisation and the performance measurement and management system: Reinforcing empowerment. International Journal of Productivity and Performance Management, 71 (4), 1059–1075. https://doi.org/10.1108/IJPPM-09-2020-0488

Latifah, S. W., & Soewarno, N. (2023). The environmental accounting strategy and waste management to achieve MSME’s sustainability performance. Cogent Business & Management, 10 (1), 2176444. https://doi.org/10.1080/23311975.2023.2176444

Lavorato, D., & Piedepalumbo, P. (2023). How smart technologies affect the decision-making and control system of food and beverage companies—a case study. Sustainability, 15 (5), 4292. https://doi.org/10.3390/su15054292

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of Internal Medicine, 151 (4), W65–W94. https://doi.org/10.7326/0003-4819-151-4-200908180-00136

Litavniece, L., Kodors, S., Adamoniene, R., & Kijasko, J. (2023). Digital twin: An approach to enhancing tourism competitiveness. Worldwide Hospitality and Tourism Themes, 15 (5), 538–548. https://doi.org/10.1108/WHATT-06-2023-0074

Malmi, T. (2001). Balanced scorecards in Finnish companies: A research note. Management Accounting Research, 12 (2), 207–220. https://doi.org/10.1006/mare.2000.0154

Martín-Peña, M. L., Sánchez-López, J. M., & Díaz-Garrido, E. (2020). Servitization and digitalization in manufacturing: The influence on firm performance. Journal of Business & Industrial Marketing, 35 (3), 564–574. https://doi.org/10.1108/JBIM-12-2018-0400

Massaro, M., Dumay, J., & Guthrie, J. (2016). On the shoulders of giants: Undertaking a structured literature review in accounting. Accounting, Auditing & Accountability Journal, 29 (5), 767–801. https://doi.org/10.1108/AAAJ-01-2015-1939

Meagher, R. (2002). The IM building blocks. Information Management, 36 (1), 26–34.

Mergel, I., Edelmann, N., & Haug, N. (2019). Defining digital transformation: Results from expert interviews. Government Information Quarterly, 36 (4), 101385. https://doi.org/10.1016/j.giq.2019.06.002

Miklosik, A., & Evans, N. (2020). Impact of big data and machine learning on digital transformation in marketing: A literature review. IEEE Access, 8 , 101284–101292. https://doi.org/10.1109/ACCESS.2020.2998754

Moretti, N., & Re Cecconi, F. (2019). A cross-domain decision support system to optimize building maintenance. Buildings, 9 (7), 161. https://doi.org/10.3390/buildings9070161

Moumtzidis, I., Kamariotou, M., & Kitsios, F. (2022). Digital transformation strategies enabled by internet of things and big data analytics: The use-case of telecommunication companies in Greece. Information, 13 (4), 196. https://doi.org/10.3390/info13040196

Nadkarni, S., & Prügl, R. (2021). Digital transformation: A review, synthesis and opportunities for future research. Management Review Quarterly, 71 , 233–341. https://doi.org/10.1007/s11301-020-00185-7

Nandi, S., Hervani, A. A., Helms, M. M., & Sarkis, J. (2023). Conceptualising Circular economy performance with non-traditional valuation methods: Lessons for a post-Pandemic recovery. International Journal of Logistics Research and Applications, 26 (6), 662–682. https://doi.org/10.1080/13675567.2021.1974365

Nayal, K., Raut, R. D., Kumar, M., Paul, S. K., & Narkhede, B. E. (2024). Role of artificial intelligence capability in the interrelation between manufacturing strategies and operational resilience. Global Journal of Flexible Systems Management, 25 (1), 137–162. https://doi.org/10.1007/s40171-023-00367-8

Ng, A. W. C. (2009). Intellectual capital for organic growth: Comparative analysis of two technology sectors. International Journal of Intelligent Enterprise, 1 (2), 156–176. https://doi.org/10.1504/IJIE.2009.024409

Ng, A. W., Wang, W. M., Cheung, B. C., Ma, R., & Or, Y. Y. (2017). Cluster-based performance measurement system for emerging technology-based ventures. International Journal of Entrepreneurship and Innovation Management, 21 (6), 485–508. https://doi.org/10.1504/IJEIM.2017.086939

Nudurupati, S. S., Garengo, P., & Bititci, U. S. (2021). Impact of the changing business environment on performance measurement and management practices. International Journal of Production Economics, 232 , 107942. https://doi.org/10.1016/j.ijpe.2020.107942

Ochoa-Urrego, R. L., & Peña-Reyes, J. I. (2021). Digital maturity models: A systematic literature review. In D. R. A. Schallmo & J. Tidd (Eds.), Digitalization. Management for professionals (pp. 71–85). Springer.

Olan, F., Arakpogun, E. O., Suklan, J., Nakpodia, F., Damij, N., & Jayawickrama, U. (2022). Artificial intelligence and knowledge sharing: Contributing factors to organizational performance. Journal of Business Research, 145 , 605–615. https://doi.org/10.1016/j.jbusres.2022.03.008

Opazo-Basáez, M., Vendrell-Herrero, F., Bustinza, O. F., Vaillant, Y., & Marić, J. (2023). Is digital transformation equally attractive to all manufacturers? Contextualizing the operational and customer benefits of smart manufacturing. International Journal of Physical Distribution & Logistics Management, 53 (4), 489–511. https://doi.org/10.1108/IJPDLM-12-2021-0538

Pandey, A. K., Daultani, Y., Pratap, S., Ip, A. W., & Zhou, F. (2024). Analyzing Industry 4.0 adoption enablers for supply chain flexibility: Impacts on resilience and sustainability. Global Journal of Flexible Systems Management . https://doi.org/10.1007/s40171-024-00396-x

Papiorek, K. L., & Hiebl, M. R. (2023). Information systems quality in management accounting and management control effectiveness. Journal of Accounting & Organizational Change, 20 (3), 433–458. https://doi.org/10.1108/JAOC-09-2022-0148

Park, W. Y., Shah, N., Shiraishi, K., & Vine, E. (2022). Improving energy performance metrics to maximize the benefits of disruptive technologies. Energy Research & Social Science, 89 , 102678. https://doi.org/10.1016/j.erss.2022.102678

Pinheiro de Lima, E., Gouvea da Costa, S. E., & Angelis, J. J. (2008). The strategic management of operations system performance. International Journal of Business Performance Management, 10 (1), 108–132. https://doi.org/10.1504/IJBPM.2008.015924

Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N., Roen, K., & Duffy, S. (2006). Guidance on the Conduct of Narrative Synthesis in Systematic Reviews . A Product from the ESRC Methods Programme, Version 1. Retrieved March 5, 2024, from https://www.lancaster.ac.uk/media/lancaster-university/content-assets/documents/fhm/dhr/chir/NSsynthesisguidanceVersion1-April2006.pdf

Pradhan, R. K., & Hati, L. (2022). The measurement of employee well-being: Development and validation of a scale. Global Business Review, 23 (2), 385–407. https://doi.org/10.1177/0972150919859101

Quille, R. V. E., Almeida, F. V. D., Borycz, J., Corrêa, P. L. P., Filgueiras, L. V. L., Machicao, J., Almeida, G. M. D., Midorikawa, E. T., Demuner, V. R. D. S., Bedoya, J. A. R., & Vajgel, B. (2023). Performance analysis method for robotic process automation. Sustainability, 15 (4), 3702. https://doi.org/10.3390/su15043702

Rasool, S. F., Wang, M., Tang, M., Saeed, A., & Iqbal, J. (2021). How toxic workplace environment effects the employee engagement: The mediating role of organizational support and employee wellbeing. International Journal of Environmental Research and Public Health, 18 (5), 2294. https://doi.org/10.3390/ijerph18052294

Reinking, J., Arnold, V., & Sutton, S. G. (2020). Synthesizing enterprise data to strategically align performance: The intentionality of strategy surrogation. International Journal of Accounting Information Systems, 36 , 100444. https://doi.org/10.1016/j.accinf.2019.100444

Rojas-Lema, X., Alfaro-Saiz, J. J., Rodríguez-Rodríguez, R., & Verdecho, M. J. (2021). Performance measurement in SMEs: Systematic literature review and research directions. Total Quality Management & Business Excellence, 32 (15–16), 1803–1828. https://doi.org/10.1080/14783363.2020.1774357

Sakhteh, S., Mohammadi, N., & Karimi, A. (2023). Factors affecting open innovation in digital entrepreneurship in Iran and the world. Global Journal of Flexible Systems Management, 25 (1), 63-80. https://doi.org/10.1007/s40171-023-00363-y

Samarghandi, H., Askarany, D., & Dehkordi, B. B. (2023). A hybrid method to predict human action actors in accounting information system. Journal of Risk and Financial Management, 16 (1), 37. https://doi.org/10.3390/jrfm16010037

Sardi, A., Sorano, E., Cantino, V., & Garengo, P. (2023). Big data and performance measurement research: Trends, evolution and future opportunities. Measuring Business Excellence, 27 (4), 531–548. https://doi.org/10.1108/MBE-06-2019-0053

Scalco, A., & Simske, S. (2023). A model for measuring multi-concern assurance of critical infrastructure control systems. Systems Engineering, 26 (6), 742–753. https://doi.org/10.1002/sys.21684

Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007). Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Medical Informatics and Decision Making, 7 (16), 1–6. https://doi.org/10.1186/1472-6947-7-16

Sharma, D. K., Chakravarthi, D. S., Shaikh, A. A., Ahmed, A. A. A., Jaiswal, S., & Naved, M. (2023). The aspect of vast data management problem in healthcare sector and implementation of cloud computing technique. Materials Today: Proceedings, 80 , 3805–3810. https://doi.org/10.1016/j.matpr.2021.07.388

Shin, J., Mollah, M. A., & Choi, J. (2023). Sustainability and organizational performance in South Korea: The effect of digital leadership on digital culture and employees’ digital capabilities. Sustainability, 15 (3), 2027. https://doi.org/10.3390/su15032027

Shukla, M., & Shankar, R. (2024). Impact Assessment of Smart Manufacturing System Implementation in Small and Medium Enterprises: Moderating Role of Enabling Technology and Government Support. Global Journal of Flexible Systems Management . https://doi.org/10.1007/s40171-024-00400-4

Singh, A., Sushil, & Sharma, H. K. (2023). Total Interpretive Structural Modeling-Polarity (TISM-P) to analyze the impact of energy on the sustainability performance of hotels: a case study. Environment, Development and Sustainability . https://doi.org/10.1007/s10668-023-03485-6

Singh, S., Dhir, S., Evans, S., & Sushil. (2021). The trajectory of two decades of global journal of flexible systems management and flexibility research: A bibliometric analysis. Global Journal of Flexible Systems Management, 22 (4), 377–401. https://doi.org/10.1007/s40171-021-00286-6

Soto Setzke, D., Riasanow, T., Böhm, M., & Krcmar, H. (2023). Pathways to digital service innovation: The role of digital transformation strategies in established organizations. Information Systems Frontiers, 25 (3), 1017–1037. https://doi.org/10.1007/s10796-021-10112-0

Speklé, R. F., & Verbeeten, F. H. (2014). The use of performance measurement systems in the public sector: Effects on performance. Management Accounting Research, 25 (2), 131–146. https://doi.org/10.1016/j.mar.2013.07.004

Szymczak, M., Ryciuk, U., Leończuk, D., Piotrowicz, W., Witkowski, K., Nazarko, J., & Jakuszewicz, J. (2018). Key factors for information integration in the supply chain–measurement, technology and information characteristics. Journal of Business Economics and Management, 19 (5), 759–776. https://doi.org/10.3846/jbem.2018.6359

Teichert, R. (2019). Digital transformation maturity: A systematic review of literature. Acta Universitatis Agriculturae Et Silviculturae Mendelianae Brunensis, 67 (6), 1673–1687. https://doi.org/10.11118/actaun201967061673

Teng, X., Wu, Z., & Yang, F. (2022). Research on the relationship between digital transformation and performance of SMEs. Sustainability, 14 (10), 6012. https://doi.org/10.3390/su14106012

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375

Treinta, F. T., Moura, L. F., Almeida Prado Cestari, J. M., Pinheiro de Lima, E., Deschamps, F., Gouvea da Costa, S. E., Van Aken, E. M., Munik, J., & Leite, L. R. (2020). Design and implementation factors for performance measurement in non-profit organizations: A literature review. Frontiers in Psychology, 11 , 1799. https://doi.org/10.3389/fpsyg.2020.01799

Trequattrini, R., Lardo, A., Cuozzo, B., & Manfredi, S. (2022). Intangible assets management and digital transformation: Evidence from intellectual property rights-intensive industries. Meditari Accountancy Research, 30 (4), 989–1006. https://doi.org/10.1108/MEDAR-03-2021-1216

Ukko, J., Nasiri, M., Saunila, M., & Rantala, T. (2019). Sustainability strategy as a moderator in the relationship between digital business strategy and financial performance. Journal of Cleaner Production, 236 , 117626. https://doi.org/10.1016/j.jclepro.2019.117626

Vărzaru, A. A. (2022). An empirical framework for assessment of the effects of digital technologies on sustainability accounting and reporting in the European Union. Electronics, 11 (22), 3812. https://doi.org/10.3390/electronics11223812

Verhoef, P. C., Broekhuizen, T., Bart, Y., Bhattacharya, A., Dong, J. Q., Fabian, N., & Haenlein, M. (2021). Digital transformation: A multidisciplinary reflection and research agenda. Journal of Business Research, 122 , 889–901. https://doi.org/10.1016/j.jbusres.2019.09.022

Vial, G. (2021). Understanding digital transformation: A review and research agenda. In A. Hinterhuber, T. Vescovi, & F. Checchinato (Eds.), Managing digital transformation: understanding the strategic process (1st ed., pp. 13–66). Routledge.

Chapter   Google Scholar  

Vrontis, D., Chaudhuri, R., & Chatterjee, S. (2022). Adoption of digital technologies by SMEs for sustainability and value creation: Moderating role of entrepreneurial orientation. Sustainability, 14 (13), 7949. https://doi.org/10.3390/su14137949

Wang, C. H., & Chien, Y. W. (2016). Combining balanced scorecard with data envelopment analysis to conduct performance diagnosis for Taiwanese LED manufacturers. International Journal of Production Research, 54 (17), 5169–5181. https://doi.org/10.1080/00207543.2016.1156780

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26 (2), xiii–xxiii.

Wengler, S., Hildmann, G., & Vossebein, U. (2021). Digital transformation in sales as an evolving process. Journal of Business & Industrial Marketing, 36 (4), 599–614. https://doi.org/10.1108/JBIM-03-2020-0124

Yadav, V. S., Singh, A. R., Gunasekaran, A., Raut, R. D., & Narkhede, B. E. (2022). A systematic literature review of the agro-food supply chain: Challenges, network design, and performance measurement perspectives. Sustainable Production and Consumption, 29 , 685–704. https://doi.org/10.1016/j.spc.2021.11.019

Zhu, X., Ge, S., & Wang, N. (2021). Digital transformation: A systematic literature review. Computers & Industrial Engineering, 162 , 107774. https://doi.org/10.1016/j.cie.2021.107774

Download references

Open access funding provided by Università Cattolica del Sacro Cuore within the CRUI-CARE Agreement. No funds, grants, or other support was received.

Author information

Authors and affiliations.

Università Cattolica del Sacro Cuore, Via Emilia Parmense, 84, 29122, Piacenza, Italy

Marcello Cosa & Riccardo Torelli

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Marcello Cosa .

Ethics declarations

Conflict of interest.

The authors, Prof. Marcello Cosa and Prof. Riccardo Torelli, declare that no conflicts of interest are associated with this manuscript. No member of the Editorial Board or Guest Editor of the Special Issue is a co-author of this study.

Ethical Approval

This manuscript is a systematic literature review not involving primary data collection, including questionnaires or surveys. Therefore, ethical approval is not applicable to this study. No personal data or identifiable information from participants was used, ensuring that ethical considerations regarding informed consent and data anonymization are irrelevant to this research.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cosa, M., Torelli, R. Digital Transformation and Flexible Performance Management: A Systematic Literature Review of the Evolution of Performance Measurement Systems. Glob J Flex Syst Manag (2024). https://doi.org/10.1007/s40171-024-00409-9

Download citation

Received : 05 March 2024

Accepted : 10 June 2024

Published : 01 July 2024

DOI : https://doi.org/10.1007/s40171-024-00409-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Adaptive management
• Decision-making
• Digital transformation
• Flexibility
• Organizational flexibility
• Performance measurement system
• Systematic review
• Find a journal
• Publish with us
• Track your research

• Search Menu
• Sign in through your institution
• Advance Articles
• Editor's Choice
• Collections
• Supplements
• InSight Papers
• BSR Registers Papers
• Virtual Roundtables
• Author Guidelines
• Submission Site
• Open Access Options
• Self-Archiving Policy
• About Rheumatology
• About the British Society for Rheumatology
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Dispatch Dates
• Terms and Conditions
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, supplementary material, data availability, acknowledgements.

• < Previous

Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis

• Article contents
• Figures & tables
• Supplementary Data

Philip J Mease, Dafna D Gladman, Joseph F Merola, Peter Nash, Stacy Grieve, Victor Laliman-Khara, Damon Willems, Vanessa Taieb, Adam R Prickett, Laura C Coates, Comparative efficacy and safety of bimekizumab in psoriatic arthritis: a systematic literature review and network meta-analysis, Rheumatology , Volume 63, Issue 7, July 2024, Pages 1779–1789, https://doi.org/10.1093/rheumatology/kead705

• Permissions Icon Permissions

To understand the relative efficacy and safety of bimekizumab, a selective inhibitor of IL-17F in addition to IL-17A, vs other biologic and targeted synthetic DMARDs (b/tsDMARDs) for PsA using network meta-analysis (NMA).

A systematic literature review (most recent update conducted on 1 January 2023) identified randomized controlled trials (RCTs) of b/tsDMARDs in PsA. Bayesian NMAs were conducted for efficacy outcomes at Weeks 12–24 for b/tsDMARD-naïve and TNF inhibitor (TNFi)-experienced patients. Safety at Weeks 12–24 was analysed in a mixed population. Odds ratios (ORs) and differences of mean change with the associated 95% credible interval (CrI) were calculated for the best-fitting models, and the surface under the cumulative ranking curve (SUCRA) values were calculated to determine relative rank.

The NMA included 41 RCTs for 22 b/tsDMARDs. For minimal disease activity (MDA), bimekizumab ranked 1st in b/tsDMARD-naïve patients and 2nd in TNFi-experienced patients. In b/tsDMARD-naïve patients, bimekizumab ranked 6th, 5th and 3rd for ACR response ACR20/50/70, respectively. In TNFi-experienced patients, bimekizumab ranked 1st, 2nd and 1st for ACR20/50/70, respectively. For Psoriasis Area and Severity Index 90/100, bimekizumab ranked 2nd and 1st in b/tsDMARD-naïve patients, respectively, and 1st and 2nd in TNFi-experienced patients, respectively. Bimekizumab was comparable to b/tsDMARDs for serious adverse events.

Bimekizumab ranked favourably among b/tsDMARDs for efficacy on joint, skin and MDA outcomes, and showed comparable safety, suggesting it may be a beneficial treatment option for patients with PsA.

For joint efficacy, bimekizumab ranked highly among approved biologic/targeted synthetic DMARDs (b/tsDMARDs).

Bimekizumab provides better skin efficacy (Psoriasis Area and Severity Index, PASI100 and PASI90) than many other available treatments in PsA.

For minimal disease activity, bimekizumab ranked highest of all available b/tsDMARDs in b/tsDMARD-naïve and TNF inhibitor–experienced patients.

PsA is a chronic, systemic, inflammatory disease in which patients experience a high burden of illness [ 1–3 ]. PsA has multiple articular and extra-articular disease manifestations including peripheral arthritis, axial disease, enthesitis, dactylitis, skin psoriasis (PSO) and psoriatic nail disease [ 4 , 5 ]. Patients with PsA can also suffer from related inflammatory conditions, uveitis and IBD [ 4 , 5 ]. Approximately one fifth of all PSO patients, increasing to one quarter of patients with moderate to severe PSO, will develop PsA over time [ 6 , 7 ].

The goal of treatment is to control inflammation and prevent structural damage to minimize disease burden, normalize function and social participation, and maximize the quality of life of patients [ 1 , 4 ]. As PsA is a heterogeneous disease, the choice of treatment is guided by individual patient characteristics, efficacy against the broad spectrum of skin and joint symptoms, and varying contraindications to treatments [ 1 , 4 ]. There are a number of current treatments classed as conventional DMARDs such as MTX, SSZ, LEF; biologic (b) DMARDs such as TNF inhibitors (TNFi), IL inhibitors and cytotoxic T lymphocyte antigen 4 (CTLA4)-immunoglobulin; and targeted synthetic (ts) DMARDs which include phosphodiesterase-4 (PDE4) and Janus kinase (JAK) inhibitors [ 1 , 8 ].

Despite the number of available treatment options, the majority of patients with PsA report that they do not achieve remission and additional therapeutic options are needed [ 9 , 10 ]. Thus, the treatment landscape for PsA continues to evolve and treatment decisions increase in complexity, especially as direct comparative data are limited [ 2 ].

Bimekizumab is a monoclonal IgG1 antibody that selectively inhibits IL-17F in addition to IL-17A, which is approved for the treatment of adults with active PsA in Europe [ 11 , 12 ]. Both IL-17A and IL-17F are pro-inflammatory cytokines implicated in PsA [ 11 , 13 ]. IL-17F is structurally similar to IL-17A and expressed by the same immune cells; however, the mechanisms that regulate expression and kinetics differ [ 13 , 14 ]. IL-17A and IL-17F are expressed as homodimers and as IL-17A–IL-17F heterodimers that bind to and signal via the same IL-17 receptor A/C complex [ 13 , 15 ].

In vitro studies have demonstrated that the dual inhibition of both IL-17A and IL-17F with bimekizumab was more effective at suppressing PsA inflammatory genes and T cell and neutrophil migration, and periosteal new bone formation, than blocking IL-17A alone [ 11 , 14 , 16 , 17 ]. Furthermore, IL-17A and IL-17F protein levels are elevated in psoriatic lesions and the superiority of bimekizumab 320 mg every 4 weeks (Q4W) or every 8 weeks (Q8W) over the IL-17A inhibitor, secukinumab, in complete clearance of psoriatic skin was demonstrated in a head-to-head trial in PSO [ 16 , 18 ]. Collectively, this evidence suggests that neutralizing both IL-17F and IL-17A may provide more potent abrogation of IL-17-mediated inflammation than IL-17A alone.

Bimekizumab 160 mg Q4W demonstrated significant improvements in efficacy outcomes compared with placebo, and an acceptable safety profile in adults with PsA in the phase 3 RCTs BE OPTIMAL (NCT03895203) (b/tsDMARD-naïve patients) and BE COMPLETE (NCT03896581) (TNFi inadequate responders) [ 19 , 20 ].

The objective of this study was to establish the comparative efficacy and safety of bimekizumab 160 mg Q4W vs other available PsA treatments, using network meta-analysis (NMA).

Search strategy

A systematic literature review (SLR) was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) guidelines [ 21 ] and adhered to the principles outlined in the Cochrane Handbook for Systematic Reviews of Interventions, Centre for Reviews and Dissemination’s Guidance for Undertaking Reviews in Healthcare, and Methods for the Development of National Institute of Health and Care Excellence (NICE) Public Health Guidance [ 22–24 ]. The SLR of English-language publications was originally conducted on 3 December 2015, with updates on 7 January 2020, 2 May 2022 and 1 January 2023 in Medical Literature Analysis and Retrieval System Online (MEDLINE ® ), Excerpta Medica Database (Embase ® ) and the Cochrane Central Register of Controlled Trials (CENTRAL) for literature published from January 1991 onward using the Ovid platform. Additionally, bibliographies of SLRs and meta-analyses identified through database searches were reviewed to ensure any publications not identified in the initial search were included in this SLR. Key clinical conference proceedings not indexed in Ovid (from October 2019 to current) and ClinicalTrials.gov were also manually searched. The search strategy is presented in Supplementary Table S1 (available at Rheumatology online).

Study inclusion

Identified records were screened independently and in duplicate by two reviewers and any discrepancies were reconciled via discussion or a third reviewer. The SLR inclusion criteria were defined by the Patient populations, Interventions, Comparators, Outcome measures, and Study designs (PICOS) Statement ( Supplementary Table S2 , available at Rheumatology online). The SLR included published studies assessing approved therapies for the treatment of PsA. Collected data included study and patient population characteristics, interventions, comparators, and reported clinical and patient-reported outcomes relevant to PsA. For efficacy outcomes, pre-crossover data were extracted in studies where crossover occurred. All publications included in the analysis were evaluated according to the Cochrane risk-of-bias tool for randomized trials as described in the Cochrane Handbook [ 25 ].

Network meta-analysis methods

NMA is the quantitative assessment of relative treatment effects and associated uncertainty of two or more interventions [ 26 , 27 ]. It is used frequently in health technology assessment, guideline development and to inform treatment decision making in clinical practice [ 26 ].

Bimekizumab 160 mg Q4W was compared with current b/tsDMARDs at regulatory-approved doses ( Table 1 ) by NMA. All comparators were selected on the basis they were relevant to clinical practice, i.e. recommended by key clinical guidelines, licensed by key regulatory bodies and/or routinely used.

NMA intervention and comparators

See Supplementary Table S4 , available at Rheumatology online for additional dosing schedules used in included studies. BID: twice daily; CTLA4-Ig: cytotoxic T lymphocyte antigen 4-immunoglobulin; IL-17A/17Fi: IL-17A/17F inhibitor; IL-17Ai: IL-17A inhibitor; IL-12/23i: IL-12/23 inhibitor; IL-23i: IL-23 inhibitor; JAKi: Janus kinase inhibitor; NMA: network meta-analysis; PDE-4i: phosphodiesterase-4 inhibitor; Q1W: once weekly; Q2W: every 2 weeks; Q4W: every 4 weeks; Q8W: every 8 weeks; Q12W: every 12 weeks; QD: once daily; TNFi: TNF inhibitor.

Two sets of primary analyses were conducted, one for a b/tsDMARD-naïve PsA population and one for a TNFi-experienced PsA population. Prior treatment with TNFis has been shown to impact the response to subsequent bDMARD treatments [ 28 ]. In addition, most trials involving b/tsDMARDs for the treatment of PsA (including bimekizumab) report separate data on both b/tsDMARD-naïve and TNFi-experienced subgroups, making NMA in each of these patient populations feasible.

For each population the following outcomes were analysed: American College of Rheumatology response (ACR20/50/70), Psoriasis Area and Severity Index (PASI90/100), and minimal disease activity (MDA). The analysis of serious adverse events (SAE) was conducted using a mixed population (i.e. b/tsDMARD-naïve, TNFi-experienced and mixed population data all were included) as patients’ previous TNFI exposure was not anticipated to impact safety outcomes following discussions with clinicians. The NMA included studies for which data were available at week 16, if 16-week data were not available (or earlier crossover occurred), data available at weeks 12, 14 or 24 were included. Pre-crossover data were included in the analyses for efficacy outcomes to avoid intercurrent events.

Heterogeneity between studies for age, sex, ethnicity, mean time since diagnosis, concomitant MTX, NSAIDs or steroid use was assessed using Grubb’s test, also called the extreme Studentized deviate method, to identify outlier studies.

All univariate analyses involved a 10 000 run-in iteration phase and a 10 000-iteration phase for parameter estimation. All calculations were performed using the R2JAGS package to run Just Another Gibbs Sampler (JAGS) 3.2.3 and the code reported in NICE Decision Support Unit (DSU) Technical Support Document Series [ 29–33 ]. Convergence was confirmed through inspection of the ratios of Monte-Carlo error to the standard deviations of the posteriors; values >5% are strong signs of convergence issues [ 31 ]. In some cases, trials reported outcome results of zero (ACR70, PASI100, SAE) in one or more arms for which a continuity correction was applied to mitigate the issue, as without the correction most models were not convergent or provided a large posterior distribution making little clinical sense [ 31 ].

Four NMA models [fixed effects (FE) unadjusted, FE baseline risk-adjusted, random effects (RE) unadjusted and RE baseline risk-adjusted] were assessed and the best-fit models were chosen using methods described in NICE DSU Technical Support Document 2 [ 31 ]. Odds ratios (ORs) and differences of mean change (MC) with the associated 95% credible intervals (CrIs) were calculated for each treatment comparison in the evidence network for the best fitting models and presented in league tables and forest plots. In addition, the probability of bimekizumab 160 mg Q4W being better than other treatments was calculated using surface under the cumulative ranking curve (SUCRA) to determine relative rank. Conclusions (i.e. better/worse or comparable) for bimekizumab 160 mg Q4W vs comparators were based on whether the pairwise 95% CrIs of the ORs/difference of MC include 1 (dichotomous outcomes), 0 (continuous outcomes) or not. In the case where the 95% CrI included 1 or 0, then bimekizumab 160 mg Q4W and the comparator were considered comparable. If the 95% CrI did not include 1 or 0, then bimekizumab 160 mg Q4W was considered either better or worse depending on the direction of the effect.

Compliance with ethics guidelines

This article is based on previously conducted studies and does not contain any new studies with human participants or animals performed by any of the authors.

Study and patient characteristics

The SLR identified 4576 records through databases and 214 records through grey literature, of which 3143 were included for abstract review. Following the exclusion of a further 1609 records, a total of 1534 records were selected for full-text review. A total of 66 primary studies from 246 records were selected for data extraction. No trial was identified as having a moderate or high risk of bias ( Supplementary Table S3 , available at Rheumatology online).

Of the 66 studies identified in the SLR, 41 studies reported outcomes at weeks 12, 16 or 24 and met the criteria for inclusion in the NMA in either a b/tsDMARD-naïve population ( n  = 20), a TNFi-experienced population ( n  = 5), a mixed population with subgroups ( n  = 13) or a mixed PsA population without subgroups reported ( n  = 3). The PRISMA diagram is presented in Fig. 1 . Included and excluded studies are presented in Supplementary Tables S4 and S5 , respectively (available at Rheumatology online).

PRISMA flow diagram. The PRISMA flow diagram for the SLR conducted to identify published studies assessing approved treatments for the treatment of PsA. cDMARD: conventional DMARD; NMA: network meta-analysis; NR: not reported; PD: pharmacodynamic; PK: pharmacokinetic; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT: randomized controlled trial; SLR: systematic literature review

The baseline study and patient characteristics (where reported) are presented in Supplementary Table S6 (available at Rheumatology online). There were 20–483 patients included in treatment arms. The median age of patients was 48.9 years, the median percentage of males was 50.3% and a median of 92.3% of patients were Caucasian. Patients had a mean time since diagnosis of 7.6 years and a mean PASI score of 8.7. The mean (range) use of concomitant MTX, NSAIDs and steroids were 53.9% (29.1% to 84.0%), 72.4% (33.3% to 100.0%) and 16.8% (9.2% to 30.0%), respectively. Heterogeneity was generally low across studies except for the concomitant use of MTX, NSAIDs and steroids. Using an approach consistent with established NMA methods in PsA [ 34–36 ], a meta-regression model using JAGS code reported in NICE DSU Technical Support Document 3 [ 33 ] was used to account for variation in placebo responses when model-fit statistics suggested that baseline risk-adjusted models provided a better fit to the data.

NMA results

The network diagrams for ACR50 in b/tsDMARD-naïve and TNFi-experienced patients are presented in Fig. 2A and B with network diagrams for other outcomes presented in Supplementary Fig. S1 (available at Rheumatology online). The networks for ACR response were larger, in terms of both number of studies and patients included, than the networks for PASI. Similarly, the networks for b/tsDMARD-naïve patients were larger than TNFi-experienced patients across all outcomes analysed. Placebo was used as a common comparator in all networks and there were a few studies that included more than two arms (OPAL-Broaden, Select-PsA-1, SPIRIT-P1 and BE OPTIMAL) that included adalimumab as the reference arm in b/tsDMARD-naïve patients. Lastly, networks included studies where the primary outcome was evaluated at time points longer than 16 weeks (e.g. EXCEED study at 52 weeks) but as per the methods, 16-week data formed the network.

Network of evidence for ACR50. ( A ) b/tsDMARD-naïve patients. ( B ) TNFi-experienced patients. The size of the circle representing each intervention is proportional to the number of patients included in the analysis. The line width is proportional to the number of studies connecting the interventions. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CZP: certolizumab pegol; ETA: etanercept; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RIS: risankizumab; SEC: secukinumab; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

The best-fit model is noted for each outcome with full model fit statistics for all outcomes presented in Supplementary Table S7 (available at Rheumatology online). Forest plots for ACR50 and PASI100 are presented in Figs 3 and 4 , with forest plots for other outcomes, along with the league tables in Supplementary Fig. S2 and Table S8 , respectively (available at Rheumatology online).

ACR50. The results for the NMA on ACR50 at week 16. ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 469.59. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 205.33. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg Q4W is considered either better or worse depending on the direction of the effect. ABA: abatacept; ADA: adalimumab; APR: apremilast; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ: bimekizumab; CrI: credible interval; CZP: certolizumab pegol; DIC: deviance information criterion; ETA: etanercept; FE: fixed effects; GOL: golimumab; GUS: guselkumab; IFX: infliximab; IV: intravenous; IXE: ixekizumab; NMA: network meta-analysis; PBO: placebo; Q4W: every 4 weeks; Q8W: every 8 weeks; RE: random effects; RIS: risankizumab; SEC: secukinumab; SUCRA: surface under the cumulative ranking curve; TNFi-experienced: TNF inhibitor–experienced; TOF: tofacitinib; UPA: upadacitinib; UST: ustekinumab; w/o LD: without loading dose

PASI100. The results for the NMA on PASI100 at week 16: ( A ) b/tsDMARD-naïve patients including forest plot and SUCRA values. FE baseline–adjusted model DIC = 150.27. ( B ) TNFi-experienced patients including forest plot and SUCRA values. RE-unadjusted model DIC = 81.76. a Week 24 data were used as week 16 data was not available. * The 95% CrI does not include 1; bimekizumab 160 mg 4W is considered better. ADA: adalimumab; b/tsDMARD-naïve: biologic and targeted synthetic DMARD-naïve; BKZ, bimekizumab; CrI, credible interval; CZP, certolizumab pegol; DIC, deviance information criterion; FE, fixed effects; GOL, golimumab; GUS, guselkumab; IXE, ixekizumab; NMA, network meta-analysis; PASI, Psoriasis Area and Severity Index; PBO, placebo; Q4W, every 4 weeks; Q8W, every 8 weeks; RE, random effects; SEC, secukinumab; SUCRA, surface under the cumulative ranking curve; TNFi-experienced, TNF inhibitor–experienced; UPA, upadacitinib

Joint outcomes

For ACR50 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

b/tsDMARD-naïve patients

Bimekizumab 160 mg Q4W ranked 6th for ACR20 (SUCRA = 0.75), 5th for ACR50 (SUCRA = 0.74) ( Fig. 3A ) and 3rd for ACR70 (SUCRA = 0.80) among 21 treatments. For ACR50, bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, guselkumab 100 mg Q4W, ustekinumab 45 mg, risankizumab 150 mg, guselkumab 100 mg Q8W and ustekinumab 90 mg; worse than golimumab 2 mg i.v.; and comparable to the remaining treatments in the network ( Fig. 3A ).

TNFi-experienced patients

Bimekizumab 160 mg Q4W ranked 1st among 16 treatments for ACR20 (SUCRA = 0.96), 2nd among 15 treatments for ACR50 (SUCRA = 0.84) ( Fig. 3B ) and 1st among 16 treatments for ACR70 (SUCRA = 0.83). Bimekizumab 160 mg Q4W was better than placebo, abatacept 125 mg, secukinumab 150 mg without loading dose, tofacitinib 5 mg and secukinumab 150 mg; and comparable to the remaining treatments in the network on ACR50 ( Fig. 3B ).

Skin outcomes

For PASI100 outcomes, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 2nd among 15 treatments (SUCRA = 0.89) for PASI90 and 1st among 11 treatments (SUCRA = 0.95) for PASI100 ( Fig. 4A ). Bimekizumab 160 mg Q4W was better than placebo, certolizumab pegol pooled, golimumab 2 mg i.v., secukinumab 150 mg, adalimumab 40 mg, upadacitinib 15 mg, secukinumab 300 mg and ixekizumab 80 mg Q4W; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4A ).

Bimekizumab 160 mg Q4W ranked 1st among 10 treatments (SUCRA = 0.85) for PASI90 and 2nd among 7 treatments (SUCRA = 0.79) for PASI100 ( Fig. 4B ). Bimekizumab 160 mg Q4W was better than placebo, ixekizumab 80 mg Q4W and upadacitinib 15 mg; and comparable to the remaining treatments in the network on PASI100 ( Fig. 4B ).

For MDA, the best-fit models for b/tsDMARD-naïve and TNFi-experienced were the FE baseline–adjusted model and RE-unadjusted model, respectively.

Bimekizumab 160 mg Q4W ranked 1st among 13 treatments (SUCRA = 0.91) and was better than placebo [OR (95% CrI) 6.31 (4.61–8.20)], guselkumab 100 mg Q4W [2.06 (1.29–3.10)], guselkumab 100 mg Q8W [1.76 (1.09–2.69)], risankizumab 150 mg [1.99 (1.40–2.76)] and adalimumab 40 mg [1.41 (1.01–1.93)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2G , available at Rheumatology online).

Bimekizumab 160 mg Q4W ranked 1st among 11 treatments (SUCRA = 0.83) and was better than placebo [12.10 (5.31–28.19)] and tofacitinib 5 mg [6.81 (2.14–21.35)]; and comparable to the remaining treatments in the network ( Supplementary Fig. S2H , available at Rheumatology online).

The network for SAEs for a mixed population included 23 treatments and the best-fit model was an RE-unadjusted model (due to study populations and time point reporting heterogeneity). Bimekizumab 160 mg Q4W showed comparable safety to all treatments in the network ( Supplementary Fig. S2I , available at Rheumatology online).

The treatment landscape for PsA is complex, with numerous treatment options and limited direct comparative evidence. Bimekizumab 160 mg Q4W has recently been approved for the treatment of active PsA by the European Medicines Agency and recommended by NICE in the UK, and the published phase 3 results warrant comparison with existing therapies by NMA.

This NMA included 41 studies evaluating 22 b/tsDMARDs including the novel IL-17F and IL-17A inhibitor, bimekizumab. Overall, bimekizumab 160 mg Q4W ranked favourably among b/tsDMARDS for efficacy in joint, skin and disease activity outcomes in PsA across both b/tsDMARD-naïve and TNFi-experienced populations. The safety of bimekizumab 160 mg Q4W was similar to the other b/tsDMARDs.

The Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA) and EULAR provide evidence-based recommendations for the treatment of PsA [ 1 , 2 ]. To treat peripheral arthritis symptoms in PsA, efficacy across the classes of current b/tsDMARDs are considered similar by both GRAPPA and EULAR, in part due to a lack of data comparing licensed therapies in a head-to-head trial setting [ 1 , 2 ]. EULAR recommends the use of JAK inhibitors in the case of inadequate response, intolerance or when a bDMARD is not appropriate [ 1 ]. This recommendation was made when tofacitinib was the only available JAK inhibitor, but reflects current marketing authorizations for tofacitinib and upadacitinib which indicate use in patients with an inadequate response or prior intolerance to TNFis (USA) or bDMARDs (Europe) [ 37–40 ]. This NMA suggests that bimekizumab 160 mg Q4W may have an advantage over current treatments, including IL-23 inhibitors in b/tsDMARD naïve patients, and secukinumab 150 mg and tofacitinib in TNFi-experienced patients, as evidenced by our analysis of ACR50 for which the pairwise comparisons were significantly in favour of bimekizumab 160 mg Q4W.

For the treatment of skin symptoms in PsA, IL-23, IL-12/23 and IL-17A inhibitors are currently recommended due to their greater efficacy compared with TNFis [ 1 , 4 ]. GRAPPA also suggests considering efficacy demonstrated in direct comparative studies in PSO when selecting a treatment for PsA skin symptoms [ 2 ]. In our analysis of complete skin clearance as measured by PASI100, bimekizumab 160 mg Q4W demonstrated the likelihood of significantly greater efficacy than IL-17A, JAK inhibitors and TNFis in b/tsDMARD-naïve patients and IL-17A and JAK inhibitors in TNFi-experienced patients. Furthermore, the NMA results for skin clearance in PsA are in alignment with previous studies in PSO that demonstrated superiority of bimekizumab 320 mg Q4W or Q8W vs secukinumab, ustekinumab and adalimumab ( P  < 0.001) (note that the dosing of bimekizumab in PSO differs from that in PsA) [ 12 , 18 , 41 , 42 ].

There are similarities between our results and other recently published NMAs of b/tsDMARDs in PsA, although methodological heterogeneity across all NMAs makes comparisons challenging [ 34–36 , 43–45 ]. Among recent NMAs, the largest evaluated 21 treatments [ 34 ] and only four considered subgroups of b/tsDMARD-naïve and TNFi-experienced patients or those with inadequate response [ 35 , 36 , 43 , 45 ]. Furthermore, different or pooled levels of response were evaluated for ACR and PASI outcomes.

Previous NMAs also support IL-17, IL-12/23 and IL-23 inhibitors having greater efficacy for skin symptoms than TNFis [ 35 , 36 ]. In an overall PsA population, McInnes et al. demonstrated that secukinumab 300 mg, ixekizumab 80 mg Q4W, and ustekinumab 45 mg and 90 mg were likely more efficacious than TNFis for PASI90 [ 35 ]. In another NMA by Ruyssen-Witrand et al. , results suggested that ixekizumab 80 mg Q4W had significantly greater efficacy than adalimumab, certolizumab pegol pooled, and etanercept 25 mg twice weekly/50 mg once weekly for any PASI score (50%, 75%, 90% and 100% reduction) in bDMARD-naïve patients [ 36 ].

For joint outcomes, Mease et al. compared guselkumab Q4W and Q8W with other b/tsDMARDs in a network of 21 treatments in an overall PsA population for ACR50 [ 34 ]. Both guselkumab dosing schedules were better than abatacept and apremilast, but golimumab 2 mg i.v. had a higher likelihood of ACR50 response than guselkumab Q8W [ 34 ]. Despite MDA being assessed in clinical trials for bDMARD therapies and a treatment target in PsA [ 46 ], evidence for comparative efficacy for this outcome is limited. None of the most recent NMAs before this one included an analysis of MDA [ 34–36 ]. With regard to safety outcomes, previous NMAs evaluating SAEs also resulted in either no difference between b/tsDMARDs vs placebo or other b/tsDMARDs [ 34 , 36 , 44 , 45 ].

This study has a number of strengths. To our knowledge this NMA represents the most comprehensive and in-depth comparative efficacy analysis of approved treatments in PsA to date. The evidence was derived from a recent SLR, ensuring that new RCTs and updated results from previously published RCTs were included. It is also the first NMA to include the phase 3 BE COMPLETE and BE OPTIMAL trials of bimekizumab [ 19 , 20 ]. Our NMA used robust methods and accounted for variation in placebo response through network meta-regression in accordance with NICE DSU Technical Support Documents [ 31–33 ]. As an acknowledgement of the evolution of treatment advances, separate analyses of b/tsDMARD-naïve and TNFi-experienced subgroups were conducted with the intent to assist healthcare decision-making in different clinical settings. In addition, a panel of clinical experts were consulted from project inception and are authors of this paper, ensuring inclusion of a comprehensive set of clinically meaningful outcomes, including the composite, treat-to-target outcome of MDA.

Despite the robust evidence base and methodology, this NMA has limitations. Indirect treatment comparisons such as this NMA are not a substitute for head-to-head trials. There was heterogeneity in the endpoints and reporting in the included studies. Fewer studies reporting PASI outcomes resulted in smaller networks compared with the network of studies evaluating ACR response criteria. Not all trials reported outcomes at the same timepoint, thereby reducing the comparability of trial results, which has been transparently addressed by noting where week 24 data were used vs week 12, 14 or 16 data. The analyses for the TNFi-experienced population were limited by potential heterogeneity, especially in the analyses where fewer studies were included in the networks, as this group could include patients who had an inadequate response to TNFi or discontinued TNFi treatment due to other reasons (e.g. lost access). Also, in the analyses for the TNFi-experienced population, very low patient numbers for some treatments resulted in less statistical power. Additionally, the data included in the analysis were derived exclusively from RCTs, for which the study populations may not reflect a typical patient population seen in real-world practice. For example, trial results may be different in patients with oligoarthritis who are not well-represented in clinical trials.

Over the years covering our SLR, we acknowledge that patient populations and the PsA treatment landscape have evolved. After a thorough review of baseline patient characteristics, no significant differences were observed across the studies included in the NMA. To further mitigate uncertainty, baseline regression was used to actively correct for changes in the placebo rate over time ensuring a consistent and fair comparison across all included treatments. In addition, our analyses were conducted in separate b/tsDMARD-naïve and TNFi-experienced populations that reflect the evolving PsA patient population over time. Radiographic progression was not within the purview of this NMA because the NMA focused on a shorter timeframe than the 52-week duration typically recommended by the literature for investigating radiographic progression. Furthermore, there is existing literature on this topic, as exemplified by the work of Wang et al. in 2022 [ 47 ]. Nevertheless, the comprehensive and current evidence base, examination of multiple endpoints, and consistency with previous reported NMAs lend credence to our results.

Overall, the results of this NMA demonstrated the favourable relative efficacy and safety of bimekizumab 160 mg Q4W vs all approved treatments for PsA. Bimekizumab ranked high in terms of efficacy on joint, skin and MDA outcomes in both b/tsDMARD-naïve and TNFi-experienced patient populations, and showed comparable safety to other treatments. In the evolving PsA treatment landscape, bimekizumab 160 mg Q4W is a potentially beneficial treatment option for patients with PsA.

Supplementary material is available at Rheumatology online.

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

This study was funded in full by UCB Pharma.

Disclosure statement : P.J.M.: has received research grants from AbbVie, Amgen, BMS, Eli Lilly, Gilead, Janssen, Novartis, Pfizer, Sun Pharma and UCB Pharma; consultancy fees from AbbVie, Acelyrin, Aclaris, Amgen, BMS, Boehringer Ingelheim, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Moonlake Pharma, Novartis, Pfizer, Sun Pharma and UCB Pharma; and speakers’ bureau for AbbVie, Amgen, Eli Lilly, Janssen, Novartis, Pfizer and UCB Pharma. L.C.C.: received grants/research support from AbbVie, Amgen, Celgene, Eli Lilly, Janssen, Novartis, Pfizer and UCB; worked as a paid consultant for AbbVie, Amgen, Bristol Myers Squibb, Celgene, Eli Lilly, Gilead, Galapagos, Janssen, Moonlake, Novartis, Pfizer and UCB; and has been paid as a speaker for AbbVie, Amgen, Biogen, Celgene, Eli Lilly, Galapagos, Gilead, GSK, Janssen, Medac, Novartis, Pfizer and UCB. D.D.G.: consultant and/or received grant support from Abbvie, Amgen, BMS, Celgene, Eli Lilly, Galapagos, Gilead, Janssen, Novartis, Pfizer and UCB. J.F.M.: consultant and/or investigator for AbbVie, Amgen, Biogen, BMS, Dermavant, Eli Lilly, Janssen, LEO Pharma, Novartis, Pfizer, Regeneron, Sanofi, Sun Pharma and UCB Pharma. P.N.: research grants, clinical trials and honoraria for advice and lectures on behalf of AbbVie, Boehringer Ingelheim, BMS, Eli Lilly, Galapagos/Gilead, GSK, Janssen, Novartis, Pfizer, Samsung, Sanofi and UCB Pharma. S.G. and V.L.-K.: employees of Cytel, Inc. which served as a consultant on the project. A.R.P., D.W. and V.T.: employees and stockholders of UCB Pharma.

The authors acknowledge Leah Wiltshire of Cytel for medical writing and editorial assistance based on the authors’ input and direction, Heather Edens (UCB Pharma, Smyrna, GA, USA) for publication coordination and Costello Medical for review management, which were funded by UCB Pharma. This analysis was funded by UCB Pharma in accordance with Good Publication Practice (GPP 2022) guidelines ( http://www.ismpp.org/gpp-2022 ). Data were previously presented at ISPOR-US 2023 (Boston, MA, USA, 7–10 May 2023).

Gossec L , Baraliakos X , Kerschbaumer A et al.  EULAR recommendations for the management of psoriatic arthritis with pharmacological therapies: 2019 update . Ann Rheum Dis 2020 ; 79 : 700 – 12 .

Google Scholar

Coates LC , Soriano ER , Corp N et al. ; GRAPPA Treatment Recommendations domain subcommittees . Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 18 : 465 – 79 .

Fitzgerald O , Ogdie A , Chandran V et al.  Psoriatic arthritis . Nat Rev Dis Primers 2021 ; 7 : 59 .

Coates LC , Soriano ER , Corp N et al.  Treatment recommendations domain subcommittees. Group for Research and Assessment of Psoriasis and Psoriatic Arthritis (GRAPPA): updated treatment recommendations for psoriatic arthritis 2021 . Nat Rev Rheumatol 2022 ; 27 : 1 – 15 .

Najm A , Goodyear CS , McInnes IB , Siebert S. Phenotypic heterogeneity in psoriatic arthritis: towards tissue pathology-based therapy . Nat Rev Rheumatol 2023 ; 19 : 153 – 65 .

Ogdie A , Weiss P. The epidemiology of psoriatic arthritis . Rheum Dis Clin North Am 2015 ; 41 : 545 – 68 .

Alinaghi F , Calov M , Kristensen LE et al.  Prevalence of psoriatic arthritis in patients with psoriasis: a systematic review and meta-analysis of observational and clinical studies . J Am Acad Dermatol 2019 ; 80 : 251 – 65.e19 .

Singh JA , Guyatt G , Ogdie A et al.  Special Article: 2018 American college of rheumatology/national psoriasis foundation guideline for the treatment of psoriatic arthritis . Arthritis Rheumatol 2019 ; 71 : 5 – 32 .

Coates LC , Robinson DE , Orbai AM et al.  What influences patients' opinion of remission and low disease activity in psoriatic arthritis? Principal component analysis of an international study . Rheumatology (Oxford) 2021 ; 60 : 5292 – 9 .

Gondo G , Mosca M , Hong J et al.  Demographic and clinical factors associated with patient-reported remission in psoriatic arthritis . Dermatol Ther (Heidelb) 2022 ; 12 : 1885 – 95 .

Glatt S , Baeten D , Baker T et al.  Dual IL-17A and IL-17F neutralisation by bimekizumab in psoriatic arthritis: evidence from preclinical experiments and a randomised placebo-controlled clinical trial that IL-17F contributes to human chronic tissue inflammation . Ann Rheum Dis 2018 ; 77 : 523 – 32 .

UCB Pharma S.A . Bimzelx ® (bimekizumab): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/bimzelx (26 June 2023, date last accessed).

Adams R , Maroof A , Baker T et al.  Bimekizumab, a novel humanized IgG1 antibody that neutralizes both IL-17A and IL-17F . Front Immunol 2020 ; 11 : 1894 .

Burns LA , Maroof A , Marshall D et al.  Presence, function, and regulation of IL-17F-expressing human CD4(+) T cells . Eur J Immunol 2020 ; 50 : 568 – 80 .

Kuestner R , Taft D , Haran A et al.  Identification of the IL-17 receptor related molecule IL-17RC as the receptor for IL-17F . J Immunol 2007 ; 179 : 5462 – 73 .

Johansen C , Usher PA , Kjellerup RB et al.  Characterization of the interleukin-17 isoforms and receptors in lesional psoriatic skin . Br J Dermatol 2009 ; 160 : 319 – 24 .

Shah M , Maroof A , Gikas P et al.  Dual neutralisation of IL-17F and IL-17A with bimekizumab blocks inflammation-driven osteogenic differentiation of human periosteal cells . RMD Open 2020 ; 6 : e001306 .

Reich K , Warren RB , Lebwohl M et al.  Bimekizumab versus Secukinumab in Plaque Psoriasis . New Engl J Med 2021 ; 385 : 142 – 52 .

McInnes IB , Asahina A , Coates LC et al.  Bimekizumab in patients with psoriatic arthritis, naive to biologic treatment: a randomised, double-blind, placebo-controlled, phase 3 trial (BE OPTIMAL) . Lancet 2023 ; 401 : 25 – 37 .

Merola JF , Landewe R , McInnes IB et al.  Bimekizumab in patients with active psoriatic arthritis and previous inadequate response or intolerance to tumour necrosis factor-alpha inhibitors: a randomised, double-blind, placebo-controlled, phase 3 trial (BE COMPLETE) . Lancet 2023 ; 401 : 38 – 48 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Higgins JP. Cochrane handbook for systematic reviews of interventions. Vol. 2. 2nd edn. Chichester, UK: John Wiley & Sons, 2019 .

National Institute for Health and Care Excellence . The guidelines manual: Process and methods [PMG6]. 2012 . https://www.nice.org.uk/process/pmg6/chapter/introduction (3 March 2023, date last accessed).

Booth AM , Wright KE , Outhwaite H. Centre for Reviews and Dissemination databases: value, content, and developments . Int J Technol Assess Health Care 2010 ; 26 : 470 – 2 .

Sterne JAC , Savovic J , Page MJ et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials . BMJ 2019 ; 366 : l4898 .

Daly C , Dias S , Welton N , Anwer S , Ades A. NICE Guidelines Technical Support Unit. Meta-Analysis: Guideline Methodology Document 1 (Version 1). 2021 . http://www.bristol.ac.uk/population-health-sciences/centres/cresyda/mpes/nice/guideline-methodology-documents-gmds/ (1 March 2023, date last accessed).

Dias S , Caldwell DM. Network meta-analysis explained . Archi Dis Childhood Fetal Neonatal Ed 2019 ; 104 : F8 – F12 .

Merola JF , Lockshin B , Mody EA. Switching biologics in the treatment of psoriatic arthritis . Semin Arthritis Rheum 2017 ; 47 : 29 – 37 .

Openbugs (website) 2014 . http://www.openbugs.net/w/FrontPage (6 April 2023, date last accessed).

Lunn D , Spiegelhalter D , Thomas A , Best N. The BUGS project: evolution, critique and future directions . Stat Med 2009 ; 28 : 3049 – 67 .

Dias S , Welton NJ , Sutton AJ , Ades AE. NICE DSU technical support document 2: a generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. London. 2016 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Welton N , Sutton AJ , Caldwell DM , Lu G , Ades AE. NICE DSU technical support document 4: inconsistency in networks of evidence based on randomised controlled trials. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Dias S , Sutton AJ , Welton N , Ades AE. NICE DSU Technical support document 3: heterogeneity: subgroups, meta-regression, bias and bias-adjustment. 2011 . https://www.sheffield.ac.uk/nice-dsu/tsds/full-list (25 January 2023, date last accessed).

Mease PJ , McInnes IB , Tam LS et al.  Comparative effectiveness of guselkumab in psoriatic arthritis: results from systematic literature review and network meta-analysis . Rheumatology (Oxford) 2021 ; 60 : 2109 – 21 .

McInnes IB , Sawyer LM , Markus K et al.  Targeted systemic therapies for psoriatic arthritis: a systematic review and comparative synthesis of short-term articular, dermatological, enthesitis and dactylitis outcomes . RMD Open 2022 ; 8 : e002074 .

Ruyssen-Witrand A , Perry R , Watkins C et al.  Efficacy and safety of biologics in psoriatic arthritis: a systematic literature review and network meta-analysis . RMD Open 2020 ; 6 : e001117 .

Pfizer Inc . XELJANZ ® (tofacitinib): Summary of Product Characteristics. 2022 . https://www.ema.europa.eu/en/medicines/human/EPAR/xeljanz (4 May 2023, date last accessed).

Pfizer Inc . XELJANZ ® (tofacitinib): US Prescribing Information. 2022 . https://labeling.pfizer.com/ShowLabeling.aspx?id=959 (4 May 2023, date last accessed).

Abbvie Inc . RINVOQ ® (upadacitinib) extended-release tablets, for oral use: US Prescribing Information. 2023 . https://www.rxabbvie.com/pdf/rinvoq_pi.pdf (4 May 2023, date last accessed).

AbbVie Deutschland GmbH & Co . KG. RINVOQ ® (upadacitinib): Summary of Product Characteristics. 2023 . https://www.ema.europa.eu/en/medicines/human/EPAR/rinvoq (4 May 2023, date last accessed).

Warren RB , Blauvelt A , Bagel J et al.  Bimekizumab versus Adalimumab in Plaque Psoriasis . N Engl J Med 2021 ; 385 : 130 – 41 .

Reich K , Papp KA , Blauvelt A et al.  Bimekizumab versus ustekinumab for the treatment of moderate to severe plaque psoriasis (BE VIVID): efficacy and safety from a 52-week, multicentre, double-blind, active comparator and placebo controlled phase 3 trial . Lancet 2021 ; 397 : 487 – 98 .

Gladman DD , Orbai AM , Gomez-Reino J et al.  Network meta-analysis of tofacitinib, biologic disease-modifying antirheumatic drugs, and apremilast for the treatment of psoriatic arthritis . Curr Ther Res Clin Exp 2020 ; 93 : 100601 .

Qiu M , Xu Z , Gao W et al.  Fourteen small molecule and biological agents for psoriatic arthritis: a network meta-analysis of randomized controlled trials . Medicine (Baltimore) 2020 ; 99 : e21447 .

Kawalec P , Holko P , Mocko P , Pilc A. Comparative effectiveness of abatacept, apremilast, secukinumab and ustekinumab treatment of psoriatic arthritis: a systematic review and network meta-analysis . Rheumatol Int 2018 ; 38 : 189 – 201 .

Gossec L , McGonagle D , Korotaeva T et al.  Minimal disease activity as a treatment target in psoriatic arthritis: a review of the literature . J Rheumatol 2018 ; 45 : 6 – 13 .

Wang SH , Yu CL , Wang TY , Yang CH , Chi CC. Biologic disease-modifying antirheumatic drugs for preventing radiographic progression in psoriatic arthritis: a systematic review and network meta-analysis . Pharmaceutics 2022 ; 14 .

Supplementary data

Email alerts

Citing articles via.

• Rheumatology Twitter
• BSR Twitter
• BSR Facebook
• Recommend to Your Librarian

Affiliations

• Online ISSN 1462-0332
• Print ISSN 1462-0324
• Copyright © 2024 British Society for Rheumatology
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC11216577

Efficacy of acupuncture combined with traditional Chinese herbal for primary epilepsy patients with cognitive impairment: A protocol for systematic review and meta-analysis

1 Department of Neurology, Sichuan Taikang Hospital, Chengdu, Sichuan, China

2 Department of Rehabilitation, Affiliated Hospital of Yunnan University, Kunming, Yunnan, China

Hongxian He

Associated data.

No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.

Epilepsy is a common and serious chronic neurological disorder, and some patients suffer from cognitive dysfunction. We aim to assess the efficacy and safety of acupuncture combined with traditional Chinese herbal for primary epilepsy patients with cognitive impairment.

To search the randomized control trials (RCTs) published before April 20, 2023 from PubMed, Embase, Cochrane Library, Chinese Biomedical Literature Database (CBM), China National Knowledge Infrastructure (CNKI), Web of science, and Wanfang Database. The risk of bias within each individual trial was evaluated using the Cochrane Collaboration tool. RevMan5.3 software was used for statistical analysis. The odds ratio (OR) or weighted mean difference (WMD) with a 95% confidence interval (CI) was calculated for each RCT before data pooling.

The primary outcomes involve changes in cognitive function and behavioral disturbances. The secondary outcomes focused on quality of life and adverse effects.

The results of this review are expected to provide new guidelines for the treatment of primary epilepsy patients with cognitive impairment.

Trial registration

This systematic review protocol was registered at the International Prospective Register of Systematic Reviews (PROSPERO) (Registration number: CRD42023415355 ).

1. Introduction

Epilepsy is a chronic disease in which sudden abnormal discharge of brain neurons leads to transient brain dysfunction [ 1 ]. Each epileptic seizure of the patient is unpredictable, which not only seriously reduces the quality of life, but also causes different degrees of cognitive dysfunction, manifested as a functional decline in attention, memory, abstract thinking ability, reasoning ability, etc. even lost [ 2 , 3 ]. Epidemiological studies revealed that the global prevalence of epilepsy has surpassed 70 million, with approximately 700,000 new cases reported annually [ 2 ]. Numerous experimental models have demonstrated that cognitive dysfunction is a common comorbidity in epilepsy. For instance, Kumar et al. discovered that all rats with pentetrazole-induced epilepsy exhibited cognitive deficits [ 4 ]. In clinical practice, the prevalence of cognitive impairment in epilepsy patients varies from 30% to 40%, depending on the assessment method and population under evaluation [ 5 ]. Cognitive impairment significantly impacts the quality of life for individuals with epilepsy, emphasizing the significance of early prediction and intervention in managing cognitive dysfunction. With the promotion and application of the “biology-psychology-social” medical model, in the process of clinical diagnosis and treatment of epilepsy, it is necessary to pay attention not only to the control of epileptic seizures but also to the accompanying cognitive impairment of patients.

The earliest records of epilepsy in China can be traced back to 2000 years ago in the Huang Di Nai Jing of traditional Chinese medicine. Traditional Chinese medicine has a long history and rich experience in the treatment of epilepsy. It has the advantages of a stable curative effect, few side effects, and can promote the recovery of brain cell function, and is expected to be applied in antiepileptic therapy [ 6 , 7 ]. Acupuncture is another important traditional healing method, which has been used and applied for more than 2000 years in China. In recent years, acupuncture has shown positive therapeutic effects in improving overall effectiveness of epilepsy treatment, reducing the severity of epileptic seizures, and minimizing adverse reactions. Research has confirmed that electroacupuncture stimulation of “Zu-san-li” and “Shang-ju-xu” for 6 weeks significantly reduces the level of cyclooxygenase-2 (COX-2) in the hippocampus of epileptic rats, resulting in an anti-inflammatory effect [ 8 ]. Furthermore, electroacupuncture intervention has been found to upregulate the expression of Nuclear factor erythroid2-related factor 2 (Nrf-2) and its downstream antioxidant factors, activating the Nrf-2-ARE signaling pathway to protect brain tissue against oxidative stress damage [ 9 ]. In summary, acupuncture’ s mechanism in treating epilepsy encompasses various aspects of the condition’ s pathogenesis, including apoptosis inhibition, inflammatory response suppression, oxidative stress reduction, ion channel regulation, and modulation of intestinal flora. However, high-quality evidence from relevant studies is lacking. Therefore, we carried out a meta-analysis of randomized controlled trials (RCTs) to reach a solid conclusion with a larger sample size. We aimed to determine the effects of acupuncture for epilepsy patients with cognitive impairment and provide evidence-based recommendations for patients with epilepsy.

2.1 Study registration

This systematic review protocol was registered at PROSPERO (Registration number: CRD42023415355). The protocol is according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement guidelines. The results of this meta-analysis will be published in a journal or conferences.

2.2 Inclusion criteria

2.2.1 participants.

Patients who are clinically diagnosed with epilepsy patients with cognitive impairment according to the diagnostic criteria of the 10TH revision of the International Classification of Diseases (ICD-10) (World Health Organization, 1992) and the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) (American Psychiatric Association, 1994) will be included. The onset of cognitive impairment was required to be definitely and causally linked with primary epilepsy. Additionally, patients with combined heart, lung, kidney, liver, bone marrow, blood, and other systemic diseases, combined intracranial occupying lesions, combined cerebral infarction, combined hysteria, schizophrenia, and another psychiatric history should be excluded.

2.2.2 Interventions

For the experimental group, trials that use acupuncture therapy combined with traditional Chinese herbs will be included; and acupuncture therapy involves manual acupuncture, electroacupuncture, fire acupuncture, warm acupuncture, and scalp acupuncture. For the corresponding control group, interventions could be placebo or waiting list control, sham acupuncture, conventional treatment, or pharmacotherapy that consists of the experimental group.

2.2.3 Outcome indicators

The primary outcomes focused on cognitive function and behavioral disturbances, which could be measured by scales such as the EpiTrack score, the Mini-mental state examination (MMSE), the Hasegawa’s Dementia Scale (HDS), the Montreal Cognitive Assessment (MoCA) score. The secondary outcomes focused on quality of life and adverse effects which could be measured by scales such as the Quality of Life Scale for Epileptic Patients-31 (QOLIE-31), the short form 36 health survey questionnaire (SF-36), the Activities of Daily Living (ADL) Scale, and the Functional Activities Questionnaire (FAQ).

2.2.4 Study type

Randomized controlled trials (RCTs) related to epilepsy patients with cognitive impairment will be included without language restrictions. Non-RCTs, inappropriate intervention, uncontrolled trials, reviews, and protocols for RCTs were excluded.

2.3 Data sources and search strategies

All published RCTs in the following databases will be searched from their inception to April 2023: PubMed, EMBASE, Web of Science, Chinese Biomedical Literature Database, the Cochrane Central Register of Controlled Trials (CENTRAL), Wanfang database, and the China National Knowledge Infrastructure (CNKI), without language restrictions.

The search terms included “epilepsy”, “Epilepsies”, “Seizure Disorder”, “Seizure Disorders”, “Cognitive Dysfunctions”, “Cognitive Impairments”, “Cognitive Impairment”, “Cognitive Disorder”, “Cognitive Decline”, “Cognitive Declines”, “Mental Deterioration”, “Acupuncture Therapy”, “Acupuncture”, “Traditional Chinese medicine”, “Traditional Chinese herbs”. An example of the search strategy for PubMed is summarized in Table 1 .

2.4 Study selection and data extraction

We will import all identified literature into EndNote X9 software to delete any duplicates. Two authors will screen the titles/abstracts of all potential studies to remove studies that are not related to the topic. Then, the full text of the remaining studies will be read carefully to further determine whether they fulfill all eligible criteria. If necessary, a third author will help to solve any divergence between the two authors. The following data were extracted from the included studies: (1) basic information of the study including the first author’s name, year of publication, sample sizes, participant’s age, and disease duration; (2) interventions details; (3) control details; (4) primary outcome indicators, secondary outcome indicators before and after the intervention; (5) adverse events. If available data could not be obtained directly from the article, the corresponding author was contacted and clarification of ambiguities and missing information was provided by phone or email. Details of the selection process will be presented in the PRISMA flow chart ( Fig 1 ).

“n” represents the number of studies.

2.5 Methodological quality assessment

The methodological quality of the included references was assessed using the Cochrane Collaboration’s Risk of Bias (ROB) tool by two reviewers independently. This assessment tool mainly including the generation of random sequences, allocation concealment, blinding of researchers and subjects, blinding in outcome assessment, and completeness of outcome data, selective reporting of research results and other sources of bias. Each item is judged as: “low risk”, “high risk” or “unclear”. Any discrepancies concerning the assessments were resolved through the discussion with a methodological researcher.

2.6 Statistical analysis

To conduct statistical analysis, we used Review Manager (RevMan) software version 5.3.0 (Cochrane Central Executive Team, United Kingdom) as recommended by Cochrane Collaboration. Heterogeneity was measured using both Cochran’s Q-test (P value ≤0.10 was used to define a significant degree of heterogeneity) and the I 2 statistic ( I 2 >50% showed the existence of heterogeneity). If the heterogeneity was significant (P < 0.1, I 2 > 50%), a random effect (RE) model was chosen to pool the data, and if there was acceptable heterogeneity (P ≥ 0.1, I 2 ≤ 50%), a fixed effect (FE) model was used. The mean difference (MD) or standard mean difference (SMD) was used to represent continuous data, and the Odds Ratio (OR) were used to represent dichotomous data. Results were reported with 95% confidence intervals (CI), and P < 0.05 was considered a significant statistical effect. Subgroup analyses will be performed for heterogeneity by intervention, type of control (placebo, sham acupuncture, usual care, or drug treatment), duration of treatment, point, and outcome measure. Sensitivity analyses were performed on the primary endpoint to test the homogeneity of the results. After excluding low-quality studies, we will rerun the meta-analysis and use other statistical methods.

2.7 Assessment of publication bias

When more than 10 studies were included, publication bias was assessed using funnel plots. We analyzed potential publication bias using Egger or Begg tests and estimated results according to the Cochrane Handbook of Systematic Reviews of Interventions.

2.8 Ethics and dissemination

No human or animal subjects or samples were used. Patients did not participate in the formulation of research questions, measurement of outcomes, and study design.

3. Discussion

At present, the influencing factors of cognitive dysfunction in epilepsy patients have been basically clarified, but the pathogenesis has not yet been fully clarified. Studies have found that epilepsy-related cognitive impairment may be related to damage such as hippocampal sclerosis, abnormal transmitter transmission, inflammatory factors and oxidative stress [ 10 – 14 ]. D’Avila et al. discovered that activated microglia release various inflammatory mediators, which can lead to tissue damage and neurotoxicity through mechanisms like oxidative stress and synaptic remodeling [ 15 ]. These processes have been linked to cognitive impairment. Mishra et al. conducted a study using a mouse model of epilepsy and found that reducing lipid peroxidation levels and increasing antioxidant mechanisms resulted in shorter seizure duration and improved cognitive function [ 16 ]. The existence of cognitive impairment will bring a certain burden to the life of epilepsy patients. Therefore, it is of great significance to diagnose and treat cognitive impairment in time to reduce the burden on patients and their families.

Modern pharmacological studies have shown that some traditional Chinese medicines (such as Gastrodia elata, Polygala tenuifolia, Acorus tatarinowii, Nardostachys chinensis, Safflower, etc.) have the effects of inhibiting platelet aggregation, prolonging thrombin time, protecting neurons, nourishing blood and calming the nerves, antiepileptic, sedative, antispasmodic, and analgesic [ 17 – 20 ]. Acupuncture and moxibustion treat epilepsy mainly by disrupting brain waves, inhibiting seizures, affecting the neuroendocrine immune system, and protecting neurons, thereby reducing seizures and improving the quality of life of patients with epilepsy. Acupuncture acts on acupoints through physical stimulation, stimulates related nerve veins, affects the secretion and release of neurotransmitters in the brain, and regulates the electrical activity of brain nerve cells [ 9 , 21 ].

However, there is no studies have not been systematically organized on the efficacy of acupuncture for primary epilepsy patients with cognitive impairment. Thus, it is necessary to organize evidence that has been proven by RCTs, so that it can be used.

Supporting information

S1 checklist, funding statement.

The author(s) received no specific funding for this work.

Data Availability

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Systematic review of factors influencing students’ performance in educational buildings: focus on lca, iot, and bim.

1. Introduction

2. Materials and Methods

3.1. life cycle assessment (lca), 3.2. building information modeling (bim), 3.3. internet of things (iot), 3.4. digital twins (dts), 4. discussion, 4.1. indoor environment factors, 4.2. new-generation vs. fundamental articles, 4.3. interdisciplinary research findings, 4.4. focus on iaq and overlooked aspects, 4.5. database comparison and article selection, 5. conclusions, author contributions, data availability statement, conflicts of interest.

• Klepeis, N.E.; Nelson, W.C.; Ott, W.R.; Robinson, J.P.; Tsang, A.M.; Switzer, P.; Behar, J.V.; Hern, S.C.; Engelmann, W.H. The National Human Activity Pattern Survey (NHAPS): A resource for assessing exposure to environmental pollutants. J. Expo. Sci. Environ. Epidemiol. 2001 , 11 , 231–252. [ Google Scholar ] [ CrossRef ]
• Wang, D.; Song, C.; Wang, Y.; Xu, Y.; Liu, Y.; Liu, J. Experimental investigation of the potential influence of indoor air velocity on students’ learning performance in summer conditions. Energy Build. 2020 , 219 , 110015. [ Google Scholar ] [ CrossRef ]
• Currie, J.; Neidell, M.; Schmieder, J.F. Air pollution and infant health: Lessons from New Jersey. J. Health Econ. 2009 , 28 , 688–703. [ Google Scholar ] [ CrossRef ]
• Currie, J.; Reed, W. Traffic Congestion and Infant Health: Evidence from E-ZPass. Am. Econ. J. Appl. Econ. 2011 , 3 , 65–90. [ Google Scholar ] [ CrossRef ]
• Knittel, C.R.; Miller, D.L.; Sanders, N.J. Caution, drivers! children present: Traffic, pollution, and infant health. Rev. Econ. Stat. 2016 , 98 , 350–366. [ Google Scholar ] [ CrossRef ]
• Schlenker, W.; Walker, W.R. Airports, Air Pollution, and Contemporaneous Health. Rev. Econ. Stud. 2016 , 83 , 768–809. [ Google Scholar ] [ CrossRef ]
• Deryugina, T.; Heutel, G.; Miller, N.H.; Molitor, D.; Reif, J. The Mortality and Medical Costs of Air Pollution: Evidence from Changes in Wind Direction. Am. Econ. Rev. 2019 , 109 , 4178–4219. [ Google Scholar ] [ CrossRef ]
• Heft-Neal, S.; Burney, J.; Bendavid, E.; Voss, K.; Burke, M. Air Pollution and Infant Mortality: Evidence from Saharan Dust. Working Paper 26107, National Bureau of Economic Research. 2019. Available online: https://www.nber.org/papers/w26107 (accessed on 20 February 2024).
• Simeonova, E.; Currie, J.; Nilsson, P.; Walker, R. Congestion Pricing, Air Pollution, and Childrens Health. J. Hum. Resour. 2019 , 56 , 971–996. [ Google Scholar ] [ CrossRef ]
• Ebenstein, A.; Lavy, V.; Roth, S. The Long-Run Economic Consequences of High-Stakes Examinations: Evidence from Transitory Variation in Pollution. Am. Econ. J. Appl. Econ. 2016 , 8 , 36–65. [ Google Scholar ] [ CrossRef ]
• Persico, C.L.; Venator, J. The effects of local industrial pollution on students and schools. J. Hum. Resour. 2021 , 56 , 406–445. [ Google Scholar ] [ CrossRef ]
• Heissel, J.A.; Persico, C.; Simon, D. Does Pollution Drive Achievement? The Effect of Traffic Pollution on Academic Performance. J. Hum. Resour. 2022 , 57 , 747–776. [ Google Scholar ] [ CrossRef ]
• Gaihre, S.; Semple, S.; Miller, J.; Fielding, S.; Turner, S. Classroom Carbon Dioxide Concentration, School Attendance, and Educational Attainment. J. Sch. Health 2014 , 84 , 569–574. [ Google Scholar ] [ CrossRef ]
• Kielb, C.; Lin, S.; Muscatiello, N.; Hord, W.; Rogers-Harrington, J.; Healy, J. Building-related health symptoms and classroom indoor air quality: A survey of school teachers in New York State. Indoor Air 2015 , 25 , 371–380. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mendell, M.J.; Heath, G.A. Do indoor pollutants and thermal conditions in schools influence student performance? A critical review of the literature. Indoor Air 2005 , 15 , 27–52. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Shendell, D.; Prill, R.J.; Fisk, W.; Apte, M.; Blake, D.; Faulkner, D. Association between class room CO 2 concentrations and student attendance in Washington and Idaho. Indoor Air 2004 , 14 , 333–341. [ Google Scholar ] [ CrossRef ]
• Wargocki, P.; Wyon, D.P. Ten questions concerning thermal and indoor air quality effects on the performance of office work and schoolwork. Build. Environ. 2017 , 112 , 359–366. [ Google Scholar ] [ CrossRef ]
• Fisk, W. The ventilation problem in schools: Literature review. Indoor Air 2017 , 27 , 1039–1051. [ Google Scholar ] [ CrossRef ]
• Madureira, J.; Paciência, I.; Pereira, C.; Teixeira, J.P.; de Oliveira Fernandes, E. Indoor air quality in Portuguese schools: Levels and sources of pollutants. Indoor Air 2015 , 26 , 526–537. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Deng, S.; Lau, J. Seasonal variations of indoor air quality and thermal conditions and their correlations in 220 classrooms in the Midwestern United States. Build. Environ. 2019 , 157 , 79–88. [ Google Scholar ] [ CrossRef ]
• Twardella, D.; Matzen, W.; Lahrz, T.; Burghardt, R.; Spegel, H.; Hendrowarsito, L.; Frenzel, A.; Fromme, H. Effect of classroom air quality on students’ concentration: Results of a cluster-randomized cross-over experimental study. Indoor Air 2012 , 22 , 378–387. [ Google Scholar ] [ CrossRef ]
• Durán-Narucki, V. School building condition, school attendance, and academic achievement in New York City public schools: A mediation model. J. Environ. Psychol. 2008 , 28 , 278–286. [ Google Scholar ] [ CrossRef ]
• Berman, J.D.; McCormack, M.C.; Koehler, K.A.; Connolly, F.; Clemons-Erby, D.; Davis, M.F.; Gummerson, C.; Leaf, P.J.; Jones, T.D.; Curriero, F.C. School environmental conditions and links to academic performance and absenteeism in urban, mid-Atlantic public schools. Int. J. Hyg. Environ. Health 2018 , 221 , 800–808. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Evans, G.W.; Yoo, M.J.; Sipple, J. The ecological context of student achievement: School building quality effects are exacerbated by high levels of student mobility. J. Environ. Psychol. 2010 , 30 , 239–244. [ Google Scholar ] [ CrossRef ]
• Crampton, F.E. Spending on school infrastructure: Does money matter? J. Educ. Adm. 2009 , 47 , 305–322. [ Google Scholar ] [ CrossRef ]
• Haverinen-Shaughnessy, U.; Shaughnessy, R.J.; Cole, E.C.; Toyinbo, O.; Moschandreas, D.J. An assessment of indoor environmental quality in schools and its association with health and performance. Build. Environ. 2015 , 93 , 35–40. [ Google Scholar ] [ CrossRef ]
• Shaughnessy, R.; Haverinen-Shaughnessy, U.; Nevalainen, A.; Moschandreas, D. A preliminary study on the association between ventilation rates in classrooms and student performance. Indoor Air 2007 , 16 , 465–468. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Godwin, C.; Batterman, S. Indoor air quality in Michigan schools. Indoor Air 2007 , 17 , 109–121. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mendell, M.; Ekaterina, E.; Davies, M.; Spears, M.; Lobscheid, A.; Fisk, W.; Apte, M. Association of Classroom Ventilation with Reduced Illness Absence: A Prospective Study in California Elementary Schools. Indoor Air 2013 , 23 , 515–528. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Daisey, J.M.; Angell, W.J.; Apte, M.G. Indoor air quality, ventilation and health symptoms in schools: An analysis of existing information. Indoor Air 2003 , 13 , 53–64. [ Google Scholar ] [ CrossRef ]
• Haverinen-Shaughnessy, U.; Shaughnessy, R. Effects of Classroom Ventilation Rate and Temperature on Students’ Test Scores. PLoS ONE 2015 , 10 , e0136165. [ Google Scholar ] [ CrossRef ]
• O’Neill, D.; Oates, A. The Impact of School Facilities on Student Achievement, Behavior, Attendance, and Teacher Turnover Rate in Central Texas Middle Schools. Educ. Facil. Plan. 2001 , 36 , 14–22. [ Google Scholar ]
• Simons, E.; Hwang, S.-A.; Fitzgerald, E.F.; Kielb, C.; Lin, S. The Impact of School Building Conditions on Student Absenteeism in Upstate New York. Am. J. Public Health 2010 , 100 , 1679–1686. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Toyinbo, O.; Shaughnessy, R.; Turunen, M.; Putus, T.; Metsämuuronen, J.; Kurnitski, J.; Haverinen-Shaughnessy, U. Building characteristics, indoor environmental quality, and mathematics achievement in Finnish elementary schools. Build. Environ. 2016 , 104 , 114–121. [ Google Scholar ] [ CrossRef ]
• Wargocki, P.; Wyon, D.P. Providing better thermal and air quality conditions in school classrooms would be cost-effective. Build. Environ. 2013 , 59 , 581–589. [ Google Scholar ] [ CrossRef ]
• Crosby, K.; Metzger, A.B. Powering Down: A Toolkit for Behavior-Based Energy Conservation in K-12 Schools ; Technical Report; U.S. Green Building Council (USGBC): Washington, DC, USA, 2012. [ Google Scholar ]
• Economidou, M.; Atanasiu, B.; Despret, C.; Maio, J.; Nolte, I.; Rapf, O.; Laustsen, J.; Ruyssevelt, P.; Staniaszek, D.; Strong, D.; et al. Europe’s Buildings under the Microscope: A Country-by-Country Review of the Energy Performance of Buildings ; Technical Report; Buildings Performance Institute Europe (BPIE): Brussels, Belgium, 2011; ISBN 9789491143014. [ Google Scholar ]
• Shan, X.; Melina, A.N.; Yang, E.-H. Impact of indoor environmental quality on students’ wellbeing and performance in educational building through life cycle costing perspective. J. Clean. Prod. 2018 , 204 , 298–309. [ Google Scholar ] [ CrossRef ]
• Reed, R.H. The heating and ventilation of the mansfield schools and churches: A Lecture delivered before the Mansfield Lyceum, February 13, 1889. J. Am. Med. Assoc. 1889 , XII , 469–478. [ Google Scholar ] [ CrossRef ]
• Reed, R.H. Original articles. original investigations on the heating and ventilation of school buildings: Read in the Section of State Medicine, at the Forty-second Annual Meeting of the American Medical Association, at Washington, D.C.; May, 1891. J. Am. Med. Assoc. 1891 , XVII , 389–396. [ Google Scholar ] [ CrossRef ]
• Wheatley, J. The ventilation of schools. Public Health 1894 , 7 , 373–374. [ Google Scholar ] [ CrossRef ]
• Rovira, C.; Codina, L.; Guerrero-Solé, F.; Lopezosa, C. Ranking by Relevance and Citation Counts, a Comparative Study: Google Scholar, Microsoft Academic, WoS and Scopus. Futur. Internet 2019 , 11 , 202. [ Google Scholar ] [ CrossRef ]
• U.S. Environmental Protection Agency. Sustainability and the ROE. 2023. Available online: https://www.epa.gov/report-environment/sustainability-and-roe (accessed on 20 February 2024).
• Center for Green Schools. Leed certification for schools. 2023. Available online: https://centerforgreenschools.org/about/leed-certification-schools (accessed on 20 February 2024).
• The Collaborative for High Performance Schools (CHPS). 2021. Available online: https://chps.net/ (accessed on 20 February 2024).
• Bre Group. BREEAM—Education. 2022. Available online: https://bregroup.com/asset_type/breeam-education/ (accessed on 20 February 2024).
• Green building council Australia. Green Star—Education v1. 2008. Available online: https://www.gbca.org.au/uploads/226/1762/Factsheet_Educationv1.pdf (accessed on 20 February 2024).
• Kosasih, A. WELL v2 and Educational Spaces: Promoting Healthy Schools. 2018. Available online: https://resources.wellcertified.com/articles/well-v2-and-educational-spaces-promoting-healthy-schools/ (accessed on 20 February 2024).
• Brink, H.W.; Krijnen, W.P.; Loomans, M.G.L.C.; Mobach, M.P.; Kort, H.S.M. Positive effects of indoor environmental conditions on students and their performance in higher education classrooms: A between-groups experiment. Sci. Total Environ. 2023 , 869 , 161813. [ Google Scholar ] [ CrossRef ]
• Choi, H.-H.; van Merriënboer, J.J.G.; Paas, F. Effects of the Physical Environment on Cognitive Load and Learning: Towards a New Model of Cognitive Load. Educ. Psychol. Rev. 2014 , 26 , 225–244. [ Google Scholar ] [ CrossRef ]
• Brink, H.W.; Loomans, M.G.L.C.; Mobach, M.P.; Kort, H.S.M. Classrooms’ indoor environmental conditions affecting the academic achievement of students and teachers in higher education: A systematic literature review. Indoor Air 2021 , 31 , 405–425. [ Google Scholar ] [ CrossRef ]
• Kim, J.; de Dear, R. Nonlinear relationships between individual IEQ factors and overall workspace satisfaction. Build. Environ. 2012 , 49 , 33–40. [ Google Scholar ] [ CrossRef ]
• Xiong, L.; Huang, X.; Li, J.; Mao, P.; Wang, X.; Wang, R.; Tang, M. Impact of Indoor Physical Environment on Learning Efficiency in Different Types of Tasks: A 3 × 4 × 3 Full Factorial Design Analysis. Int. J. Environ. Res. Public Health 2018 , 15 , 1256. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Calderón-Garcidueñas, L.; Mora-Tiscareño, A.; Ontiveros, E.; Gómez-Garza, G.; Barragán-Mejía, G.; Broadway, J.; Chapman, S.; Valencia-Salazar, G.; Jewells, V.; Maronpot, R.R.; et al. Air polldogs. Brain Cogn. 2008 , 68 , 117–127. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gardin, T.N.; Requia, W.J. Air quality and individual-level academic performance in Brazil: A nationwide study of more than 15 million students between 2000 and 2020. Environ. Res. 2023 , 226 , 115689. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Duque, V.; Gilraine, M. Coal use, air pollution, and student performance. J. Public Econ. 2022 , 213 , 104712. [ Google Scholar ] [ CrossRef ]
• Kabirikopaei, A.; Lau, J.; Nord, J.; Bovaird, J. Identifying the K-12 classrooms’ indoor air quality factors that affect student academic performance. Sci. Total Environ. 2021 , 786 , 147498. [ Google Scholar ] [ CrossRef ]
• Requia, W.J.; Saenger, C.C.; Cicerelli, R.E.; Monteiro de Abreu, L.; Cruvinel, V.R.N. Air quality around schools and school-level academic performance in Brazil. Atmos. Environ. 2022 , 279 , 119125. [ Google Scholar ] [ CrossRef ]
• Guo, H.; Morawska, L.; He, C.; Zhang, Y.L.; Ayoko, G.; Cao, M. Characterization of particle number concentrations and PM2.5 in a school: Influence of outdoor air pollution on indoor air. Environ. Sci. Pollut. Res. 2010 , 17 , 1268–1278. [ Google Scholar ] [ CrossRef ]
• Richmond-Bryant, J.; Saganich, C.; Bukiewicz, L.; Kalin, R. Associations of PM2.5 and black carbon concentrations with traffic, idling, background pollution, and meteorology during school dismissals. Sci. Total Environ. 2009 , 407 , 3357–3364. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rivas, I.; Viana, M.; Moreno, T.; Pandolfi, M.; Amato, F.; Reche, C.; Bouso, L.; Àlvarez-Pedrerol, M.; Alastuey, A.; Sunyer, J.; et al. Child exposure to indoor and outdoor air pollutants in schools in Barcelona, Spain. Environ. Int. 2014 , 69 , 200–212. [ Google Scholar ] [ CrossRef ]
• Martínez-Lazcano, J.; González Guevara, E.; Rubio, C.; Franco-Pérez, J.; Custodio, V.; Hernández-Cerón, M.; CARLOS, L.; Paz, C. The effects of ozone exposure and associated injury mechanisms on the central nervous system. Rev. Neurosci. 2013 , 24 , 337–352. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Forns, J.; Dadvand, P.; Esnaola, M.; Alvarez-Pedrerol, M.; López-Vicente, M.; Garcia-Esteban, R.; Cirach, M.; Basagaña, X.; Guxens, M.; Sunyer, J. Longitudinal association between air pollution exposure at school and cognitive development in school children over a period of 3.5 years. Environ. Res. 2017 , 159 , 416–421. [ Google Scholar ] [ CrossRef ]
• Benka-Coker, W.; Young, B.; Oliver, S.; Schaeffer, J.W.; Manning, D.; Suter, J.; Cross, J.; Magzamen, S. Sociodemographic variations in the association between indoor environmental quality in school buildings and student performance. Build. Environ. 2021 , 206 , 108390. [ Google Scholar ] [ CrossRef ]
• Choi, N.; Yamanaka, T.; Takemura, A.; Kobayashi, T.; Eto, A.; Hirano, M. Impact of indoor aroma on students’ mood and learning performance. Build. Environ. 2022 , 223 , 109490. [ Google Scholar ] [ CrossRef ]
• Ryan, I.; Deng, X.; Thurston, G.; Khwaja, H.; Romeiko, X.; Zhang, W.; Marks, T.; Yu, F.; Lin, S. Measuring students’ exposure to temperature and relative humidity in various indoor environments and across seasons using personal air monitors. Hyg. Environ. Health Adv. 2022 , 4 , 100029. [ Google Scholar ] [ CrossRef ]
• Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre). EPPI-Centre Methods for Conducting Systematic Reviews ; EPPI-Centre, Social Science Research Unit, Institute of Education, University of London: London, UK, 2007. [ Google Scholar ]
• Harden, A.; Thomas, J. Methodological Issues in Combining Diverse Study Types in Systematic Reviews. Int. J. Soc. Res. Methodol. 2005 , 8 , 257–271. [ Google Scholar ] [ CrossRef ]
• Lamé, G. Systematic Literature Reviews: An Introduction. In Proceedings of the Design Society: International Conference on Engineering Design, Delft, The Netherlands, 5–8 August 2019; Volume 1. [ Google Scholar ] [ CrossRef ]
• MacLure, M. ‘Clarity bordering on stupidity’: Where’s the quality in systematic review? J. Educ. Policy 2005 , 20 , 393–416. [ Google Scholar ] [ CrossRef ]
• Higgins, J.P.T.; Green, S. (Eds.) Cochrane Handbook for Systematic Reviews of Interventions ; The Cochrane Collaboration: London, UK, 2009. [ Google Scholar ]
• Chadegani, A.A.; Salehi, H.; Yunus, M.M.; Farhadi, H.; Fooladi, M.; Farhadi, M.; Ebrahim, N.A. A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases. Asian Soc. Sci. 2013 , 9 , 18–26. [ Google Scholar ] [ CrossRef ]
• Chen, C.; Hu, Z.; Liu, S.; Tseng, H. Emerging trends in regenerative medicine: A scientometric analysis in CiteSpace. Expert Opin. Biol. Ther. 2012 , 12 , 593–608. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chen, C. Searching for intellectual turning points: Progressive knowledge domain visualization. Proc. Natl. Acad. Sci. USA 2004 , 101 (Suppl. S1), 5303–5310. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chen, C. CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature. J. Am. Soc. Inf. Sci. Technol. 2006 , 57 , 359–377. [ Google Scholar ] [ CrossRef ]
• Chen, C.; Song, M. Visualizing a Field of Research: A Methodology of Systematic Scientometric Reviews. PLoS ONE 2019 , 14 , e0223994. [ Google Scholar ] [ CrossRef ]
• Chen, C. A Glimpse of the First Eight Months of the COVID-19 Literature on Microsoft Academic Graph: Themes, Citation Contexts, and Uncertainties. Front. Res. Metr. Anal. 2020 , 5 , 607286. [ Google Scholar ] [ CrossRef ]
• Alshamrani, O.S.; Galal, K.; Alkass, S. Integrated LCA–LEED sustainability assessment model for structure and envelope systems of school buildings. Build. Environ. 2014 , 80 , 61–70. [ Google Scholar ] [ CrossRef ]
• Ji, C.; Hong, T.; Jeong, J.; Kim, J.; Lee, M.; Jeong, K. Establishing environmental benchmarks to determine the environmental performance of elementary school buildings using LCA. Energy Build. 2016 , 127 , 818–829. [ Google Scholar ] [ CrossRef ]
• Brás, A.; Gomes, V. LCA implementation in the selection of thermal enhanced mortars for energetic rehabilitation of school buil ings. Energy Build. 2015 , 92 , 1–9. [ Google Scholar ] [ CrossRef ]
• Pachta, V.; Giourou, V. Comparative Life Cycle Assessment of a Historic and a Modern School Building, Located in the City of Naoussa, Greece. Sustainability 2022 , 14 , 4216. [ Google Scholar ] [ CrossRef ]
• Gamarra, A.R.; Herrera, I.; Lechón, Y. Assessing sustainability performance in the educational sector. A High Sch. Case Study. Sci. Total Environ. 2019 , 692 , 465–478. [ Google Scholar ] [ CrossRef ]
• Gamarra, A.R.; Istrate, I.R.; Herrera, I.; Lago, C.; Lizana, J.; Lechon, Y. Energy and water consumption and carbon footprint of school buildings in hot climate conditions. Results Life Cycle Assess. J. Clean. Prod. 2018 , 195 , 1326–1337. [ Google Scholar ] [ CrossRef ]
• Munoz, P.; Morales, P.; Letelier, V.; Munoz, L.; Morad, D. Implications of Life Cycle Energy Assessment of a new school building, regarding the nearly Zero Energy Buildings targets in EU: A case OF study. Sustain. Cities Soc. 2017 , 32 , 142–152. [ Google Scholar ] [ CrossRef ]
• Su, S.; Zhu, C.; Li, X. A dynamic weighting system considering temporal variations using the DTT approach in LCA of buildings. J. Clean. Prod. 2019 , 220 , 398–407. [ Google Scholar ] [ CrossRef ]
• Li, X.; Su, S.; Zhang, Z.; Kong, X. An integrated environmental and health performance quantification model for pre-occupancy phase of buildings in China. Environ. Impact Assess. Rev. 2017 , 63 , 1–11. [ Google Scholar ] [ CrossRef ]
• Su, S.; Wang, Q.; Han, L.; Hong, J.; Liu, Z. BIM-DLCA: An integrated dynamic environmental impact assessment model for buildings. Build. Environ. 2020 , 183 , 107218. [ Google Scholar ] [ CrossRef ]
• Su, S.; Zhang, H.; Zuo, J. Assessment models and dynamic variables for dynamic life cycle assessment of buildings: A review. Environ. Sci. Pollut. Res. 2021 , 28 , 26199–26214. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lee, S.; Yu, J.; Jeong, D. BIM acceptance model in construction organizations. J. Manag. Eng. 2015 , 31 , 04014048. [ Google Scholar ] [ CrossRef ]
• Gheisari, M.; Irizarry, J. Investigating human and technological requirements for successful implementation of a BIM-based mobile augmented reality environment in facility management practices. Facilities 2016 , 34 , 69–84. [ Google Scholar ] [ CrossRef ]
• Wang, J.; Wang, X.; Wang, J.; Yung, P.; Jun, G. Engagement of facilities management in design stage through BIM: Framework and a case study. Adv. Civ. Eng. 2013 , 2013 , 189105. [ Google Scholar ] [ CrossRef ]
• Liu, Y.; van Nederveen, S.; Wu, C.; Hertogh, M. Sustainable infrastructure design framework through integration of rating systems and building information modeling. Adv. Civ. Eng. 2018 , 2018 , 8183536. [ Google Scholar ] [ CrossRef ]
• Zhang, J.; Issa, R. Collecting fire evacuation performance data using BIM-based immersive serious games for performance-based fire safety design. In Proceedings of the 2015 International Workshop on Computing in Civil Engineering, Austin, TX, USA, 21–23 June 2015; pp. 612–619. [ Google Scholar ]
• Vaughan, E. “Elementary School”, Whole Building Design Guide ; National Institute of Building Sciences: Washington, DC, USA, 2017; Available online: https://www.wbdg.org/buildingtypes/education-facilities/elementary-school (accessed on 20 February 2024).
• Eastman, C.; Teicholz, P.; Sacks, R.; Liston, K. BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers, and Contractors , 2nd ed.; John Wiley and Sons: Hoboken, NJ, USA, 2011. [ Google Scholar ]
• Bynum, P.; Issa, R.R.A.; Olbina, S. Building information modeling in support of sustainable design and construction. J. Constr. Eng. Manag. 2013 , 139 , 24–34. [ Google Scholar ] [ CrossRef ]
• Lee, K.; Choo, S. A hierarchy of architectural design elements for energy saving of tower buildings in Korea using green BIM simulation. Adv. Civ. Eng. 2018 , 2018 , 7139196. [ Google Scholar ] [ CrossRef ]
• Zhuang, D.; Zhang, X.; Lu, Y.; Wang, C.; Jin, X.; Zhou, X.; Shi, X. A performance data integrated BIM framework for building life-cycle energy efficiency and environmental optimization design. Autom. Constr. 2021 , 127 , 103712. [ Google Scholar ] [ CrossRef ]
• Neasden Primary School. Application of BIM in Energy Management of Individual Departments Occupying University Facilities ; Neasden Primary School: Hull, UK, 2019; Available online: https://neasdenprimary.org.uk/school-new-build/ (accessed on 20 February 2024).
• Konings, K.D.; Bovill, C.; Woolner, P. Towards an interdisciplinary model of practice for participatory building design in education. Eur. J. Educ. 2017 , 52 , 306–317. [ Google Scholar ] [ CrossRef ]
• Van Merrienboer, J.J.G.; McKenney, S.; Cullinan, D.; Heuer, J. Aligning pedagogy with physical learning spaces. Eur. J. Educ. 2017 , 52 , 253–267. [ Google Scholar ] [ CrossRef ]
• Succar, B. Building information modelling framework: A research and delivery foundation for industry stakeholders. Autom. Constr. 2009 , 18 , 357–375. [ Google Scholar ] [ CrossRef ]
• Volk, R.; Stengel, J.; Schultmann, F. Building Information Modeling (BIM) for existing buildings—Literature review and future needs. Autom. Constr. 2014 , 38 , 109–127. [ Google Scholar ] [ CrossRef ]
• Azhar, S. Building Information Modeling (BIM): Trends, Benefits, Risks, and Challenges for the AEC Industry. Leadersh. Manag. Eng. 2011 , 11 , 241–252. [ Google Scholar ] [ CrossRef ]
• Gaur, A.; Scotney, B.; Parr, G.; McClean, S. Smart City Architecture and its Applications Based on IoT. Procedia Comput. Sci. 2015 , 52 , 1089–1094. [ Google Scholar ] [ CrossRef ]
• Sethi, P.; Sarangi, S.R. Internet of Things: Architectures, Protocols, and Applications. J. Electr. Comput. Eng. 2017 , 2017 , 1–25. [ Google Scholar ] [ CrossRef ]
• Li, S.; Xu, L.D.; Zhao, S. The internet of things: A survey. Inf. Syst. Front. 2015 , 17 , 243–259. [ Google Scholar ] [ CrossRef ]
• Amaxilatis, D.; Akrivopoulos, O.; Mylonas, G.; Chatzigiannakis, I. An IoT-Based Solution for Monitoring a Fleet of Educational Buildings Focusing on Energy Efficiency. Sensors 2017 , 17 , 2296. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Hossain, M.; Weng, Z.; Schiano-Phan, R.; Scott, D.; Lau, B. Application of IoT and BEMS to Visualise the Environmental Performance of an Educational Building. Energies 2020 , 13 , 4009. [ Google Scholar ] [ CrossRef ]
• Martínez, I.; Zalba, B.; Trillo-Lado, R.; Blanco, T.; Cambra, D.; Casas, R. Internet of Things (IoT) as Sustainable Development Goals (SDG) Enabling Technology towards Smart Readiness Indicators (SRI) for University Buildings. Sustainability 2021 , 13 , 7647. [ Google Scholar ] [ CrossRef ]
• Kamel, S.; Jamal, A.; Omri, K.; Khyyat, M. An IoT-based Fire Safety Management System for Educational Buildings: A Case Study. Int. J. Adv. Comput. Sci. Appl. 2022 , 13 , 765–771. [ Google Scholar ] [ CrossRef ]
• Paganelli, F.; Mylonas, G.; Cuffaro, G.; Nesi, I. Experiences from Using Gamification and IoT-Based Educational Tools in High Schools Towards Energy Savings. In Ambient Intelligence. AmI 2019 ; Lecture Notes in Computer Science; Chatzigiannakis, I., De Ruyter, B., Mavrommati, I., Eds.; Springer: Cham, Switzerland, 2019; Volume 11912. [ Google Scholar ] [ CrossRef ]
• Mylonas, G.; Amaxilatis, D.; Chatzigiannakis, I.; Paganelli, A.A.F. Enabling Sustainability and Energy Awareness in Schools Based on IoT and Real-World Data. IEEE Pervasive Comput. 2018 , 17 , 53–63. [ Google Scholar ] [ CrossRef ]
• Jia, M.; Komeily, A.; Wang, Y.; Srinivasan, R.S. Adopting Internet of Things for the development of smart buildings: A review of enabling technologies and applications. Autom. Constr. 2019 , 101 , 111–126. [ Google Scholar ] [ CrossRef ]
• Atzori, L.; Iera, A.; Morabito, G. The Internet of Things: A survey. Comput. Netw. 2010 , 54 , 2787–2805. [ Google Scholar ] [ CrossRef ]
• Gubbi, J.; Buyya, R.; Marusic, S.; Palaniswami, M. Internet of Things (IoT): A vision, architectural elements, and future directions. Future Gener. Comput. Syst. 2013 , 29 , 1645–1660. [ Google Scholar ] [ CrossRef ]
• Hoda, I.; Elsayed, M.S.; Wael, S.M.; Abdou, H.M. Functional analysis as a method on sustainable building design: A case study in educational buildings implementing the triple bottom line. Alex. Eng. J. 2023 , 62 , 63–73. [ Google Scholar ] [ CrossRef ]
• Peng, C. Calculation of a building’s life cycle carbon emissions based on Ecotect and building information modeling. J. Clean. Prod. 2016 , 112 , 453–465. [ Google Scholar ] [ CrossRef ]
• Schwab, K. The Fourth Industrial Revolution ; Penguin Books Limited: London, UK, 2017; Available online: https://books.google.lt/books?id=ST_FDAAAQBAJ (accessed on 20 June 2024).

Click here to enlarge figure

Share and Cite

Vestfal, P.; Seduikyte, L. Systematic Review of Factors Influencing Students’ Performance in Educational Buildings: Focus on LCA, IoT, and BIM. Buildings 2024 , 14 , 2007. https://doi.org/10.3390/buildings14072007

Vestfal P, Seduikyte L. Systematic Review of Factors Influencing Students’ Performance in Educational Buildings: Focus on LCA, IoT, and BIM. Buildings . 2024; 14(7):2007. https://doi.org/10.3390/buildings14072007

Vestfal, Paulius, and Lina Seduikyte. 2024. "Systematic Review of Factors Influencing Students’ Performance in Educational Buildings: Focus on LCA, IoT, and BIM" Buildings 14, no. 7: 2007. https://doi.org/10.3390/buildings14072007

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals