systematic review critique example

Appraising systematic reviews

Systematic reviews may or may not include a meta-analysis of the primary RCTs identified. Although systematic reviews of RCTs with meta-analysis are often said to provide the most compelling evidence of effectiveness and causality, not all systematic reviews are of the highest methodological quality. 

There are numerous checklists available. The SR toolbox  is an online catalogue providing summaries and links to the available guidance and software for each stage of the systematic review process including critical appraisal. Examples for systematic reviews include:

• AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• CASP systematic review checklist.

The following framework forms an initial memory aid for assessing the quality of a systematic review.

• Framework for assessing systematic reviews

Content created by BMJ Knowledge Centre

• What is EBM?
• How to clarify a clinical question
• Searching for evidence
• Search strategies
• Case study of a search
• Appraise the evidence
• Evidence synthesis
• What is GRADE?
• Understanding statistics: BMJ Learning modules
• Understanding statistics: risk

EBM Toolkit home

Learn, Practise, Discuss, Tools

• Search Menu
• Advance Articles
• Specialty Certificate Examination Cases
• Cover Archive
• Virtual Issues
• Trending Articles
• Author Guidelines
• Reviewer Guidelines
• Submission Site
• Open Access Options
• Reasons to Publish
• Self-Archiving Policy
• About Clinical and Experimental Dermatology
• About British Association of Dermatologists
• Editorial Board
• Advertising & Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Discussion and conclusions, learning points, funding sources, data availability, ethics statement, cpd questions, instructions for answering questions.

• < Previous

How to critically appraise a systematic review: an aide for the reader and reviewer

Conflicts of interest H.W. founded the Cochrane Skin Group in 1987 and was coordinator editor until 2018. The other authors declare they have no conflicts of interest.

• Article contents
• Figures & tables
• Supplementary Data

John Frewen, Marianne de Brito, Anjali Pathak, Richard Barlow, Hywel C Williams, How to critically appraise a systematic review: an aide for the reader and reviewer, Clinical and Experimental Dermatology , Volume 48, Issue 8, August 2023, Pages 854–859, https://doi.org/10.1093/ced/llad141

• Permissions Icon Permissions

The number of published systematic reviews has soared rapidly in recent years. Sadly, the quality of most systematic reviews in dermatology is substandard. With the continued increase in exposure to systematic reviews, and their potential to influence clinical practice, we sought to describe a sequence of useful tips for the busy clinician reader to determine study quality and clinical utility. Important factors to consider when assessing systematic reviews include: determining the motivation to performing the study, establishing if the study protocol was prepublished, assessing quality of reporting using the PRISMA checklist, assessing study quality using the AMSTAR 2 critical appraisal checklist, assessing for evidence of spin, and summarizing the main strengths and limitations of the study to determine if it could change clinical practice. Having a set of heuristics to consider when reading systematic reviews serves to save time, enabling assessment of quality in a structured way, and come to a prompt conclusion of the merits of a review article in order to inform the care of dermatology patients.

A systematic review aims to systematically and transparently summarize the available data on a defined clinical question, via a rigorous search for studies, a critique of the quality of included studies and a qualitative and/or quantitative synthesis. 1 Systematic reviews are at the top of the pyramid in most evidence hierarchies for informing evidence-based healthcare as they are considered of greater validity and clinical applicability than those study types lower down, such as case series or individual trials. 2

A good systematic review should provide an unbiased overview of studies to inform clinical practice. Systematic reviews can reconcile apparently conflicting results, add precision to estimating smaller treatment effects, highlight the evidence’s limitations and biases and identify research gaps. Guidelines are available to assist systematic reviewers to transparently report why the review was done, the authors’ methods and findings via the PRISMA checklist. 3

The sharp rise in systematic review publications over time raises concern that the majority are unnecessary, misleading and/or conflicted. 4 A review of dermatology systematic reviews noted that 93% failed to report at least one PRISMA checklist item. 5 Another review of a random sample of 140/732 dermatology systematic reviews in 2017 found 90% were low quality. 6 Some improvements have occurred: reporting standards compliance has improved slightly (between 2013 and 2017), 5 and several leading dermatology journals including the British Journal of Dermatology have changed editorial policies, mandating authors to preregister review protocols.

Given the surge in poor-quality systematic review publications, we sought to describe a checklist of seven practical tips from the authors’ collective experience of writing and critically appraising systematic reviews, hoping that they will assist busy clinicians to critically appraise systematic reviews both as manuscript reviewers and as readers and research users.

Read the abstract to develop a sense of the subject.

What was the motivation for completing the review?

Has the review protocol been published and have changes been made to it.

Review the reporting quality .

Review the quality of the article and the depth of the review question.

Consider the authors’ interpretation and assess for spin .

Summarize and come to a position .

Read the abstract to develop a sense of the subject

From the abstract, use the PICO (population, intervention, comparator and outcome) framework to establish if the subject, intervention and outcomes are relevant to clinical practice. Is the review question clear and appropriate?

Inspect the authors’ conflicts of interest and funding sources. Self-disclosed financial conflicts are often insufficiently described or not declared at all. 7 If you suspect conflicts for authors with no stated conflicts, briefly searching the senior authors’ names on PubMed, or the Open Payments website (for US authors) may reveal hidden conflicts. 8 Is the motivation for the systematic review justified in the introduction? Can new insights be formed by combining studies? If the systematic review is an update, what new available data justifies this? Search for similar recent systematic reviews (which may have been omitted intentionally). Is it a redundant duplicate review that adds little new useful information? 9 Has the author recently published reviews on similar subjects? Salami publications refer to authors chopping up a topic into smaller pieces to obtain maximum publications. 10

Search PROSPERO for publication of the review protocol. 11 A prepublished review protocol in a publicly accessible site offers reassurance that the systematic review followed a clear plan with prespecified PICO elements. Put bluntly, it reduces authors’ opportunity for deception by selective analysis and highlighting of results that are more likely to get published. If a protocol is found, assess deviation from this protocol and justification, if present. Protocol registration allows improved PRISMA reporting. 12 A registered protocol with reporting of deviations allows the reader to judge whether any modifications are justified, for example adjusting for unexpected challenges during analysis. 10

Review the reporting quality

Look for supplementary material detailing the PRISMA checklist. Commonly under-reported PRISMA items include protocol and registration, risk of bias across studies, risk of bias in individual studies, the data collection process and review objectives. 5 Adequate reporting quality using PRISMA does not necessarily indicate the review is clinically useful; however, it allows the reader to assess the study’s utility (see Table 1 ). Additional assessments of review quality are described below.

The relationship between systematic review reporting quality and study quality a

Adapted with permission from Williams. 21

Review the quality of the article and the depth of the review question

Distinct from quality of reporting completeness, assessing the review's quality allows for assessment of the overall clinical meaningfulness of the results. Does the PICO make sense in respect to this? The AMSTAR 2 critical appraisal instrument is useful in determining quantitative systematic review quality. 13 This checklist marks the key aspects of a systematic review and computes an outcome of the review quality. 14 If meta-analysis was performed, did the authors justify and use appropriate methods for statistical combination of results? Were weighted techniques used to combine results and adjusted for heterogeneity, if present? If heterogeneity was present, were sources of this investigated? Did authors assess the potential impact of the individual study’s risk of bias (RoB) and perform analysis to investigate the impact of RoB on the summary estimates of affect? See Table 2 for an example of a completed AMSTAR 2 checklist on a recently published poor-quality systematic review. 15

An example of assessment of the quality of a systematic review (Drake et al. ) 15 using the AMSTAR 2 checklist an explanation of which can be found at https://amstar.ca/Amstar-2.php

N/A, not applicable; PICO, population, intervention, comparator and outcome. a Denotes AMSTAR 2 critical domain. The overall confidence in the results of the review is dependent on such critical domains. When one critical domain is not satisfied, the confidence is rated as ‘low’ and the review may not provide an accurate and comprehensive summary of the available studies that address the question of interest. When more than one critical domain are not satisfied, the confidence in the results of the review is rated as ‘critically low’ and the review should not be relied on to provide an accurate and comprehensive summary of the available studies.

Quality checklists for assessment of qualitative research include Consolidated Criteria for Reporting Qualitative research (COREQ), Standards for Reporting Qualitative Research (SRQR) and Critical Appraisal Skills Programme (CASP). 16 Such checklists aim to improve identification of high-quality qualitative research in journal articles, as well as acting as a guide for conducting research. 16

Consider the authors’ interpretation and assess for spin

Spin is a distorted interpretation of results. This manifests itself in studies as (i) misleading reporting, (ii) misleading interpretation, and (iii) inappropriate extrapolation. 14 Are the conclusion’s clinical practice recommendations not supported by the studies’ findings? Is the title misleading? Is there selective reporting? These are the three most severe forms of spin occurring in systematic reviews. 17

Summarize and come to a position

Summarize the reviews main positives and negatives and establish if there is sufficient quality to merit changing clinical practice, or are fatal flaws present that nullify the review’s clinical utility? Consider internal validity (are the results true?) and external validity (are the results applicable to my patient group?). When applying the systematic review results to a particular patient, it may help to consider these points: (i) how similar are the study participants to my patient?; (ii) do the outcomes make sense to me?; (iii) what was the magnitude of treatment benefit? – work out the number needed to treat; 18 (iv) what are the adverse events?; and (v) what are my patient's values and preferences? 19

Although systematic reviews have potential for summarizing evidence for dermatological interventions in a systematic and unbiased way, the rapid expansion of poorly reported and poor-quality reviews (Table 3 ) is regrettable. We do not claim our checklist items (Table 4 ) are superior to other checklists such as those suggested by CASP, 20 but they are based on the practical experience of critical appraisal of dermatology systematic reviews conducted by the authors.

The top seven ‘sins’ of dermatology systematic reviews a

Adapted with permission from Williams. 10

Checklist of questions, considerations and tips for critical appraisal of systematic reviews

NNT, number needed to treat.

Considering each question suggested in our checklist when faced with yet another systematic review draws a timely conclusion on its quality and application to clinical practice, when acting as a reviewer or reader. Although the checklist may sound exhaustive and time-consuming, we recommend cutting it short if there are major red flags early on, such as absence of a protocol or assessment of RoB. Given the growing number of systematic reviews, having an efficient and succinct aide for appraising articles saves the reader time and energy, while simplifying the decision regarding what merits a change in clinical practice. Our intention is not to criticize others’ well-intentioned efforts, but to improve standards of reliable evidence to inform patient care.

Systematic reviews of randomized controlled trials offer one of the best methods to summarize the evidence surrounding therapeutic interventions for skin conditions.

The number of systematic reviews in the dermatology literature is increasing rapidly.

The quality of dermatology systematic reviews is generally poor.

We describe a checklist for the busy clinician or reviewer to consider when faced with a systematic review.

Key factors to consider include: determining the review motivation, establishing if the study protocol was prepublished, assessing quality of reporting and study quality using PRISMA, and AMSTAR 2 critical appraisal checklists, and assessing for evidence of spin.

Summarizing the main qualities and limitations of a systematic review will help to determine if the review is robust enough to potentially change clinical practice for patient benefit.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

No new data generated.

Ethical approval: not applicable. Informed consent: not applicable.

Moher D , Liberati A , Tetzlaff J et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann Intern Med 2009 ; 151 : 264 – 9 .

Google Scholar

Murad MH , Asi N , Alsawas M , Alahdab F . New evidence pyramid . Evid Based Med 2016 ; 21 : 125 – 7 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Ioannidis JP . The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses . Milbank Q 2016 ; 94 : 485 – 514 .

Croitoru DO , Huang Y , Kurdina A et al.  Quality of reporting in systematic reviews published in dermatology journals . Br J Dermatol 2020 ; 182 : 1469 – 76 .

Smires S , Afach S , Mazaud C et al.  Quality and reporting completeness of systematic reviews and meta-analyses in dermatology . J Invest Dermatol 2021 ; 141 : 64 – 71 .

Baraldi JH , Picozzo SA , Arnold JC et al.  A cross-sectional examination of conflict-of-interest disclosures of physician-authors publishing in high-impact US medical journals . BMJ Open 2022 ; 12 : e057598 .

Centers for Medicare & Medicaid Services . Open Payments Search Tool. About. Available at : https://openpaymentsdata.cms.gov/about (last accessed 22 April 2023).

Guelimi R , Afach S , Régnaux JP et al.  Overlapping network meta-analyses on psoriasis systemic treatments, an overview: quantity does not make quality . Br J Dermatol 2022 ; 187 : 29 – 41 .

Williams HC . Are dermatology systematic reviews spinning out of control? Dermatology 2021 ; 237 : 493 – 5 .

National Institute for Health and Care Research . About Prospero. Available at : https://www.crd.york.ac.uk/prospero/#aboutpage (last accessed 22 April 2023).

Barbieri JS , Wehner MR . Systematic reviews in dermatology: opportunities for improvement . Br J Dermatol 2020 ; 182 : 1329 – 30 .

Shea BJ , Reeves BC , Wells G et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both . BMJ 2017 ; 358 : j4008 .

AMSTAR . AMSTAR checklist. Available at : https://amstar.ca/Amstar_Checklist.php (last accessed 22 April 2023).

Drake L , Reyes-Hadsall S , Martinez J et al.  Evaluation of the safety and effectiveness of nutritional supplements for treating hair loss: a systematic review . JAMA Dermatol 2023 ; 159 : 79 – 86 .

Stenfors T , Kajamaa A , Bennett D . How to … assess the quality of qualitative research . Clin Teach 2020 ; 17 : 596 – 9 .

Yavchitz A , Ravaud P , Altman DG et al.  A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity . J Clin Epidemiol 2016 ; 75 : 56 – 65 .

Manriquez JJ , Villouta MF , Williams HC . Evidence-based dermatology: number needed to treat and its relation to other risk measures . J Am Acad Dermatol 2007 ; 56 : 664 – 71 .

Williams HC . Applying trial evidence back to the patient . Arch Dermatol 2003 ; 139 : 1195 – 200 .

CASP . CASP checklists. Available at : https://casp-uk.net/casp-tools-checklists/  (last accessed 22 April 2023).

Williams HC . Cars, CONSORT 2010, and clinical practice . Trials 2010 ; 11 : 33 .

Learning objective

To demonstrate up-to-date knowledge on assessing systematic reviews.

Which of the following critical appraisal checklists is useful for assessment of items that should be reported in a systematic review?

Which one of the following statements is correct?

The number of published systematic reviews in the dermatology literature is falling.

The quality of published dermatology systematic reviews is generally very good.

Publishing details of the PRISMA checklist in a systematic review indicates that the study quality is high.

External validity refers to the applicability of results to your patient group.

Internal validity refers to the applicability of results to your patient group.

Spin in systematic reviews can be described by which one of the following measures?

Authors declaring all conflicts of interest.

Title suggesting beneficial effect not supported by findings.

Adequate reporting of study limitations.

Conclusion formulating recommendations for clinical practice supported by findings.

Reporting a departure from study protocol that may modify interpretation of results.

PICO stands for which of the following.

PubMed, inclusion, comparator, outcome.

Population, items, comparator, outcome.

Population, intervention, context, observations.

Protocol, intervention, certainty, outcome.

Population, intervention, comparator, outcome.

Publication of a systematic review study protocol can be found at which source?

Cochrane Library.

ClinicalTrials.gov.

This learning activity is freely available online at https://oupce.rievent.com/a/TWWDCK

Users are encouraged to

Read the article in print or online, paying particular attention to the learning points and any author conflict of interest disclosures.

Reflect on the article.

Register or login online at https://oupce.rievent.com/a/TWWDCK and answer the CPD questions.

Complete the required evaluation component of the activity.

Once the test is passed, you will receive a certificate and the learning activity can be added to your RCP CPD diary as a self-certified entry.

This activity will be available for CPD credit for 5 years following its publication date. At that time, it will be reviewed and potentially updated and extended for an additional period.

Author notes

Email alerts, citing articles via.

• Recommend to Your Librarian
• Advertising and Corporate Services
• Journals Career Network

Affiliations

• Online ISSN 1365-2230
• Copyright © 2024 British Association of Dermatologists
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Asking a clear question
• Sources of evidence within a systematic review
• The hazards of "quick" searches
• Publication bias
• Data abstraction
• Pooling results
• Article Information

See More About

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Others Also Liked

• Download PDF
• X Facebook More LinkedIn

Bigby M , Williams H. Appraising Systematic Reviews and Meta-analyses. Arch Dermatol. 2003;139(6):795–798. doi:10.1001/archderm.139.6.795

Manage citations:

© 2024

• Permissions

Appraising Systematic Reviews and Meta-analyses

A systematic review is an overview that answers a specific clinical question and contains a thorough, unbiased search of the relevant literature, explicit criteria for assessing studies, and a structured presentation of the results. Many systematic reviews incorporate a meta-analysis, ie, a quantitative pooling of several similar studies to produce an overall summary of treatment effect. 1 , 2 Meta-analysis provides an objective and quantitative summary of evidence that is amenable to statistical analysis, 1 and it allows recognition of important treatment effects by combining the results of small trials that individually might have lacked the power to consistently demonstrate differences among treatments. Meta-analysis has been criticized for the discrepancies between its findings and those of large clinical trials. 3 - 6 The frequency of discrepancies ranges from 10% to 23% 3 and can often be explained by differences in treatment protocols or study populations or changes that occur over time. 3

In conducting a meta-analysis, the authors should recognize the importance of having clear objectives, explicit criteria for study selection, an assessment of the quality of included studies, and prior consideration of which studies to combine. These items are the esssential features of a systematic review. Meta-analyses that are not conducted within the context of a systematic review should be viewed with great caution. 7

A systematic review can be viewed as a scientific and systematic examination of the available evidence. A good systematic review will have explicitly stated objectives (the focused clinical question), materials (the relevant medical literature), and methods (the way studies are assessed and summarized). The steps taken during a systematic review are detailed in Table 1 .

Not all systematic reviews and meta-analyses are equal. A systematic review should be conducted in a manner that will include all of the relevant trials, minimize the introduction of bias, and synthesize the results to be as truthful and useful to clinicians as possible. A systematic review can only be as good as the clinical trials that it contains. The criteria used to critically appraise systematic reviews and meta-analyses 8 are listed in Table 2 . In general, these criteria are similar to the criteria used to appraise the individual studies that make up the systematic review. Detailed explanations of each criterion are available. 1

The validity criteria are designed to ensure that the systematic review is conducted in a manner that minimizes the introduction of bias. Like the "well-built clinical question" 9 for an individual study, a focused clinical question for a systematic review should clearly articulate the following 4 elements of the material under review: (1) the patient, group of patients, or problem being evaluated; (2) the intervention; (3) comparison interventions; and (4) specific outcomes. The patient populations in the reviewed studies should be similar to the actual population most likely to benefit from the review results. The interventions studied should be those commonly available in practice. Outcomes reported should be those that are most relevant to physicians and patients.

The overwhelming majority of systematic reviews involve therapy. Therefore, randomized, controlled, clinical trials should be used when available for systematic reviews of therapy because they are generally less susceptible to selection and information bias than other study designs. The quality of included trials is assessed using the criteria that are used to evaluate individual, randomized, controlled clinical trials. The quality criteria commonly used include concealed, random allocation; groups with similar known prognostic factors; equal treatment of groups; and inclusion of all trial patients in the results analysis (intent-to-treat design).

Randomized controlled trials are rarely a reliable source of identification of adverse reactions, unless those reactions are very common. Other evidence sources such as case-control studies, case reports, and postmarketing surveillance studies should therefore be examined. Systematic reviews of treatment efficacy should always include an assessment of common and serious adverse events to reach an informed and balanced decision about the utility of a treatment.

A sound systematic review can be performed only if most or all of the available data are examined. Simply performing a quick MEDLINE search using "clinical trial" as publication type is rarely adequate because complex and sensitive search strategies are needed to identify all potential trials and because some clinical trials will be missed if they are published in a journal not listed by MEDLINE. Potential sources for finding studies about treatment include the Cochrane Controlled Trials Registry, which is part of the Cochrane Library; MEDLINE; EMBASE; bibliographies of studies; review articles and textbooks; symposia proceedings; pharmaceutical companies; and direct communication with experts in the field.

The Cochrane Controlled Trials Registry, the largest and most complete database of clinical trials in the world, includes more than 300 000 controlled clinical trials. It is compiled through several complex searches of the MEDLINE and EMBASE databases and by hand searching many journals, a process that is quality controlled and monitored by the Cochrane Collaboration in Oxford, England. Hand searching journals to identify controlled clinical trials and randomized, controlled clinical trials was undertaken because members of the Cochrane Collaboration noticed that many trials were incorrectly classified in the MEDLINE database. As an example, Adetugbo and Williams 10 hand searched the Archives of Dermatology from 1990 through 1998 and identified 99 controlled clinical trials. Nineteen of these trials were not classified as controlled clinical trials in MEDLINE, and 11 trials that were not controlled clinical trials were misclassified as controlled clinical trials in MEDLINE. 10

MEDLINE is the National Library of Medicine's bibliographic database covering medicine, nursing, dentistry, veterinary medicine, the health care system, and the preclinical sciences. The MEDLINE file contains bibliographic citations and author abstracts from approximately 3900 current biomedical journals published in the United States and 70 foreign countries. The file contains approximately 9 million records dating back to 1966. 11

MEDLINE searches have inherent limitations that make their reliability less than ideal. 12 For example, Spuls et al 13 conducted a systematic review of systemic treatments of psoriasis. Treatments analyzed included UV-B, psoralen plus UV-A, methotrexate, cyclosporin A, and retinoids. To find relevant references, the authors used an exhaustive strategy that included searching MEDLINE, contacting pharmaceutical companies, polling leading authorities, reviewing abstract books of symposia and congresses, and reviewing textbooks, reviews, editorials, guideline articles, and the reference lists of all articles identified. Of 665 studies found, 356 (54%) were identified by MEDLINE search (30%-70% for different treatment modalities). 13

EMBASE is Excerpta Medica's database covering drugs, pharmacology, and biomedical specialties. 1 EMBASE has better coverage of European and non-English language sources than MEDLINE and may be more up to date. 1 The overlap in journals covered by MEDLINE and EMBASE is about 34% (10%-75%, depending on the subject). 14 , 15

Publication bias (ie, the tendency of easy-to-locate studies to show more "positive" effects) is an important concern for systematic reviews, and a useful analysis of this issue can be found elsewhere. 16 Publication bias results when issues other than the quality of the study are allowed to influence the decision to publish. Several studies have shown that factors such as sample size, direction and statistical significance of findings, and investigators' perceptions of whether the findings are "interesting" are related to the likelihood of publication. 17 , 18

Language bias may also be a problem: studies with positive findings are more likely to be published in an English-language journal and also more quickly than those with inconclusive or negative findings. 19 A thorough systematic review should therefore include a search for high-quality unpublished trials and not restrict itself to journals written in English.

Studies with small samples are less likely to be published than those with larger samples, especially if they have negative results. 17 , 18 This type of publication bias jeopardizes one of the main goals of meta-analysis: to increase power by pooling the results of numerous small studies. Creation of study registers and advance publication of research designs have been proposed as ways to prevent publication bias. 20 , 21

Publication bias can be detected by using a simple graphic test (funnel plot) or by calculating the "fail-safe N." 22 , 23 These techniques are of limited value when fewer than 10 randomized controlled trials are included.

For many diseases, the studies published are dominated by drug company–sponsored trials of new expensive treatments. This bias in publication can result in data-driven systematic reviews that draw more attention to those medicines. In contrast, question-driven systematic reviews answer the sorts of clinical questions of most concern to practitioners. In many cases, studies that are of most relevance to doctors and patients have not been done in the field of dermatology owing to inadequate sources of independent funding.

Systematic reviews that have been sponsored directly or indirectly by industry are also prone to bias by overinclusion of unpublished studies with positive findings that are kept "on file" by that industry. Until it becomes mandatory to register all clinical trials conducted on human beings in a central registry and to make all of the results available in the public domain, all sorts of distortions due to selective withholding or release of data may occur.

Generally, reviews that have been conducted by volunteers in the Cochrane Collaboration are of better quality than non-Cochrane reviews. 24 However, potentially serious errors have been noted in up to a third of Cochrane reviews. 25

In general, the studies included in systematic reviews are reviewed by at least 2 reviewers. Data such as numbers of people entered into studies, numbers lost to follow-up, effects sizes, and quality criteria are recorded on predesigned data abstraction forms by at least 2 reviewers. Differences among reviewers are usually settled by consensus or by a third-person arbitrator. A systematic review in which there are large areas of disagreement among reviewers should lead the reader to question the validity of the review.

Results in the individual clinical trials that make up a systematic review may be similar in magnitude and direction (eg, they may all indicate that treatment A is superior to treatment B by a similar magnitude). Assuming that publication bias can be excluded, systematic reviews of studies with findings that are similar in magnitude and direction provide results that are most likely to be true and useful. It may be impossible to draw firm conclusions from systematic reviews of studies that have results of widely different magnitude and direction.

The magnitude of the difference between the treatment groups in achieving meaningful outcomes is the most useful summary result of a systematic review. The most easily understood measures of the magnitude of the treatment effect are the difference in response rate and its reciprocal, the number needed to treat (NNT). 1 , 8 , 12 The NNT represents the number of patients one would need to treat to achieve 1 additional cure. Whereas the interpretation of NNT might be straightforward within a single trial, interpretation of NNT requires some caution within a systematic review because this statistic is highly sensitive to baseline event rates. For example, if treatment A is 30% more effective than treatment B for clearing psoriasis, and 50% of people who undergo treatment B are cleared with therapy, then 65% will clear with treatment A. These results correspond to a rate difference of 15% (65 − 50) and an NNT of 7 (1.00/0.15). This difference sounds quite worthwhile clinically. But if the baseline clearance rate for treatment B in another trial or setting is only 30%, the rate difference will be only 9%, and the NNT now becomes 11, and if the baseline clearance rate is 10%, then the NNT for treatment A will be 33, which is perhaps less worthwhile. In other words, it rarely makes sense to provide 1 NNT summary measure within a systematic review because "control" or baseline events rates usually differ considerably between studies owing to differences in study populations, interventions, and trial conditions. 26 Instead, a range of NNTs for a range of plausible control event rates that occur in different clinical settings should be given, along with their 95% confidence intervals.

The precision of the estimate of the differences among treatments should be estimated. The confidence interval provides a useful measure of the precision of the treatment effect. 1 , 8 , 12 , 27 , 28 The calculation and interpretation of confidence intervals has been extensively described. 29 In simple terms, the reported result (known as the point estimate ) provides the best estimate of the treatment effect. The population or "true" response to treatment will most likely lie near the middle of the confidence interval and will rarely be found at or near the ends of the interval. The population or true response to treatment has only a 1 in 20 chance of being outside the 95% confidence interval.

Certain conditions must be met when meta-analysis is performed to synthesize results from different trials. The trials should have conceptual homogeneity. They must involve similar patient populations, have used similar treatments, and have measured results in a similar fashion at a similar point in time. There are 2 main statistical models used to combine results: random effects models and fixed effects models. Random effects models assume that the different studies' results may come from different populations with varying responses to treatment. Fixed effects models assume that each trial represents a random sample of a single population with a single response to treatment. In general, random effects models are more conservative (ie, less likely to show statistically significant results) than fixed effects models. When the combined studies have statistical homogeneity (ie, when the studies are reasonably similar), random effects and fixed effects models give similar results.

The key principle to keep in mind when considering combining results from several studies is that conceptual homogeneity precedes statistical homogeneity. In other words, results of several different studies should not be combined if it does not make sense to combine them (eg, if the patient groups or interventions studied are not sufficiently similar to each other). Although what constitutes "sufficiently similar" is a matter of judgment, the important thing is to explicitly articulate the decision to combine or not combine different studies. Tests for statistical heterogeneity are typically of very low power, so that statistical homogeneity does not mean clinical homogeneity. When there is evidence of heterogeneity, reasons for heterogeneity between studies such as different disease subgroups, intervention dosage, or study quality should be sought.

Sometimes, the robustness of an overall meta-analysis is tested further by means of a sensitivity analysis. In a sensitivity analysis the data are reanalyzed excluding those studies that are suspect because of quality or patient factors, to see whether their exclusion makes a substantial difference in the direction or magnitude of the main original results. In some systematic reviews in which a large number of trials have been performed, it is possible to evaluate whether certain subgroups (eg, children vs adults) are more likely to benefit than others. Subgroup analysis is rarely possible in dermatology because few trials are available.

The conclusions in the discussion section of a systematic review should closely reflect the data that have been presented within that review. The authors should make it clear which of the treatment recommendations are based on the review data and which reflect their own judgments. In addition to making clinical recommendations of therapies when evidence exists, many reviews in dermatology find little evidence to address the questions posed. This lack of conclusive evidence does not mean that the review is a waste of time, especially if the question addressed appears to be an important one. For example, the systematic review of antistreptococcal therapy for guttate psoriasis by Owen et al 30 provided the authors with an opportunity to call for primary research in this area and to make recommendations on study design and outcomes that might help future researchers.

Applying evidence summarized in a systematic review to specific patients requires the same processes used to apply the results of individual controlled clinical trials to patients.

Section Editors

Michael E. Bigby, MD, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Mass Rosamaria Corona, DSc, MD, Istituto Dermopatico dell'Immacolata, Rome, Italy Damiano Abeni, MD, MPH, Paolo Pasquini, MD, MPH, Istituto Dermopatico dell'Immacolata Moyses Szklo, MD, MPH, DrPH, The Johns Hopkins University, Baltimore, Md Hywel Williams, MSc, PhD, FRCP, Queen's Medical Centre, Nottingham, England

Corresponding author and reprints: Michael Bigby, MD, Beth Israel Deaconess Medical Center, 330 Brookline Ave, Boston, MA 02215 (e-mail: [email protected] ).

Accepted for publication February 6, 2003.

This article is being published in Williams H. Evidence-Based Dermatology . London, England: BMJ Books; 2003. Further details on this book can be found at http://www.evidbasedderm.com .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: May 10, 2024 7:59 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

801k Accesses

295 Citations

93 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Critical Appraisal of Research Articles: Systematic Reviews

• Systematic Reviews
• Clinical Practice Guidelines
• Qualitative Studies

What is a Systematic Review?

A systematic review is a review of a clearly formulated queston that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from studies that are included in the review. Statistical methods may or may not be used to analyze and summarize the results of the included studies.

How to Find Systematic Reviews

1. Search the Cochrane Database of Systematic Reviews

2.  Using  PubMed , either use the 'Systematic Reviews' filter or add this to the end of your search 'AND (systematic review [ti])

3. If searching CINAHL , limit by publication type (select "Systematic Review").

Questions to Ask

• Is it a systematic review of the right type of studies which are relevant to your question?
• Does the methods section describe how all the relevant trials were found and assessed?    The paper should give a comprehensive account of the sources consulted in the search for relevant papers, the search strategy used to find them, and the quality and relevance criteria used to decide whether to include them in the review.
• The authors should include hand searching of journals and searching for unpublished literature.
• Were any obvious databases missed?
• Did the authors check the reference lists of articles and textbooks?
• Did they contact experts (to get their list of references checked for completeness and to try to find out about ongoing or unpublished research)?
• Did they use an appropriate search strategy; were important subject terms missed?
• Who were the study participants and how is their disease status defined?
• What intervention(s) were given, how, and in what setting?
• How were outcomes assessed?

      3.   Are the studies consistent, both clinically and statistically?

      4.   Compare with PRISMA

• Look at the most recent PRISMA checklist to see how well the authors documented the various preferred reporting items.

Appraisal Checklists for Systematic Reviews

• Critical Appraisals Skills Programme (CASP)
• Joanna Briggs Institute
• << Previous: Prognosis
• Next: Clinical Practice Guidelines >>

• Last Updated: Mar 1, 2024 11:56 AM
• URL: https://guides.himmelfarb.gwu.edu/CriticalAppraisal

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

About Systematic Reviews

How Do You Critically Appraise a Systematic Review?

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

However, you have to appraise your systematic review to ensure that the evidence collected is incontrovertible and relevant. Fortunately, there are several simple ways of appraising your systematic reviews. But why are systematic reviews good ? And what is quality appraisal in a systematic review ? This article will attempt to answer these and other related questions.

What Are Systematic Reviews?

Systematic reviews are structured and methodical assessments, of the available evidence to draw conclusions, which will guide important policy-making processes. A systematic review won’t be complete without an explicitly stated research question, a materials section, and a methods section. The research question of a systematic review is the specific clinical question that the evidence collected is supposed to answer, while materials are the relevant literature. The methods section comprises techniques used in data extraction, quality appraisal, and study data analysis. Types of data analysis commonly used are thematic, narrative, qualitative, and quantitative analysis. The type of included studies and the method used to synthesize results differ according to the type of review.

It’s important to note that all systematic reviews are not equal. Each review depends on the objectives of the study and the kind of literature to be evaluated. But you need to ensure that your systematic review includes all the pertinent trials, isn’t biased, and the synthesized results are factual and useful. To achieve this, you need to simplify your systematic reviews by incorporating the necessary tools. Fortunately, there are numerous software tools designed to help conduct systematic reviews of literature.

One of the advantages of systematic reviews is that they involve the appraisal of the quality of evidence. To assess the methodological quality of a systematic review, an assessment criterion is used. You can develop your assessment criterion or ‘borrow’ or modify an assessment criterion used by another author.

Learn More About DistillerSR

(Article continues below)

Appraising Systematic Reviews

In some instances, systematic reviews use meta-analyses, which involve the quantitative pooling of similar studies to generate the overall summary of evidence. The evidence from individual studies should be responsive to arithmetic analysis. You need to be certain that the review evidence you’ve collected is useful. Apart from appraising the quality of the systematic review, you can also appraise the quality of the included studies.

• When you are assessing the quality of your systematic reviews, you should ask yourself the following questions:
• Does the systematic review clearly report and perform a complete and replicable literature search?
• Does the systematic review frame a clearly focused question?
• Does the review synthesize primary studies correctly?
• Does the systematic review show how the results should be combined statistically?
• Does the systematic review provide complete numbers and the right summary statistics?
• Does the methods section of the systematic review clearly state the basis for inclusion or exclusion of primary Randomized Control Trials (RCTs)?
• Does the systematic review report information like the size, relevant interventions, and outcomes from the primary RCTs?
• Does the systematic review evaluate the procedural quality of primary studies?
• Does the review discuss the reasons for variations or heterogeneity between individual RCTs and overall results?
• Does the review report on the clinical relevance or benefits of the synthesized results?

3 Reasons to Connect

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills
• Systematic reviews

Describing and appraising studies

• What are systematic reviews?
• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

On this page:

• Describing stuides

Examples of describing studies

Critical appraisal of quality and relevance, appraising the quality of a systematic review, describing studies.

Studies that are used in a review are described in a standardised way that is suitable for each review.  The detail provided facilitates transparency in how each study contributes to the overall findings of the review, and the overall reliabillity of the review.

There are three key reasons for describing (or coding) the studies in a systematic review.

• To know more about the included studies. Studies may be described on characteristics such as the aims, methods, or particular elements to describe the research sample and any outcomes measured.  Historically such descriptions about the studies may have been limited to basic information such as authors names, place of publication, and research methods. It is now recognized that information about what has or has not been studied is a useful product in its own right, as highlighted in the box on systematic evidence maps .
• To identify the research findings of the individual studies to be synthesized
• To identify the methods used in each study, so the study can be critically appraised for trustworthiness and relevance to the review. Methodological aspects of each study might be described in terms of how sampling was undertaken, recruitment of the sample, data collection methods, data analysis methods.
• Coding for systematic reviews and meta-analysis, Sandra Wilson Video (78 mins) on describing studies)

• Young people, pregnancy and social exclusion Brunton et al 2006. An example of the coding used for all the included studies for a review on Young people, pregnancy and social exclusion.

The EPPI-Centre coding guidelines in education:

• Education guidelines version 0.94 (2001) Guideline for descriptive keywording.
• Education guidelines version 0.97 (2003) Guideline for extracting and appraising data from studies.

Critical appraisal involves checking the quality, reliability and relevance of the studies in the review in relation to the review question. It appraises each study in terms of the following aspects:. 

• Is the study relevant to the research question?
• Is the study valid? E.g. Were the study methods applied appropriately?
• Were appropriate methods used in relation to the review question?

In addition, the studies are collectively appraised in terms of how they support the review findings and evidence claims of the review. For example, if the research evidence comprises of studies that have wide variation of findings, this reduces the strength of the evidence claims.

There are many standardised tools available for critical appraisal depending on the study design and the type of review. The approach to critical appraisal and the appraisal decisions for each study should be reported.

• Critical Appraisal Skills Programme (CASP) Tools (checklists) for systematic reviews, randomised controlled trials, cohort studies, case control studies, economic evaluations, diagnostic studies, qualitative studies, clinical prediction rule
• EBP checklists: University of Glasgow Adaptations of the CASP checklists for appraising other types of study, including an education intervention, and studies on treatment and prevention, and guidelines.
• Newcastle-Ottawa Scale Useful for assessing quality of non-randomised studies such as case-control and cohort studies.
• Risk of Bias Cochrane tool Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2). RoB 2 is structured into a fixed set of domains of bias, focussing on different aspects of trial design, conduct, and reporting.

Commonly-used tools for appraising research evidence in reviews:

• GRADE For appraising reviews of effectiveness.
• CerQUAL For appraising reviews including qualitative research evidence.

It is important for users of systematic reviews to consider the quality of the whole review. There are three separate elements that contribute an appraisal:

• the quality and relevance of the methods used to address the review questions;
• the quality and relevance  of the methods used by the individual studies included in the review;
• the nature and extent of the total evidence from studies included in the review.

There are tools to help with the appraisal of a whole review. Some of these are specific to certain types of reviews, and others are more generic. 

• JBI checklist for systematic reviews An appraisal tool that can be used for a range of types of systematic reviews. From the Joanna Briggs Institute.

Some tools focus only on appraising the methods of specific types of reviews:

• AMSTAR 2 Used to appraise only reviews that examine the effectiveness of interventions.
• ROBIS A tool from the University of Bristol that focuses on testing statistical bias in certain types of reviews.

Further reading:

• Evidence Standards: A Dimensions of Difference Framework for Justifiable Evidence Claim Gough (2016). This short paper provides an overarching conceptual framework for considering the main dimensions involved in appraising evidence claims of a review.

Related guides

• LibrarySkills@UCL: Evaluating information
• << Previous: Searching databases
• Next: Synthesis and systematic maps >>
• Last Updated: Apr 4, 2024 10:09 AM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

Systematic Reviews

• Introduction
• Guidelines and procedures
• Management tools
• Define the question
• Check the topic
• Determine inclusion/exclusion criteria
• Develop a protocol
• Identify keywords
• Databases and search strategies
• Grey literature
• Manage and organise
• Screen & Select
• Locate full text
• Extract data

Example reviews

• Examples of systematic reviews
• Accessing help This link opens in a new window
• Systematic Style Reviews Guide This link opens in a new window

Please choose the tab below for your discipline to see relevant examples.

For more information about how to conduct and write reviews, please see the Guidelines section of this guide.

• Health & Medicine
• Social sciences
• Vibration and bubbles: a systematic review of the effects of helicopter retrieval on injured divers. (2018).
• Nicotine effects on exercise performance and physiological responses in nicotine‐naïve individuals: a systematic review. (2018).
• Association of total white cell count with mortality and major adverse events in patients with peripheral arterial disease: A systematic review. (2014).
• Do MOOCs contribute to student equity and social inclusion? A systematic review 2014–18. (2020).
• Interventions in Foster Family Care: A Systematic Review. (2020).
• Determinants of happiness among healthcare professionals between 2009 and 2019: a systematic review. (2020).
• Systematic review of the outcomes and trade-offs of ten types of decarbonization policy instruments. (2021).
• A systematic review on Asian's farmers' adaptation practices towards climate change. (2018).
• Are concentrations of pollutants in sharks, rays and skates (Elasmobranchii) a cause for concern? A systematic review. (2020).
• << Previous: Write
• Next: Publish >>
• Last Updated: May 13, 2024 11:34 AM
• URL: https://libguides.jcu.edu.au/systematic-review

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard
• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: May 8, 2024 1:44 PM
• URL: https://lib.guides.umd.edu/SR

Limitations of the international approach to anti-corruption: a systematic review of South Africa’s compelling case of failing anti-corruption

• Open access
• Published: 09 May 2024

Cite this article

You have full access to this open access article

• Thomas Duke Labik Amanquandor   ORCID: orcid.org/0000-0002-4846-2293 1  

99 Accesses

Explore all metrics

In the Global South, anti-corruption initiatives continue to fail despite varying commitments to the international anti-corruption agenda. Concurrently, as this study demonstrates, researchers investigating this paradox appear confined within orthodox explanations for failing anti-corruption efforts. Through a systematic review of 58 studies, this paper demonstrates that South Africa’s anti-corruption corpora from 1995 to 2022 fall to this critique. By employing socio-legal theoretical perspectives, the paper elucidates how and why the orthodoxy dominates the corpora and subsequently suggests a more nuanced understanding of the country’s ongoing failure to combat corruption. For example, the paper argues that the intricacies of South Africa’s corruption challenge the perspective that anti-corruption measures fail simply due to widespread rule-breaking. Through the prism of legal pluralism, this paper demonstrates that adherence to rules is indeed prevalent in South Africa, albeit often not aligned with formal state anti-corruption legislation and regulations. Finally, the paper posits innovative approaches to enhance and broaden our comprehension of why anti-corruption efforts fail, especially in the Global South.

Similar content being viewed by others

Introduction: towards a better understanding of corruption and anti-corruption

International Anti-Corruption Initiatives: a Classification of Policy Interventions

What next for anti-corruption research in vietnam.

Avoid common mistakes on your manuscript.

Introduction

The United Nations, the World Bank, the International Monetary Fund (IMF), and non-governmental entities like Transparency International converge upon a set of shared tenets and best practices regarding how to eradicate corruption globally. This commonality in perspective underpins what might be termed an international anti-corruption consensus (Gephart, 2009 , p. 8; Kuldova et al., 2024 ). Sampson ( 2010 , p. 262) articulates this notion further, observing that this consensus projects itself globally as an extensive array of policies, regulatory measures, conventions, training modules, and programmes dedicated to enhancing integrity and ameliorating public governance.

Regrettably, corruption persists as a potent scourge worldwide, casting a shadow of doubt over the efficacy of the international anti-corruption consensus (Gutterman & Lohaus, 2018 ). Numerous scholars and policy experts concur that the strategies and technologies of international anti-corruption consensus disproportionately depend on a principal-agent theoretical framework that misrepresents the complexity of corruption (Persson et al., 2010 , 2013 ) and neglects the nuanced local realities that are crucial for the successful implementation of anti-corruption measures (Jackson, 2020 ; Khan et al., 2019 ). In this paper, the aforementioned theoretical postulations are termed “the orthodoxy.”

Through a critical interpretive synthesis (CIS) of studies published between 1995 and 2022 on South Africa’s anti-corruption efforts, this paper elucidates how anti-corruption researchers appear confined within the orthodoxy in their attempt to unravel why the country’s anti-corruption efforts have yielded minimal results. Subsequently, this paper draws on socio-legal theories and critical corruption scholarships to present a nuanced understanding of the country’s failing anti-corruption and posit innovative approaches that can enhance our comprehension of corruption and anti-corruption in general.

This paper is structured as follows. The subsequent section provides a background on (anti) corruption in South Africa. Section three delves into the theoretical underpinnings that guide this study. Section four articulates how the Critical Interpretive Synthesis (CIS) method was employed in the study, and the subsequent section— section five —presents the study’s findings on why South Africa’s anti-corruption regime is failing. These findings are assembled under three interrelated thematic corpora from an interpretive synthesis of the eligible literature. Section six incorporates key arguments from the thematic corpora to illuminate their overall contribution to understanding the country’s failing anti-corruption, thereby suggesting how the corpora appear confined within the orthodoxy. Subsequently, in this section, the paper attempts a more nuanced understanding of the country’s ongoing failure to combat corruption. Lastly, the final section ( section seven ) proffers a summary of primary findings and their implications, supplemented by research recommendations for future studies.

Anti-corruption in South Africa: a compelling case

South Africa’s violent history, apartheid, and white supremacy are crucial to understanding the country’s anti-corruption challenges. The apartheid regime was inherently corrupt. According to Lodge ( 1998 ), the National Party, especially in the 1950s and 1960s, promoted transactive corruption by favoring white civil servants to foster the socio-economic fortunes of the white minority. Although this system was driven by the goals of Afrikaner Nationalism rather than individual enrichment (Seegers, 1993 ), it paved the way for a corrupt regime by the 1980s, especially at the homeland government level (Bauer, 2000 ; Lodge, 1998 ). The impending end of white rule led to a rush to exploit and plunder state resources (Hyslop, 2005 ; Van Vuuren, 2006 ). As one official described, many felt they were missing out if they did not engage in corruption (Bauer, 1999 , p. 78). Consequently, lack of accountability, kickbacks, favoritism, cronyism, and bribery became common in various state departments (Bauer, 1999 ; Hyslop, 2005 ; Lodge, 1998 ).

Despite democratic reforms and increased scrutiny, corruption remained pervasive across various sectors due to pre-existing structural weaknesses and ingrained habits (Lodge, 2001 ). Integrating former homeland administrations into new local governments and the infusion of political solidarity among the new ruling elite and its supporters perpetuated patrimonial habits, noticeably intensifying corruption in provincial and local governments (Lodge, 2001 ). Moreover, various new stimulants for corrupt behaviour, such as the shortages of skilled staffing in financial control systems and the expansion of citizen entitlements to public resources, arose (Camerer, 2009 ; Lodge, 1998 ; Van Vuuren, 2006 ). The systemic and structural weaknesses of the new state, coupled with the emergence of a new economic elite, culminated in creating an environment in which corruption flourished (Hyslop, 2005 ; Lodge, 1998 ).

The extent to which the country’s violent and racialized histories shape its current anti-corruption legislative framework is unclear. Today, South Africa is a devoted signatory to several anti-corruption conventions, such as the United Nations Convention Against Corruption (UNCAC) and the African Union Convention, to mention a few (Langendorf, 2015 , p. 57). It is essential to highlight, however, that before the ratification of the UNCAC, South Africa did not have legislation explicitly targeting corruption. Hence, corruption was fought through a general National Crime Prevention Strategy.

Meanwhile, the country has implemented several anti-corruption interventions over the last two decades. Nevertheless, these interventions have yielded minimal results (Budhram & Geldenhuys, 2018 ; Gray, 2021 ). The recent misappropriation of the Covid-19 relief fund, evidence of tender irregularities and scandals at the Ministry of Health, and the killing of a whistleblower in the Gauteng Health Department suggest that corruption is still endemic in the country (Patel & Govindasamy, 2021 ).

South Africa is distinct in many ways but shares some significant commonalities with other sub-Saharan countries. The country’s multi-racial demography, settler colonial historical legacy, relatively strong constitutionalism, multi-party liberal democracy, and economic development over the last two decades set it free from the trappings of the stereotypical African context where bad governance and the lack of the rule of law are used to explain away failing anti-corruption efforts. Aside from its strong constitutionalism and the rule of law, South Africa had the best anti-corruption legislation on the African continent, having significantly implemented the UNCAC provisions and receiving a “Very Good” rating in the integrity index (Integrity, 2008 ). Hence, South Africa’s failing anti-corruption regime represents a compelling analytical case for understanding the intricacies of the international anti-corruption consensus.

Theoretical perspectives on [anti]-corruption

Combatting corruption comprises three interdependent efforts: understanding corruption, designing counteractive strategies, and establishing anti-corruption institutions to enforce and implement the strategies. Developing a far-reaching theoretical characterization of corruption has perhaps been the most challenging among these interdependent efforts.

The orthodoxy and its critique

Rose-Ackerman’s ( 1997 , 2008 ) conceptualization of corruption as the “misuse of public office for private gain” and variants of it (see, for example, Nye, 1967 ; Transparency International, 2022 ; World Bank, 1997 ) has resounded well with academics, policymakers, and shaped international anti-corruption practice for decades (Mungiu-Pippidi, 2013 ). In this view, corruption occurs when the authority of a public official is exercised in a manner that violates the public trust and contravenes its anticipated purpose of pursuing the public’s collective interest (Ganahl, 2014 ; Rose-Ackerman, 2008 ). According to Tanzi ( 1998 ), the public-private divide stresses corruption among public officials and shadows corrupt practices in the private sector or private-private corruption. Nevertheless, the Transparency International variant of the definition seems to offset this critique. By conceptualizing corruption as the abuse of ‘entrusted power’ for private gain, the dichotomy remains. However, the description encompasses public and private officials (Transparency International, 2022 ).

Critical corruption scholars argue that this conceptualisation frames corruption within a rational-legal bureaucratic system marked by structured hierarchy and distinct roles within the public sector aimed at functioning on codified, logical, and societal rules. In contrast, the private sphere is shaped by familial obligations, emotions, beliefs, and customs. Moreover, it views corruption as dysfunctional, ethically reprehensible, transactional, and definitively harmful (Gutterman & Lohaus, 2018 ). However, this perspective is inherently Western and may not align with non-Western societies, where the boundary between public and private realms is more fluid, with private aspects like family obligations and spirituality influencing the public sphere (Zaloznaya, 2013 ). Thus, cloaked as universal, this perspective is reductive and imposes Western notions of corruption on variable local experiences and cultural understandings (Zaloznaya, 2013 ).

Furthermore, this conceptualisation primarily construes corruption as motivated by an individual’s cost-and-benefit calculus or, as Zaloznaya ( 2013 , p. 711) rehearses March and Olsen ( 1994 ), a form of rationality that reflects the logics of consequence whereby people break legal and ethical codes for the sake of material benefits and power. Thus, individuals’ self-seeking instrumental calculus amidst entrusted power creates corruption opportunities. Also known as the principal-agent theory, this perspective views corruption as occurring in a dualistic relation where the official (agent with entrusted power) has more information and discretional power in a specific situation and uses this advantage for their gain, even when it goes against the interests of the public (principal) (Rose-Ackerman, 2008 ; Tanzi, 1998 ). Within this perspective, corruption persists amidst principals’ poor supervision, monitoring, and sanctioning of officials (Jackson, 2020 ).

Anti-corruption reforms, especially those promoted by the international anti-corruption consensus through international best practices and conventions, as well as through political pressure from Western governments and aid conditionalities from international organizations such as the World Bank and IMF, are along the lines of this theoretical perspective and, therefore, often promote counteractive measures that seek to primarily enhance monitoring, supervision, transparency, accountability, and the compliance of public agents with bureaucratic norms, procedures, and formal rules (Khan et al., 2019 ; Kuldova et al., 2024 ; Sampson, 2005 ; Zaloznaya, 2013 , p. 707).

Meanwhile, several scholars insist that the principal-agent perspective mischaracterises the nature and mechanism of corruption, especially in systemic corrupt contexts. Rothstein ( 2021 ) argues that “principals,” as assumed in the principal-agent perspective, barely exist in these contexts because the top politicians and bureaucratic leaders accrue the most from corruption and thus are less incentivised to combat it. In the case of developing countries, Khan et al. ( 2019 , p. 8) add that the perspective erroneously assumes that people are generally rule-abiding; hence, corruption results from occasional violations that can be addressed with improvements in good governance, transparency, and accountability. However, in reality, rule violation is generally more prevalent. As a result of this theoretical mischaracterisation, the implementation of the UNCAC and other international best practices has yielded minimal results, especially in systemic corrupt contexts (Jackson, 2020 ; Khan et al., 2019 ; Persson et al., 2010 ).

Critics of the principal-agent perspective argue that corruption, when systemic, is mainly a collective action problem. The collective-action theory proposes that corrupt practices persist in the contexts of their occurrence because they are generally considered the norm. As a result, people either lose or sometimes gain little from behaving otherwise, especially if it is impossible to trust that others in the same context will follow suit (Bauhr, 2017 ; Kaufmann et al., 2015 ; Marquette & Peiffer, 2015a ; Mungiu-Pippidi, 2013 ). In this context, corruption is seen as a manifestation of “free-riding” behaviour (Olson, 1971 ). This interpretation of corruption considers good governance, rule of law, transparency, and accountability as public goods that are non-excludable (meaning it is impossible to prevent people from benefiting) and non-rivalrous (one person’s use does not diminish availability for others) (Marquette & Peiffer, 2015b , p. 3; Rothstein, 2011 ). As a result, even individuals who do not actively contribute to producing good governance, accountability, and transparency can still reap the benefits, becoming what is known as “free riders.” Hence, those who engage in corruption do so out of self-interest, knowing their participation will not lead to losing their beneficial status (Marquette & Peiffer, 2015b , p. 6). Furthermore, free riders also perceive their contribution to the production of good governance as insignificant, with the expectation that others will also free-ride on their efforts.

The collective-action perspective emphasizes that corruption is a collective rather than an individual issue (Marquette & Peiffer, 2015a ). It implies that effective anti-corruption initiatives must build mutual trust in producing good governance, transparency, and accountability. Nevertheless, both principal-agent and collective action theories postulate that individuals are motivated by self-interest; therefore, increasing surveillance and implementing punitive measures can enhance accountability and curb corruption (Marquette & Peiffer, 2015b , p. 6).

Socio-legal perspectives

According to Krygier ( 1990 ), law is a tool the state uses to translate its wishes into action and maintain social order. However, several socio-legal scholarships have critiqued this instrumental view, which construes law as external to society. These scholarships demonstrate that law is constitutive or integral to society (Halliday & Morgan, 2013 ; Sarat & Kearns, 2009 ; Silbey, 2005 ), thus emphasizing the importance of individuals’ understanding and interpretation of the law (Halliday & Morgan, 2013 ) which shape their everyday behavior (Hertogh, 2004 ).

Informed by this constitutive perspective, recent socio-legal studies on anti-corruption have presented more nuanced explanations for the limited success of anti-corruption efforts, particularly in non-Western countries. For example, these studies highlight the importance of legal pluralism in understanding why anti-corruption efforts fail (Bierschenk, 2008 ; Urinboyev & Svensson, 2018 ). Legal pluralism refers to the coexistence and clash of multiple sets of rules or ‘legal orders’ that interrelate with and influence people’s social behaviour. These legal orders encompass various forms of law, including national/state law, customary rules, religious decrees, moral codes, and practical norms (Griffiths, 2003 ; Merry, 1988 ). The concept of legal pluralism recognizes that state law is just one of many legal orders within society. Especially in non-Western societies, legal anthropology studies have demonstrated the existence of non-state normative systems or “semi-autonomous fields,” as Sally Moore calls them, that possess defined boundaries and internal mechanisms, including extra-legal activities that challenge externally directed behavioural changes that contradict the field’s normative order (Moore, 1972 , p. 720). Members of semi-autonomous fields feel a moral obligation to conform to the internal rules and moral codes because their membership is contingent upon their conformity to internal norms rather than external expectations (Overman et al., 2014 ). Consequently, this poses crucial challenges to legal and administrative reforms that contravene the internal norms of semi-autonomous fields.

The legal pluralism perspective ties in with legal consciousness studies investigating the taken-for-granted worldviews and assumptions about law and legality that shape people’s everyday behavior (Halliday & Morgan, 2013 , p. 2). Legality in the legal consciousness approach signifies the ‘meaning, sources of authority, and cultural practices commonly recognised as legal, regardless of who employs them or for what ends’ (Ewick & Silbey, 1998 , p. 22). Holen ( 2023 , p. 22) succinctly defines this notion of legality as an assemblage of normativity, i.e., the state laws, morals, social norms, religious norms, commands, customs, expectations, and etiquette. Legal consciousness scholarships suggest that at the individual level, on the one hand, this intricate assemblage of normativity shapes people’s understanding of legality—what is right or wrong, corrupt or not—in varying situations. On the other hand, at the structural level, this intricate assemblage of normativity could account for why a discrepancy comes to exist between what the law purports to offer and what it achieves in reality— typically called “the gap problem” in socio-legal research (Halliday & Morgan, 2013 , p. 3; Silbey, 2005 , p. 323).

The legal pluralism and legal consciousness approaches can deepen our understanding of why corruption persists even amid international-standard anti-corruption legislative and institutional frameworks, especially in third-world countries. These socio-legal perspectives provide a critical theoretical toolkit for understanding this discrepancy beyond the orthodoxy . By empirically focusing on everyday social logic, local cultural categories, and norms concerning the law, these perspectives can disentangle the intricacies of legality that shape individual understanding and attitude regarding corruption on the one hand and, on the other hand, how these complexities come to produce the structural ineffectiveness of anti-corruption initiatives.

Systematic review method

According to Tranfield et al. ( 2003 ), systematic review methods differ from traditional narrative reviews because they rely on a more transparent and rigorous process of gathering and selecting literature. This study employed the Critical Interpretive Synthesis (CIS) systematic review method. The CIS approach is a novel review method that combines conventional systematic methodology with qualitative analysis techniques from grounded theory and meta-ethnography (Depraetere et al., 2021 ; Flemming, 2010 ). This method comprises the following steps: searching the literature, eligibility assessment, data extraction, and interpretive synthesis (Dixon-Woods et al., 2006 ). In searching for literature, this study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodological guidelines (Moher et al., 2009 ). Thus, a search was conducted on the Web of Science (WoS), Scopus, and Google Scholar databases for studies published between January 1995 and June 2022 via the following search term in the title or abstract:

[corruption OR anti-corruption OR anti-corruption]

[South Africa]

The search resulted in 529 records. These comprised 89 papers from Web of Science, 58 from Scorpus, and 382 from Google Scholar. Afterward, duplicates were identified and removed, bringing the total down to 474 records. Then, the titles and abstracts of the 474 records were screened, after which all potentially eligible studies were extracted via the Zotero reference management software for further eligibility assessment. The potential eligible studies comprised only peer-reviewed journal articles, books, or chapters of books published in English within the social sciences from 1995 to 2022. However, to be eligible for the interpretive synthesis, a potential study should have been concerned with, but not limited to, South Africa’s anti-corruption institutions, agencies, strategies, laws, regulations, and policies. Eventually, 58 of the 474 records met the eligibility and quality criteria and were selected for the critical interpretive synthesis (Fig. 1 ).

PRISMA flow diagram

Subsequently, the lines-of-argument synthesis strategy (LOA) (Noblit et al., 1988 ) was used to identify and synthesise evidence, key findings, and critical arguments found in the 58 eligible literature into three interrelated scholarly corpora that comprehensively account for South Africa’s failing anti-corruption. It is possible that some relevant studies may have used words synonymous with corruption or anti-corruption and were thus uncaptured by this systematic method’s search strategy. Also, studies that use specific names of places or organisations in South Africa without mentioning “South Africa” in the abstract or title may have been omitted. I conducted a “retrospective reference list checking” of the 58 eligible studies to address these limitations (Gough & Richardson, 2018 , p. 125). However, no additional eligible studies were found. Thus, the 58 eligible studies reflect a substantial proportion of research on South Africa’s anti-corruption regime.

This study found an overall increase in peer-reviewed anti-corruption research on South Africa, particularly in the last decade, a period where the country’s corruption woe is reported to have peaked (Global Initiative Against Transnational Organized Crime, 2022 ) and public discourse was marked by a growing discernment of corruption’s pervasiveness in the country (James, 2023 ). However, most of these studies employed secondary research methods in investigating anti-corruption at the macro or national level. The lack of primary empirical investigation, chiefly ethnographic inquiries of anti-corruption at South Africa’s sub-national or local level, is alarming because understanding local contextualities is crucial to effective anti-corruption design and implementation. Notwithstanding, the synthesis of findings and critical arguments within the 58 eligible anti-corruption studies produced three mutually supporting corpora that comprehensively account for South Africa’s failing anti-corruption. Broadly, the first corpus demonstrates the country’s lack of ethical and dedicated political and bureaucratic leaders. The second corpus appraises South Africa’s anti-corruption regime, i.e., legal and institutional framework. It examines its susceptibility to politicisation and political pressure. Lastly, the third corpus shifts attention to the actor’s anti-corruption perspective. It investigates the intricacies of anti-corruption compliance in South Africa (Fig. 2 ).

Number of eligible studies per year

Corpus one: lack of ethical and dedicated leaders

Data Box 1: Review-Data in Corpus One

In 2001, Tom Lodge asserted that “detracting from the government’s proclaimed commitment to fighting corruption are attacks by senior politicians on the work of anti-corruption agencies” (Lodge, 2001 , p. 62). Lodge’s assertion still holds. Bruce ( 2014 ) observed that political actors attain and maintain control over crucial anti-corruption agencies by controlling their politically appointed top officials.

In Naidoo’s ( 2013 , p. 531) view, “no aspect of South Africa’s anti-corruption efforts have coveted as much political controversy as the country’s specialised institutional responses.” The controversial disbandment of the Directorate of Special Operations (DSO) and verdicts from Hugh Glenister v President of the Republic of South Africa & Others are watershed events that epitomise the country’s challenges with political interference and the politicisation of its anti-corruption regime (Mphendu & Holtzhausen, 2016 ; Pillay, 2017 ).

Shortly after its establishment in 2001, the DSO launched a fierce campaign against corruption and organised crimes. Eventually, they gained a reputation for investigating and prosecuting high-profile cases occasionally involving top-ranking members of the ruling African National Congress (ANC) (Berning & Montesh, 2012 ; Naidoo, 2013 ; Pillay, 2017 , p. 6).

Parenthetically, in 2007, the ANC, during its National Congress, resolved that the DSO should be disbanded (Kinnes & Newham, 2012 ). As a result, the Parliament passed the South African Police Service Amendment Bill to dissolve the DSO and replace it with the Directorate for Priority Crime Investigation (DPCI) (Berning & Montesh, 2012 ). Whereas the disbanded DSO was under the National Prosecution Authority (NPA), its successor, i.e., the DPCI, was placed under the South African Police Service (SAPS). Ultimately, the DSO’s closure severely compromised the state’s ability to investigate and counter corruption (April & Sebola, 2016 ; Berning & Montesh, 2012 ). For example, April and Sebola ( 2016 ) demonstrate that under the DPCI, corruption-related arrests and convictions rates declined by 60 per cent and 83 per cent, respectively.

Consequently, the public accused the ANC government of dissolving the DSO to undermine investigations to protect corrupt ANC party officials (Berning & Montesh, 2012 ; Bruce, 2014 ; Keightley, 2011 ; Kinnes & Newham, 2012 ; Mphendu & Holtzhausen, 2016 ; Tushnet, 2019 ). Eventually, the Constitutional Court ruled that the legislation establishing the DPCI was unconstitutional because it failed to secure an adequate degree of independence for the DPCI (Berning & Montesh, 2012 ; Kinnes & Newham, 2012 ; Olutola, 2014 ; Tushnet, 2019 ). Besides, the DCPI has been plagued with unstable leadership, factionalism, and illegal promotion and appointments (Pillay, 2017 ).

However, the DSO’s demise was also due to several mistakes during their investigation and prosecution of high-profile cases. For example, they violated the principles of attorney-client privileges and exceeded their jurisdiction by collecting intelligence without a legal basis (Berning & Montesh, 2012 ). Moreover, Quarcoo ( 2009 ) insists that the DSO’s demise became imminent early in its establishment when they made the prosecution of high-profile individuals their primary focus. He claims that the investigation and prosecution of top politicians by anti-corruption agencies like the DSO may suggest that the law reserves no haven for the political elite. However, due to the political ramifications of such endeavours, the agency’s prosecutorial powers invariably are vulnerable to partisan application, or the accusation thereof, both impugning the agency’s legitimacy” (Quarcoo, 2009 , p. 33). Quarcoo argues that a country that overemphasises the prosecutorial instead of preventive functions of its anti-corruption agencies risks jeopardising its sustainability, which seems to have been the fate of the DSO.

South Africa’s struggles with political interference and politicisation of anti-corruption are complex. There are no indications, at least in this corpus, that this problem can be circumvented through institutional design due to the rather unfortunate efficacy of collective political actions. One may, therefore, assume that South Africa’s anti-corruption enforcement will improve if its top political leaders become dedicated, willing, and ethical (Lodge, 2001 ; Mphendu & Holtzhausen, 2016 ; Okafor, 2009 ).

However, the corpus cautions that improved political will and dedication must be accompanied by innovative bureaucratic leadership required to design, adjust, and enforce measures (e.g., ethics training) to improve the moral climate of their institutions and reduce corruption (Lekubu & Sibanda, 2021 ; Mantzaris, 2016 ; Naidoo, 2012 ; Parboteeah et al., 2014 ). In the country’s local governance, for example, April and Sebola ( 2016 ) observed that appointing senior officials based on only political connectivity and employment equity is rampant. This absence of meritocracy has ushered many departments under incompetent leaders who do not effectively apply the country’s anti-corruption laws and policies, thereby creating institutional environments that entrench unethical and corrupt practices (Majila et al., 2014 ; Manyaka & Nkuna, 2014 ).

In any case, leaders must exemplify their institution’s code of conduct and other professional ethics in their daily behaviour to improve the ethical climate of their institutions (Parboteeah et al., 2014 ). Sadly, many public sector departments in South Africa are led by unethical leaders who instead exploit loopholes in their systems for personal gain (Naidoo, 2012 ; Odeku, 2019 , p. 11). Bribery and corruption prevail even within the top leadership of the country’s anti-corruption enforcement agencies (Olutola, 2014 ). In the last decade alone, two successive National Commissioners of the SAPS have been implicated, and one was convicted for corrupt practices (Keightley, 2011 ; Kinnes & Newham, 2012 ). In addition, there is evidence of some clerks and judges engaging in bribery (Roelofse et al., 2014 ; Sundström, 2015 ) and corrupt senior managers converting anti-corruption incentives into reward schemes for colluding subordinates (Sundström, 2019 ). Meanwhile, leaders who resist political interference are unceremoniously removed from office (Pillay, 2017 ).

Corpus two: political susceptibility of anti-corruption regime

Data Box 2: Review-Data in Corpus Two

South Africa’s anti-corruption legal framework is considered sound because it conforms to international anti-corruption standards (Kurakin & Sukharenko, 2018 ; Langendorf, 2015 , pp. 57–58; Majila et al., 2014 ; Safara & Odeku, 2021 , pp. 209–214). Aside from the existing legal framework, there are recent calls for a “right to freedom from corruption law” (Maseko, 2021 , p. 127; Mubangizi, 2020 , p. 245; Mubangizi & Sewpersadh, 2017 , p. 67). Advocates of this law maintain that it will empower ordinary people to demand transparency and accountability and claim constitutional damages from corrupt public officials. Simultaneously, the legal framework is plagued with voluminous laws and technical formalities, resulting in ambiguities and conflicting provisions (Kanjere & Koto, 2021 ; Sewpersadh & Mubangizi, 2017 , p. 14). Hence, it is unclear how the passing of more laws will improve the country’s fight against corruption.

Besides, for Langendorf ( 2015 ), passing anti-corruption laws in South Africa has primarily been mere adherence to the prescripts of international law. Makiva ( 2021 ) shows evidence of Langendorf’s claim in her mixed-method comparative study of public procurement corruption in Kenya and South Africa. She found that although the design of institutional architecture adheres to international law and best practices, leaders often deliberately put weak internal controls in place to lubricate corrupt activities. As a result, she concludes that the adherence of these countries’ anti-corruption design to international law and best practices is perhaps primarily to attain legitimacy from the international community (Makiva, 2021 ).

Analyses of the DSO’s disbandment and the Glenister case suggest that in practice, South Africa’s anti-corruption laws are often either not applied or selectively applied, and its anti-corruption institutions are not sufficiently independent (Langendorf, 2015 , pp. 84–87; Sewpersadh & Mubangizi, 2017 , p. 14). Nonetheless, it is the primary responsibility of anti-corruption institutions and agencies to enforce and implement the country’s anti-corruption measures. Therefore, if the legal framework is presumably sound amid widespread corruption, one may be inclined to scrutinise the country’s anti-corruption institutional framework.

South Africa’s anti-corruption institutional model relies on rich and varied laws, rules, and regulations, creating about 19 institutions, agencies, and coordinating mechanisms (Pillay, 2017 ). According to Dassah ( 2014 ), the country’s model suffers from every possible weakness of the multi-agency model, such as red-tapism, patronage networking, and rivalry among agencies. In addition, the mandates of the anti-corruption agencies are unclear and overlap, leading some observers to argue that the model lacks proper oversight, coordination, a unified strategic approach, and a leading central agency (Bruce, 2014 ; Chetty & Pillay, 2017 ; Faull, 2011 ; Pillay, 2017 ). These scholars concur that South Africa should establish an independent and centralised single anti-corruption agency (Majila et al., 2014 , p. 236; Montesh & Berning, 2012 , p. 135). However, Mphendu and Holtzhausen ( 2016 ) maintain that because anti-corruption requires a multi-faceted approach that reinforces all pillars of South Africa’s integrity system, no single institution can eliminate the country’s corruption single-handedly. Instead, improving the coordination between agencies can enhance speedier prosecution and the effectiveness of the anti-corruption architecture (Naidoo, 2013 , p. 533).

Subsequently, instead of a single centralised agency, Budhram ( 2015 , p. 53), for example, calls for the formation of a Corruption Intelligence Centre (CIC) that will gather and share crucial information and intelligence needed for other anti-corruption agencies to function effectively. Accordingly, South Africa’s National Development Planning Commission (NDP) refutes claims that fragmentation of anti-corruption efforts is a crucial problem for the country’s fight against corruption (Bruce, 2014 ; Pillay, 2017 ). To this end, the commission emphasises the importance of the existing checks and balances in the current multi-agency model. It claims that the country does not have the institutional foundation suitable and adequate for establishing a single-agency model (Bruce, 2014 ).

Nevertheless, it is discernible within the literature that the most crucial challenge of the country’s multi-agency framework is its susceptibility to political interference and pressure (Bruce, 2014 ; Dassah, 2014 ; Imiera, 2020 ; Keightley, 2011 ; Lodge, 2001 ; Majila et al., 2014 ; Mphendu & Holtzhausen, 2016 ; Naidoo, 2013 ; Pillay, 2017 ; Roelofse et al., 2014 ). Ostensibly, it is usual for corrupt employees to gain impunity through protection from government officials (Roelofse et al., 2014 ). For example, Majila et al. ( 2014 ) employed a self-administered structured questionnaire to several officials from provincial departments in the Eastern and Northern Cape to examine whether anti-corruption agencies are apolitical and capable of exercising their duties effectively. They found that, in most cases, political power was used to protect the corrupt activities of family members, friends, and political supporters. Similarly, Nzo’s ( 2016 , p. 114) ethnographic study of the political complexities involved in council decision-making in the Northern Cape revealed that political pressure to show loyalty to the ANC political party influenced ANC councillors and other political officeholders to ignore legal recourse against certain persons implicated in financial misconduct. Due to political interference, anti-corruption agencies have become increasingly unmotivated to devote themselves wholly to their duties because the effective and appropriate enforcement of laws and regulations often depends on the will and determination of political actors (Majila et al., 2014 ).

However, Naidoo ( 2013 ) argues that the problem of political interference and pressure has little to do with the current multi-agency approach and more to do with the principal-agent lines of accountability upon which the country’s anti-corruption strategies and institutional framework are designed. After an intra-institutional and inter-institutional analysis of South Africa’s anti-corruption enforcement, Naidoo ( 2013 , p. 523) observes that because the integrity of the country’s approach is shaped by principal-agent accountability, institutional or political actors can compromise or sideline anti-corruption mechanisms through collective action efforts. Therefore, even though the call for a single-agency model is profound in the eligible literature and chiefly inspired by the successes of Hong Kong’s Independent Commission Against Corruption (ICAC), proponents have struggled to demonstrate how this framework will be more resilient to collective action efforts and political pressure (Bruce, 2014 ; Mphendu & Holtzhausen, 2016 ; Pillay, 2017 ).

Corpus three: contextual realities of anti-corruption compliance

Data Box 3: Review-Data in Corpus Three

Generally, understanding contextual realities requires primary empirical, often ethnographic, research methods. As shown in the descriptive analysis of the eligible literature, there is a lack of ethnographic enquiries into South Africa’s anti-corruption efforts. This dearth explains why corpus three comprises relatively few studies compared to one and two, which focus on more structural and macro-level intricacies.

Nevertheless, this corpus highlights the complexities of the country’s corruption problem that are crucial to understanding compliance and non-compliance with anti-corruption rules and regulations. In South Africa, corruption can be embedded in loyalties and solidarities forged around a contemporary political morality that rationalises the act as a form of redress and a means to reverse historic racial inequities (Bruce, 2014 ; Gray, 2021 , p. 377; Mubangizi, 2020 ). According to Bruce ( 2014 , p. 57), these contemporary solidarities and loyalties are “manifestations of political and other solidarities, partly animated by ideas of justice and associated with opposition to apartheid and the apartheid period more generally.” Gray ( 2021 , p. 377) describes this as a sense of entitlement, a feeling of the state’s indebtedness to one who struggled during the apartheid and was deprived because of it.

Even though this political morality may emanate from deeply held beliefs and worldviews concerning the country’s socio-economic realities, such rationalisation could also serve as a neutralisation technique (Bruce, 2014 ). Due to this prevailing political morality, the prescripts of the Constitution have become just one of several moral points of reference and, hence, not unambiguously adhered to by many (Bruce, 2014 , p. 54; Munzhedzi, 2016 ; Bähre, 2005 ). Consequently, the implementation of government policies such as the Black Economic Empowerment (BEE), ideally aimed at empowering the previously disadvantaged, has instead culminated into fertile avenues with opportunities for individuals and groups with political connections to pursue self-interest (Bruce, 2014 , p. 57; Munzhedzi, 2016 , p. 1). Furthermore, Bähre’s ( 2005 ) anthropological study indicates that this prevailing political morality perpetuates the pervasive culture of silence and impunity surrounding corrupt practices.

Meanwhile, other scholarships underplay the socio-political dimension of this morality to argue that corruption’s pervasiveness is instead indicative of a general moral crisis in South Africa (Mantzaris, 2016 ; Odeku, 2019 ; Pitsoe, 2013 , p. 751; van Niekerk, 2003 , p. 137). To illustrate this, Mantzaris ( 2016 ) explains that in South Africa, individual desires for personal gratification have heightened due to the portrayal of lavish lifestyles as a valued norm through commercialism and advertisement. Owing to this status quo, individuals with inadequate training in moral ethics and values cannot restrain themselves when encountering corruption opportunities (Mantzaris, 2016 ; Odeku, 2019 ; Pitsoe, 2013 , p. 751; van Niekerk, 2003 , p. 137). However, Nomtha Gray’s ( 2021 ) case study demonstrates that construing compliance/non-compliance as simply a product of individual ethics and morals is perhaps reductionist.

Gray found that “authority and seniority also conferred rectitude,” leading low-level officials to treat irregular instruction as ethics and principles of their ‘principals’ (Gray, 2021 , pp. 378, 380). His interviews demonstrate that informal sanctioning and micro-power relations in South Africa’s public procurement departments lead low-level officials to believe that acting contrary to irregular instructions from their superiors amounts to insubordination (Gray, 2021 , p. 378). Consequently, officials execute irregular instructions even when they do not understand them. This practice led him to conclude that South Africa’s public procurement practices are reminiscent of apartheid-era levels of compliance that prioritise ‘following orders’ above formal procurement policies (Gray, 2021 , p. 369).

Gray’s observation corroborates Brogden and Nijhar’s ( 1998 , p. 104) arguments that corrupt practices in South Africa are sustained through interstices of power masked within permissive formal procedures. To a large extent, Gray ( 2021 ) demonstrates the implication of informal norms and micro-power relations on compliance or non-compliance with formal rules and bureaucratic protocols. Alexander et al. ( 2022 ) further unravel these intricacies by showing that compliance or non-compliance is also shaped by officials’ perceptions of the applicability, suitability, and relatability of formal rules and bureaucratic protocols to the contextual realities of their daily work.

In a mixed-method study of anti-corruption challenges and opportunities in urban planning, Alexander et al. ( 2022 ) found that most public planners believe that their code of conduct is vague and complex and does not correspond to the daily socio-political realities of their work. These socio-political realities include financial and human resource constraints, overlapping administrative and political systems, and competing socio-economic, environmental, and spatial needs. According to Alexander et al. ( 2022 , p. 8), this discrepancy has resulted in the planners’ dependence on individual negotiation and deal-making outside the formal planning process. However, in doing so, they establish the normative validity of their conduct on the “logic of appropriateness” — they conduct themselves in ways they deem appropriate for specific situations (Alexander et al., 2022 , p. 8). Therefore, they may not view non-compliance with formal rules and regulations as corruption and vice versa.

South Africa’s anti-corruption appears to be confined within the orthodoxy. Essentially, the corpora suggest that the country’s anti-corruption regime adheres to the principal-agent model, which hinges primarily on oversights, regulations, and enforcing formal regulations and bureaucratic procedures. However, this model is undermined by the absence of ethical ‘principals,’ whose integrity is essential for its success. The corpora indicate that corruption in South Africa is better understood as a collective action problem, wherein corrupt behavior is perceived as “normal” and mutually expected by both principals and agents. This situation results in a lack of incentive from both parties to combat it. Nonetheless, the collective action theory does not offer an exhaustive account for understanding and addressing corruption in South Africa.

Beyond the orthodoxy

The consensus apparent within the corpora concerning the robustness of South Africa’s anti-corruption legislation is fraught with challenges. This consensus is based on South Africa’s adherence to the UNCAC and international best practices. As a result, the corpora exonerate the legislation prematurely, shifting the attention toward law enforcement and policy implementation. However, as posited by Langendorf ( 2015 ), South Africa’s corruption problems indicate that it may have followed the UNCAC roadmap but has undoubtedly gotten lost because the map is not the territory. Undoubtedly, the international anti-corruption consensus faces particular difficulties concerning the tension between the universality of the anti-corruption norms and its simultaneous contextualization of specific and local application” (Gephart, 2009 , p. 4). Hence, the conformance of South Africa’s anti-corruption regime with the UNCAC and international best practices may inadvertently undermine its sensitivity to the country’s socio-political and legal context.

Furthermore, such adherence to international law may also subjugate the country’s anti-corruption regime to the domination, control, and neo-liberal interests of the Global North or the West. However, this claim cannot be considered a given because universal norms can arise through participatory crafting processes (Fassbender et al., 2012 ). Besides, Africa is an active innovator and generator of norms within the international legal order (Gathii, 2012 ; Levitt, 2008 ).

Nevertheless, scholars of the Third World Approaches to International Law (TWAIL) maintain that international law extends a historical pattern that privileges Western interests. These scholars argue that international law has played a complicit role in fostering neo-colonial relationships, perpetuating a relation in which the Global North imposes its social, economic, and political interests on the Global South (Mutua, 2000 , p. 31). As Chimni ( 2022 , p. 46) contends, contemporary imperialism has shifted from direct colonization to consensually negotiated shared rules, deploying multi-layered strategies like power, mediation, and revolutionary approaches to achieve harmonization objectives. These also embody a specific vision of how states around the world are supposed to function - namely, according to the ideal of a Western state, and if they deviate from this ideal, they are seen as failed, fragile, or corrupt. Therefore, anti-corruption researchers must be critical of the UNCAC and other international standards as they may primarily disseminate, validate, sustain, and impose Western interests, ideologies, and governance structures on the Global South (Bracking, 2014 ; Brown & Cloke, 2011 ; Snyman, 2021 ).

As part of a global disciplinary effort to manage uncertainties, reduce risks, and establish normalized governance, the international anti-corruption consensus allows its actors to present their involvement as impartial, apolitical, and driven by combating a universal problem. However, their proposed solutions inherently carry political and ideological underpinnings, albeit obscured by layers of technocratic language (Snyman, 2022 ). Indeed, recent scholarship from critical corruption researchers such as Joaquin Villanueva ( 2019 ) and Jose Atiles ( 2020 , 2023a , b ; Atiles et al., 2022 ) serves to illuminate how discourses on corruption operate politically to extend, justify, and contest neo-colonial relationships. Unfortunately, appraisals of South Africa’s anti-corruption regime often lack a critical approach to international law and do not adequately draw on actor perspectives and the rich social contexts within which regulations are enforced.

Nonetheless, the smaller corpus three suggests that the UNCAC norms embedded in the country’s anti-corruption laws and policies do not automatically resonate with some cross-section of people in South Africa who, through informal norms, practices, and local rationalities, contest them. Meanwhile, this nuanced and impactful disparity is yet to garner significant attention within South Africa’s anti-corruption corpora. Nevertheless, the corpora lay an essential empirical and theoretical foundation for future research.

Furthermore, the corpora indicate the prevalence of other normative orders besides the state legal system that shape public attitudes, experiences, and behaviors toward anti-corruption laws and regulations. This phenomenon is exemplified by the incongruent normative stances between the ANC political party and the South African Parliament on the one hand—which resolved to disband the DSO—and the prescripts of state law as manifest in the South African Supreme Court’s adjudication that deemed the disbandment unconstitutional (refer to Corpus two ). Although various studies in the reviewed body of work hint at the existence of semi-autonomous social fields, they fail to thoroughly analyze these phenomena through the lens of legal pluralism (see, for example, Alexander et al., 2022 ; Bähre, 2005 ; Bruce, 2014 ; Gray, 2021 ).

Notwithstanding, empirical evidence supporting the prevalence of semi-autonomous fields in South Africa challenges the idea that anti-corruption measures fail due to widespread rule-breaking. Through the prism of legal pluralism, it becomes evident that adherence to rules is indeed prevalent in the country, albeit often not aligned with formal state legislation and regulations. These findings also corroborate legal consciousness scholarship, which proposes that a complex combination of norms shapes individuals’ understanding of legality. Therefore, prioritizing empirical examination of everyday social logic, local cultural categories, and norms concerning law can assist researchers in unravelling the complexities of legality that shape individuals’ understanding and attitudes towards anti-corruption initiatives. The country’s anti-corruption corpora suggest a significant interconnection between its social, cultural, and legal spheres. However, when examining the role of law in combatting corruption, the corpora predominantly adopt an instrumentalist perspective. Meanwhile, this is not peculiar to anti-corruption research on South Africa. The instrumental perspective dominates anti-corruption studies globally, which, in de Sa e Silva’s view ( 2022 , p. 365), is perhaps due to the field’s orientation to policy.

Thus, rooted in the instrumental view, the failure of South Africa’s anti-corruption regime to significantly reduce corruption is almost entirely attributed to the country’s inability to effectively enforce its sound anti-corruption legislation and institutional framework due to a lack of principled principals, political will, and dedication. Indeed, a perplexing gap remains between the theoretical intentions of the country’s anti-corruption regime (promising accountability, transparency, and institutional autonomy) and the practical realities of endemic impunity and political interference. However, the corpora fail to provide a sufficient explanation of this paradox. The corpora’s limitation suggests the necessity for a critical understanding of the intricate interaction between law and society.

This paper contends that the complex nature of corruption in South Africa calls for a critical reorientation within its anti-corruption scholarship. Scholars must shift from an instrumental perspective to a constitutive understanding of the law’s role in addressing corruption. This shift entails a comprehensive understanding of the interplay between law and society, considering both the “pull” – the constraints imposed by formal law – and the “push” – individuals’ interpretations of the law (Marshall & Barclay, 2003 , p. 618). This shift could provide a more nuanced understanding of the intricacies of corruption and contribute to formulating more effective strategies to combat it. Despite their potential value, these socio-legal perspectives have not garnered significant focus from anti-corruption scholarship in South Africa and globally.

South Africa’s anti-corruption corpora spanning the last two decades attribute the country’s failing anti-corruption primarily to the lack of dedicated and proactive leadership to guarantee effective monitoring, supervision, and enforcement of the country’s anti-corruption regime. This paper maintains that although comprehensible, this accumulated scholarship appears to be confined within ‘the orthodoxy’ and does not offer a sufficient understanding of why corruption is still endemic in the country. Subsequently, the paper critically engages with the corpora in a humble attempt to drive it beyond the orthodoxy.

Subsequently, it draws on TWAIL scholarships to critique the corpora’s consensus that South Africa’s anti-corruption legislative and institutional framework is sound primarily because it adheres closely to the UNCAC and international best practices. It argues that although the UNCAC as international law may appear to be apolitical or neutral, it may also embed political and ideological underpinnings that advance the interests of the Global North over the Global South. Subsequently, the close adherence of South Africa’s anti-corruption regime to the UNCAC and other international best practices instead necessitates critical empirical enquiry.

Moreover, the corpora demonstrate that a nexus between formal and informal norms shapes individuals’ worldviews and behavior toward anti-corruption regulations and their perception of legality. This intricacy calls for a nuanced empirical enquiry into the country’s anti-corruption regulations—one that delves into people’s internalized perceptions and ideas of the law that shape their daily conduct. However, scholars of South Africa’s anti-corruption regime have yet to inculcate such a constitutive perspective of law into their inquiries. This paper concludes that adopting a constitutive perspective, using socio-legal approaches like legal pluralism and legal consciousness, can offer a more nuanced understanding of South Africa’s failing anti-corruption regime beyond the ‘orthodoxy.’

Data availability

The author confirms that all data generated or analysed during this study are included in this published article and its supplementary information files.

Alexander, C., Berrisford, S., Nkula-Wenz, L., Ndhlovu, D., Siame, G., Watson, V., & Zinnbauer, D. (2022). Challenges and opportunities of curbing urban corruption and building professional integrity: Experiences of planners in South Africa and Zambia. Habitat International, 122 , 102541.  https://doi.org/10.1016/j.habitatint.2022.102541

April, F. Y., & Sebola, M. (2016). A comparative analysis of corruption in South Africa and China: Evidence from the application of governance theory. Africa Insight, 46 (3), 83–100.

Google Scholar  

Atiles, J. (2020). One of the most corrupt places on earth: colonialism, (anti) corruption, and the Puerto Rican Summer of 2019. Special issue. Society and space. https://www.societyandspace.org/articles/one-of-the-most-corrupt-places-on-earth-colonialism-anti-corruption-and-the-puerto-rican-summer-of-2019 . Accessed 2023-12-05.

Atiles, J. (2023a). Coloniality of anti-corruption: Whiteness, disasters, and the US anti-corruption policies in Puerto Rico. The Sociological Review, 71 (6), 1277–1298. https://doi.org/10.1177/00380261231153751

Article   Google Scholar  

Atiles, J. (2023b). On colonial exceptionality, neoliberal coloniality, and legal interruptions. Dialogues in Human Geography, 13 (1), 149–152. https://doi.org/10.1177/20438206221102936

Atiles, J., García López, G. A., & Villanueva, J. (2022). Beyond corruption and anti-corruption narratives: Introducing a critical research agenda for Puerto Rican studies. Centro Journal, 34 (2), 9–25.

Bähre, E. (2005). How to ignore corruption: reporting the shortcomings of development in South Africa. Current Anthropology, 46 (1), 107–120. https://doi.org/10.1086/424768

Bauer, C. (2000). Public sector corruption and its control in South Africa. In K. R. Hope & B. C. Chikulo (Eds.), Corruption and Development in Africa (pp. 218–233). Palgrave Macmillan UK. https://doi.org/10.1057/9780333982440_12

Bauer, C. (1999). Administrative corruption in South Africa. Southern Journal for Contemporary History, 24 (1), 76–97.

Bauhr, M. (2017). Need or Greed? Conditions for collective action against corruption. Governance, 30 (4), 561–581. https://doi.org/10.1111/gove.12232

Berning, J., & Montesh, M. (2012). Countering corruption in South Africa the rise and fall of the scorpions and hawks. South African Crime Quarterly-Sacq, 39 , 3–10.

Bierschenk, T. (2008). The everyday functioning of an African Public Service: Informalization, privatization and corruption in Benin’s Legal System. The Journal of Legal Pluralism and Unofficial Law, 40 (57), 101–139. https://doi.org/10.1080/07329113.2008.10756619

Bracking, S. (2014). Corruption and development: The mutable edges of morality in modern markets. Routledge Handbook of Political Corruption (pp. 225–241). https://doi.org/10.4324/9781315739175-27

Brogden, M., & Nijhar, P. (1998). Corruption and the South African police. Crime, Law and Social Change, 30 , 89–106.

Brown, E., & Cloke, J. (2011). Critical perspectives on corruption: an overview. Critical Perspectives on International Business, 7 , 116–124. https://doi.org/10.1108/17422041111128203

Bruce, D. (2014). Control, discipline and punish? Addressing corruption in South Africa. South African Crime Quarterly-Sacq, 48 , 49–62. https://doi.org/10.4314/sacq.v48i1.5

Budhram, T. (2015). Intelligence-led policing a proactive approach to combating corruption. South African Crime Quarterly-Sacq, 52 , 49–55. https://doi.org/10.4314/sacq.v52i1.5

Budhram, T., & Geldenhuys, N. (2018). Combating corruption in South Africa: Assessing the performance of investigating and prosecuting agencies. Acta Criminologica : African Journal of Criminology & Victimology, 31 (2), 23–46. https://doi.org/10.10520/EJC-139f4dd49f

Camerer, M. (2009). Corruption and reform in democratic South Africa . https://www.researchgate.net/profile/Marianne-Camerer/publication/265620450_Corruption_And_Reform_In_Democratic_South_Africa/links/5cda7cea299bf14d959513a5/Corruption-And-Reform-In-Democratic-South-Africa.pdf . Accessed 2023-12-07.

Chetty, J., & Pillay, P. (2017). Independence of anti–corruption agencies: A comparative study of South Africa and India. African Journal of Public Affairs, 9 (8), 105–120. https://doi.org/10.10520/EJC-ab561c268

Chimni, B. S. (2022). The international law of jurisdiction: A TWAIL perspective. Leiden Journal of International Law, 35 (1), 29–54.

Dassah, M. O. (2014). Single-and multi-agency approaches to fighting corruption: Lessons for South Africa. Loyola Journal of Social Sciences, 28 (2), 265–287.

Depraetere, J., Vandeviver, C., Keygnaert, I., & Beken, T. V. (2021). The critical interpretive synthesis: An assessment of reporting practices. International Journal of Social Research Methodology, 24 (6), 669–689. https://doi.org/10.1080/13645579.2020.1799637

Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., ... & Sutton, A. J. (2006). Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC medical research methodology , 6 , 1–13.

Ewick, P., & Silbey, S. S. (1998). The common place of law: stories from everyday life . University of Chicago Press. https://books.google.com/books?hl=en&lr=&id=H6Cil7eLCUcC&oi=fnd&pg=PR9&dq=the+commonplace+of+law&ots=emD-JEk0Pf&sig=S99HaSzrytavT29zohvlKd9scws . Accessed 2023-11-15.

e Silva, F. D. S. (2022). Relational legal consciousness and anticorruption: Lava Jato, social media interactions, and the co-production of law’s detraction in Brazil (2017–2019). Law & Society Review, 56 (3), 344–368.

Fassbender, B., Peters, A., Peter, S., & Högger, D. (2012). The Oxford Handbook of the History of International Law . OUP Oxford.

Book   Google Scholar  

Faull, A. (2011). Oversight agencies in South Africa and the challenge of police corruption. Institute for Security Studies Papers, 2011 (227), 20. https://doi.org/10.10520/EJC48895

Flemming, K. (2010). Synthesis of quantitative and qualitative research: An example using critical interpretive synthesis. Journal of Advanced Nursing, 66 (1), 201–217. https://doi.org/10.1111/j.1365-2648.2009.05173.x

Ganahl, J. P. (2014). Corruption, good governance, and the African state: A critical analysis of the political-economic foundations of corruption in sub-saharan Africa (Vol. 2). Universitätsverlag Potsdam.

Gathii, J. T. (2012). Africa and the history of international law. In B. Fassbender, A. Peters, S. Peter, & D. Högger (Eds.), The Oxford handbook of the history of international law. OUP Oxford.

Gephart, M. (2009). Contextualizing conceptions of corruption: Challenges for the international anti-corruption campaign . German Institute for Global and Area Studies (GIGA).

Global Initiative Against Transnational Organized Crime. (2022). Strategic organized crime risk assessment . Global Initiative Against Transnational Organized Crime.

Gough, D., & Richardson, M. (2018). Systematic reviews. In Advanced research methods for applied psychology (pp. 63–75). Routledge.

Gray, N. (2021). When anti-corruption fails: The dynamics of procurement in contemporary South Africa. Review of African Political Economy, 48 (169), 369–384. https://doi.org/10.1080/03056244.2021.1932789

Griffiths, J. (2003). The social working of legal rules. The Journal of Legal Pluralism and Unofficial Law, 35 (48), 1–84.

Gutterman, E., & Lohaus, M. (2018). What is the anti-corruption norm in global politics? In I. Kubbe & A. Engelbert (Eds.), Corruption and Norms (pp. 241–268). Springer International Publishing. https://doi.org/10.1007/978-3-319-66254-1_12

Chapter   Google Scholar  

Halliday, S., & Morgan, B. (2013). I fought the law and the law won? Legal consciousness and the critical imagination. Current Legal Problems, 66 (1), 1–32.

Hertogh, M. (2004). A ‘European’conception of legal consciousness: rediscovering Eugen Ehrlich. Journal of Law and Society, 31 (4), 457–481.

Holen, S. V. (2023). A duty to protect? Legal consciousness among military officers in armed conflict. Journal of Law and Society, 50 (1), 17–38. https://doi.org/10.1111/jols.12405

Hyslop, J. (2005). Political corruption: Before and after apartheid. Journal of Southern African Studies, 31 (4), 773–789. https://doi.org/10.1080/03057070500370555

Imiera, P. P. (2020). The corruption race in Africa: Nigeria versus South Africa, who cleans the mess first? De Jure Law Journal, 53 (1), 70–89. https://doi.org/10.17159/2225-7160/2020/v53a5

Integrity, G. (2008). Global integrity report . Washington, DC: Global Integrity.

Jackson, D. (2020). How change happens in anti-corruption: A map of policy perspectives. U4.Chr . Michelsen Institute (CMI).

James, E. (2023). South Africa anti-corruption charter ‘first step’ towards meaningful change . Pinsent Masons. https://www.pinsentmasons.com/out-law/analysis/south-africa-anti-corruption-charter-first-step-towards-meaningful-change . Accessed 2023-04-19.

Kanjere, M., & Koto, A. (2021). Public procurement corruption and corporate scandals in Limpopo Province, South Africa: A content analysis perspective. Acta Universitatis Danubius Relationes Internationales, 14 (2), 2. https://dj.univ-danubius.ro/index.php/AUDRI/article/view/1495

Kaufmann, D., Gallina, A., & Senderowitsch, R. (2015). A Collective Action Approach Against Corruption: The Case of the Dominican Republic . Washington, DC: World Bank.

Keightley, R. (2011). Fat cats, slim pickings: An examination of corporate corruption and the efficacy of existing countermeasures in South Africa. Journal of Corporate Law Studies, 11 (2), 343–368. https://doi.org/10.5235/147359711798110619

Khan, M., Andreoni, A., & Roy, P. (2019). Anti-corruption in adverse contexts: strategies for improving implementation. Anti-Corruption Evidence (ACE) Research Consortium. SOAS-ACE.

Kinnes, I., & Newham, G. (2012). Feeling the Hawks: Why an anti-corruption agency should not be in the SAPS. South African Crime Quarterly, 39 , 33–39. https://doi.org/10.4314/sacq.v39i0

Krygier, M. (1990). Marxism and the rule of law: Reflections after the collapse of communism. Law & Social Inquiry, 15 (4), 633–663.

Kuldova, T. Ø., Østbø, J., & Raymen, T. (2024). Luxury and Corruption: Challenging the Anti-Corruption Consensus . Policy Press. https://books.google.com/books?hl=enandlr=andid=ZB7oEAAAQBAJandoi=fndandpg=PR4anddq=luxury+and+corruption+challenging+the+andots=1BhlCLavhDandsig=I8DQIlGSVit6OEjRiMABHhTq-d4 . Accessed 2024-03-26.

Kurakin, A., & Sukharenko, A. (2018). Anti-corruption in the BRICS countries. BRICS Law Journal, 5 (1), 56–77. https://doi.org/10.21684/2412-2343-2017-5-1-56-77

Langendorf, G. (2015). The map is not the territory: How South Africa followed the anti-corruption roadmap and got lost along the way. University of Baltimore Journal of International Law, 3 (2), 31–88.

Lekubu, B. K., & Sibanda, O. S. (2021). Moral values and ethics as antidotes for corruption in the South African public service and administration. Koers, 86 (1), 1–12. https://doi.org/10.19108/KOERS.86.1.2482

Levitt, J. I. (2008). Africa: Mapping new boundaries in international law . Bloomsbury Publishing.

Lodge, T. (2001). Countering public corruption in South Africa. Transformation , 46 . Retrieved 29 April 2024, from https://scholar.archive.org/work/iryd3ye7zbft7lk3m3dhmz6sim/access/wayback/http://pdfproc.lib.msu.edu/?file=/DMC/African%20Journals/pdfs/transformation/tran046/tran046004.pdf

Lodge, T. (1998). Political corruption in South Africa. African Affairs, 97 (387), 157–187. https://doi.org/10.1093/Oxfordjournals.Afraf.A007924

Majila, T., Taylor, J. D., & Raga, K. (2014). A comparative analysis of the implementation of anti-corruption legislation by anti-corruption agencies in the provinces of the Eastern and Northern Cape. Td-the Journal for Transdisciplinary Research in Southern Africa , 10 (1). https://td-sa.net/index.php/td/article/download/21/148 . Accessed 2022-05-25.

Makiva, M. (2021). Public procurement oversight and the scourge of corruption in the public sector: A comparative analysis of South Africa and Kenya. In N. Dorasamy & O. Fagbadebo (Eds.), Public Procurement, Corruption and the Crisis of Governance in Africa (pp. 223–248). Springer International Publishing. https://doi.org/10.1007/978-3-030-63857-3_12

Mantzaris, E. (2016). Innovative leadership against corruption in the public sector: The case for South Africa . https://repository.up.ac.za/handle/2263/58185 . Accessed 5/24/2022.

Manyaka, R. K., & Nkuna, N. W. (2014). The phenomenon of corruption in the South African public sector: Challenges and opportunities. Mediterranean Journal of Social Sciences, 5 (27), 1572–1580. https://doi.org/10.5901/MJSS.2014.V5N27P1572

March, J. G., & Olsen, J. P. (1994). Institutional perspectives on governance. Arena . https://gestionypoliticapublica.cide.edu/ojscide/num_anteriores/Vol.VI._No.I_1ersem/Abstracts_Vol.6_No.I_1sem.pdf . Accessed 2024-01-08.

Marquette, H., & Peiffer, C. (2015a). Collective action and systemic corruption (p. 29). ECPR Joint Sessions of Workshops University of Warsaw.

Marquette, H., & Peiffer, C. (2015b). Corruption and collective action . DLP Research Paper.

Marshall, A. M., & Barclay, S. (2003). In their own words: How ordinary people construct the legal world. Law & Social Inquiry, 28 (3), 617–628.

Maseko, T. W. (2021). The feasibility of the victims of corruption’s claim for constitutional damages against corrupt public officials in South Africa. De Jure, 54 , 127–140. https://doi.org/10.17159/2225-7160/2021/v54a8

Merry, S. E. (1988). Legal pluralism. Law & Soc’y Rev, 22 , 869.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & Group, T. P. (2009). Preferred reporting items for systematic reviews and Meta-analyses: The PRISMA statement. PLOS Medicine, 6 (7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

Montesh, M., & Berning, J. (2012). A need for a single anti-corruption agency in South Africa: A comparative study. Acta Criminologica: African Journal of Criminology & Victimology, 2012 (sed-1), 117–137.

Moore, S. F. (1972). Law and social change: The semi-autonomous social field as an appropriate subject of study. Law & Soc’y Rev, 7 , 719.

Mphendu, U., & Holtzhausen, N. (2016). Successful anti-corruption initiatives in Botswana, Singapore and Georgia: Lessons for South Africa . https://repository.up.ac.za/handle/2263/61220 . Accessed 5/30/2022.

Mubangizi, J. C. (2020). A human rights based approach to fighting corruption in Uganda and South Africa: Shared perspectives and comparative lessons. Law Democracy & Development, 24 (1), 1.

Mubangizi, J. C., & Sewpersadh, P. (2017). A Human rights-based approach to combating public procurement corruption in Africa. African Journal of Legal Studies, 10 (1), 66–90. https://doi.org/10.1163/17087384-12340015

Mungiu-Pippidi, A. (2013). Controlling corruption through collective action. Journal of Democracy, 24 (1), 101–115.

Munzhedzi, P. H. (2016). South African public sector procurement and corruption: Inseparable twins? Journal of Transport and Supply Chain Management, 10 (1), 1–8.

Mutua, M. (2000). What is TWAIL? Proceedings of the ASIL Annual Meeting, 94 , 31–38. https://www.cambridge.org/core/journals/proceedings-of-the-asil-annual-meeting/article/what-is-twail/F6186DDA7E7CBFB50CC61A2D7836C5F0 . Accessed 2023-12-11.

Naidoo, G. (2012). A critical need for ethical leadership to curb corruption and promote good governance in the public sector of South Africa. African Journal of Public Affairs. https://repository.up.ac.za/handle/2263/57856 . Accessed 2022-08-11.

Naidoo, V. (2013). The politics of anti-corruption enforcement in South Africa. Journal of Contemporary African Studies, 31 (4), 523–542. https://doi.org/10.1080/02589001.2013.839369

Noblit, G. W., Hare, R. D., & Hare, R. (1988). Meta-ethnography: Synthesizing qualitative studies . Sage.

Nye, J. S. (1967). Corruption and political development: A cost-benefit analysis. American Political Science Review, 61 (2), 417–427.

Nzo, T. (2016). Provincial-regional ANC politics in the Northern Cape: Corruption or everyday informal practices? Commonwealth Journal of Local Governance, 19 , 96–118. https://doi.org/10.5130/cjlg.v0i19.5490

Odeku, K. O. (2019). Unmasking life-style audit as a proactive mechanism to root out corruption: the case of South Africa. Journal of Legal, Ethical and Regulatory, 22 (6), 13.

Okafor, E. E. (2009). Combating high-profile corruption in Transitional1Societies: Overview of experiences from some African Countries. The Anthropologist, 11 (2), 117–127. https://doi.org/10.1080/09720073.2009.11891091

Olson, M. (1971). The logic of collective action: Public goods and the theory of groups (Vol. 124). Harvard University Press. https://books.google.com/books?hl=en&lr=&id=jzTeOLtf7_wC&oi=fnd&pg=PA1&ots=5IHxcKEORf&sig=0BFawWGJppQrfWnlJ7728NwO2QY . Accessed 2024-01-04.

Olutola, A. (2014). The ruling political parties and the war against crime and corruption in Nigeria and South Africa: The case of two brothers with headless bodies. Acta Criminologica : African Journal of Criminology & Victimology, 2014 (sed-2), 81–95. https://doi.org/10.10520/EJC171117

Overman, S., Van Thiel, S., & Lafarge, F. (2014). Resisting governmental control: How semi-autonomous agencies use strategic resources to challenge state coordination. International Review of Administrative Sciences, 80 (1), 172–192.

Parboteeah, K. P., Seriki, H. T., & Hoegl, M. (2014). Ethnic diversity, corruption and ethical climates in sub-saharan Africa: Recognizing the significance of human resource management. The International Journal of Human Resource Management, 25 (7), 979–1001. https://doi.org/10.1080/09585192.2013.815251

Patel, J., & Govindasamy, P. (2021). South Africans see corruption as worsening during President Ramaphosa’s Tenure.  Afrobarometer .

Persson, A., Rothstein, B., & Teorell, J. (2010). The failure of anti-corruption policies: A theoretical mischaracterization of the problem . https://gupea.ub.gu.se/handle/2077/39039 . Accessed 2022-06-21.

Persson, A., Rothstein, B., & Teorell, J. (2013). Why anticorruption reforms fail—systemic corruption as a collective action problem. Governance, 26 (3), 449–471.

Pillay, P. (2017). Anti-corruption agencies in South Africa and Brazil and challenges. African Journal of Public Affairs, 9 (8), 1–14. https://doi.org/10.10520/EJC-ab4f8b4ab

Pitsoe, V. J. (2013). Values education as a social instrument for reducing corruption, poverty and inequality. Mediterranean Journal of Social Sciences, 4 (13), 745–753. https://doi.org/10.5901/MJSS.2013.V4N13P745

Quarcoo, S. C. (2009). Prosecution politics: Recalibrating the role of prosecution within the anticorruption agency agenda. African Security Review, 18 (4), 32–49. https://doi.org/10.1080/10246029.2009.9627556

Roelofse, C. J., Simonovic, B., & Potgieter, P. J. (2014). A comparative study of corruption and government efforts to combat it across Borders: The Case for South Africa and Serbia. Internal Security, 6 (2), 7–28.

Rose-Ackerman, S. (1997). Corruption, inefficiency and economic growth. Nordic Journal of Political Economy, 24 (1), 3–20.

Rose-Ackerman, S. (2008). Corruption. Readings in public choice and constitutional political economy (pp. 551–566). Springer.

Rothstein, B. (2021). Three reasons anti-corruption programs fail . CJL. https://www.corruptionjusticeandlegitimacy.org/post/three-reasons-anti-corruption-programs-fail . Accessed 2023-04-14.

Rothstein, B. (2011). Anti-corruption: The indirect ‘big bang’ approach. Review of International Political Economy, 18 (2), 228–250. https://doi.org/10.1080/09692291003607834

Safara, L., & Odeku, K. O. (2021). Critical legal perspective of international anti-corruption laws for tackling corruption in South Africa. Perspectives of Law and Public Administration, 10 (1), 203–216.

Sampson, S. (2005). Integrity warriors: Global morality and the anti-corruption movement in the Balkans. Corruption: Anthropological Perspectives, 6 , 103–130.

Sampson, S. (2010). The anti-corruption industry: From movement to institution. Global Crime, 11 (2), 261–278. https://doi.org/10.1080/17440571003669258

Sarat, A., & Kearns, T. R. (2009). Law in everyday life . University of Michigan Press. https://books.google.com/books?hl=en&lr=&id=YyfUMtIjEEIC&oi=fnd&pg=PP8&dq=related:OheblcHIlmIJ:scholar.google.com/&ots=ucTMbMfNA-&sig=Nd_XontUUDmngpgKFftHFEpHMoE . Accessed 2023-11-15.

Seegers, A. (1993). Towards an understanding of the afrikanerisation of the South African state. Africa, 63 (4), 477–497.

Sewpersadh, P., & Mubangizi, J. C. (2017). Using the law to combat public procurement corruption in South Africa: Lessons from Hong Kong. Potchefstroom Electronic Law Journal , 20 . https://doi.org/10.17159/1727-3781/2017/V20I0A1359

Silbey, S. S. (2005). After legal consciousness. Annual Review of Law and Social Science, 1 , 323–368.

Snyman, R. A. (2021). Games of Truth in the age of transparency: International Organisations and the construction of corruption. Journal of Business Ethics . https://doi.org/10.1007/s10551-021-04922-0

Snyman, R. A. (2022). Games of truth in the age of transparency: International organisations and the construction of corruption. Journal of Business Ethics, 181 (1), 83–96.

Sundström, A. (2015). Covenants with broken swords: Corruption and law enforcement in governance of the commons. Global Environmental Change, 31 , 253–262. https://doi.org/10.1016/j.gloenvcha.2015.02.002

Sundström, A. (2019). Exploring performance-related pay as an anticorruption tool. Studies in Comparative International Development, 54 (1), 1–18. https://doi.org/10.1007/S12116-017-9251-0

Tanzi, V. (1998). Corruption around the world: Causes, consequences, scope, and cures. Staff Papers , 45 (4), 559–594.

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375

Transparency International (2022). What is corruption? Transparency.Org. https://www.transparency.org/en/what-is-corruption . Accessed 2022-06-28.

Tushnet, M. (2019). Institutions supporting constitutional democracy: Some thoughts about anti-corruption (and other) agencies. Singapore Journal of Legal Studies , 440 .

Urinboyev, R., & Svensson, M. (2018). Corruption, Social Norms and Everyday Life in Uzbekistan. In I. Kubbe & A. Engelbert (Eds.), Corruption and Norms. Political Corruption and Governance. Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-319-66254-1_10

van Niekerk, A. A. (2003). Can more business ethics teaching halt corruption in companies? South African Journal of Philosophy, 22 (2), 128–138. https://doi.org/10.4314/sajpem.v22i2.31365

Van Vuuren, H. (2006). Apartheid grand corruption. Addressing the scale of crimes of profit in South Africa from 1976 to 1994. A report prepared by civil society at the request of the Second National Anti-corruption Summit . Institute for Security Studies.

Villanueva, J. (2019). Corruption narratives and Colonial Technologies in Puerto Rico: How a long-term view of U.S colonialism in Puerto Rico reveals the contradictory political work of corruption discourse, both in extending colonial rule and in resisting it. NACLA Report on the Americas, 51 (2), 188–193. https://doi.org/10.1080/10714839.2019.1617489

World Bank. (1997). Helping countries combat corruption: The role of the World Bank. Poverty reduction and Economic Management (pp. 1–69). World Bank.

Zaloznaya, M. (2013). Beyond anti-corruptionism: Sociological imagination and comparative study of corruption*. Comparative Sociology, 12 (5), 705–751. https://doi.org/10.1163/15691330-12341281

Download references

Open access funding provided by University of Oslo (incl Oslo University Hospital) This study was funded by the Faculty of Law, University of Oslo.

Author information

Authors and affiliations.

Faculty of Law, Department of Criminology and Sociology of Law, University of Oslo, Oslo, Norway

Thomas Duke Labik Amanquandor

You can also search for this author in PubMed   Google Scholar

Contributions

This study was solely conducted and written by the author.

Corresponding author

Correspondence to Thomas Duke Labik Amanquandor .

Ethics declarations

Ethical approval.

Not applicable.

Informed consent

Statement regarding research involving human participants and/or animals.

Not applicable. 

Competing interests

The author has no competing interests to declare in relation to the content of this article.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Labik Amanquandor, T. Limitations of the international approach to anti-corruption: a systematic review of South Africa’s compelling case of failing anti-corruption. Crime Law Soc Change (2024). https://doi.org/10.1007/s10611-024-10152-y

Download citation

Accepted : 19 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1007/s10611-024-10152-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Anti-corruption
• South Africa
• Legal pluralism
• Legal consciousness
• Collective action theory
• Principal-agent theory
• Find a journal
• Publish with us
• Track your research