literature synthesis approach

Literature Syntheis 101

How To Synthesise The Existing Research (With Examples)

By: Derek Jansen (MBA) | Expert Reviewer: Eunice Rautenbach (DTech) | August 2023

One of the most common mistakes that students make when writing a literature review is that they err on the side of describing the existing literature rather than providing a critical synthesis of it. In this post, we’ll unpack what exactly synthesis means and show you how to craft a strong literature synthesis using practical examples.

This post is based on our popular online course, Literature Review Bootcamp . In the course, we walk you through the full process of developing a literature review, step by step. If it’s your first time writing a literature review, you definitely want to use this link to get 50% off the course (limited-time offer).

Overview: Literature Synthesis

• What exactly does “synthesis” mean?
• Aspect 1: Agreement
• Aspect 2: Disagreement
• Aspect 3: Key theories
• Aspect 4: Contexts
• Aspect 5: Methodologies
• Bringing it all together

What does “synthesis” actually mean?

As a starting point, let’s quickly define what exactly we mean when we use the term “synthesis” within the context of a literature review.

Simply put, literature synthesis means going beyond just describing what everyone has said and found. Instead, synthesis is about bringing together all the information from various sources to present a cohesive assessment of the current state of knowledge in relation to your study’s research aims and questions .

Put another way, a good synthesis tells the reader exactly where the current research is “at” in terms of the topic you’re interested in – specifically, what’s known , what’s not , and where there’s a need for more research .

So, how do you go about doing this?

Well, there’s no “one right way” when it comes to literature synthesis, but we’ve found that it’s particularly useful to ask yourself five key questions when you’re working on your literature review. Having done so,  you can then address them more articulately within your actual write up. So, let’s take a look at each of these questions.

1. Points Of Agreement

The first question that you need to ask yourself is: “Overall, what things seem to be agreed upon by the vast majority of the literature?”

For example, if your research aim is to identify which factors contribute toward job satisfaction, you’ll need to identify which factors are broadly agreed upon and “settled” within the literature. Naturally, there may at times be some lone contrarian that has a radical viewpoint , but, provided that the vast majority of researchers are in agreement, you can put these random outliers to the side. That is, of course, unless your research aims to explore a contrarian viewpoint and there’s a clear justification for doing so. 

Identifying what’s broadly agreed upon is an essential starting point for synthesising the literature, because you generally don’t want (or need) to reinvent the wheel or run down a road investigating something that is already well established . So, addressing this question first lays a foundation of “settled” knowledge.

Need a helping hand?

2. Points Of Disagreement

Related to the previous point, but on the other end of the spectrum, is the equally important question: “Where do the disagreements lie?” .

In other words, which things are not well agreed upon by current researchers? It’s important to clarify here that by disagreement, we don’t mean that researchers are (necessarily) fighting over it – just that there are relatively mixed findings within the empirical research , with no firm consensus amongst researchers.

This is a really important question to address as these “disagreements” will often set the stage for the research gap(s). In other words, they provide clues regarding potential opportunities for further research, which your study can then (hopefully) contribute toward filling. If you’re not familiar with the concept of a research gap, be sure to check out our explainer video covering exactly that .

3. Key Theories

The next question you need to ask yourself is: “Which key theories seem to be coming up repeatedly?” .

Within most research spaces, you’ll find that you keep running into a handful of key theories that are referred to over and over again. Apart from identifying these theories, you’ll also need to think about how they’re connected to each other. Specifically, you need to ask yourself:

• Are they all covering the same ground or do they have different focal points  or underlying assumptions ?
• Do some of them feed into each other and if so, is there an opportunity to integrate them into a more cohesive theory?
• Do some of them pull in different directions ? If so, why might this be?
• Do all of the theories define the key concepts and variables in the same way, or is there some disconnect? If so, what’s the impact of this ?

Simply put, you’ll need to pay careful attention to the key theories in your research area, as they will need to feature within your theoretical framework , which will form a critical component within your final literature review. This will set the foundation for your entire study, so it’s essential that you be critical in this area of your literature synthesis.

If this sounds a bit fluffy, don’t worry. We deep dive into the theoretical framework (as well as the conceptual framework) and look at practical examples in Literature Review Bootcamp . If you’d like to learn more, take advantage of our limited-time offer to get 60% off the standard price.

4. Contexts

The next question that you need to address in your literature synthesis is an important one, and that is: “Which contexts have (and have not) been covered by the existing research?” .

For example, sticking with our earlier hypothetical topic (factors that impact job satisfaction), you may find that most of the research has focused on white-collar , management-level staff within a primarily Western context, but little has been done on blue-collar workers in an Eastern context. Given the significant socio-cultural differences between these two groups, this is an important observation, as it could present a contextual research gap .

In practical terms, this means that you’ll need to carefully assess the context of each piece of literature that you’re engaging with, especially the empirical research (i.e., studies that have collected and analysed real-world data). Ideally, you should keep notes regarding the context of each study in some sort of catalogue or sheet, so that you can easily make sense of this before you start the writing phase. If you’d like, our free literature catalogue worksheet is a great tool for this task.

5. Methodological Approaches

Last but certainly not least, you need to ask yourself the question: “What types of research methodologies have (and haven’t) been used?”

For example, you might find that most studies have approached the topic using qualitative methods such as interviews and thematic analysis. Alternatively, you might find that most studies have used quantitative methods such as online surveys and statistical analysis.

But why does this matter?

Well, it can run in one of two potential directions . If you find that the vast majority of studies use a specific methodological approach, this could provide you with a firm foundation on which to base your own study’s methodology . In other words, you can use the methodologies of similar studies to inform (and justify) your own study’s research design .

On the other hand, you might argue that the lack of diverse methodological approaches presents a research gap , and therefore your study could contribute toward filling that gap by taking a different approach. For example, taking a qualitative approach to a research area that is typically approached quantitatively. Of course, if you’re going to go against the methodological grain, you’ll need to provide a strong justification for why your proposed approach makes sense. Nevertheless, it is something worth at least considering.

Regardless of which route you opt for, you need to pay careful attention to the methodologies used in the relevant studies and provide at least some discussion about this in your write-up. Again, it’s useful to keep track of this on some sort of spreadsheet or catalogue as you digest each article, so consider grabbing a copy of our free literature catalogue if you don’t have anything in place.

Bringing It All Together

Alright, so we’ve looked at five important questions that you need to ask (and answer) to help you develop a strong synthesis within your literature review.  To recap, these are:

• Which things are broadly agreed upon within the current research?
• Which things are the subject of disagreement (or at least, present mixed findings)?
• Which theories seem to be central to your research topic and how do they relate or compare to each other?
• Which contexts have (and haven’t) been covered?
• Which methodological approaches are most common?

Importantly, you’re not just asking yourself these questions for the sake of asking them – they’re not just a reflection exercise. You need to weave your answers to them into your actual literature review when you write it up. How exactly you do this will vary from project to project depending on the structure you opt for, but you’ll still need to address them within your literature review, whichever route you go.

The best approach is to spend some time actually writing out your answers to these questions, as opposed to just thinking about them in your head. Putting your thoughts onto paper really helps you flesh out your thinking . As you do this, don’t just write down the answers – instead, think about what they mean in terms of the research gap you’ll present , as well as the methodological approach you’ll take . Your literature synthesis needs to lay the groundwork for these two things, so it’s essential that you link all of it together in your mind, and of course, on paper.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

excellent , thank you

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• University of Oregon Libraries
• Research Guides

How to Write a Literature Review

• 6. Synthesize
• Literature Reviews: A Recap
• Reading Journal Articles
• Does it Describe a Literature Review?
• 1. Identify the Question
• 2. Review Discipline Styles
• Searching Article Databases
• Finding Full-Text of an Article
• Citation Chaining
• When to Stop Searching
• 4. Manage Your References
• 5. Critically Analyze and Evaluate

Synthesis Visualization

Synthesis matrix example.

• 7. Write a Literature Review

• Synthesis Worksheet

About Synthesis

Approaches to synthesis.

You can sort the literature in various ways, for example:

How to Begin?

Read your sources carefully and find the main idea(s) of each source

Look for similarities in your sources – which sources are talking about the same main ideas? (for example, sources that discuss the historical background on your topic)

Use the worksheet (above) or synthesis matrix (below) to get organized

This work can be messy. Don't worry if you have to go through a few iterations of the worksheet or matrix as you work on your lit review!

Four Examples of Student Writing

In the four examples below, only ONE shows a good example of synthesis: the fourth column, or  Student D . For a web accessible version, click the link below the image.

Long description of "Four Examples of Student Writing" for web accessibility

• Download a copy of the "Four Examples of Student Writing" chart

Click on the example to view the pdf.

From Jennifer Lim

• << Previous: 5. Critically Analyze and Evaluate
• Next: 7. Write a Literature Review >>
• Last Updated: May 3, 2024 5:17 PM
• URL: https://researchguides.uoregon.edu/litreview

Contact Us Library Accessibility UO Libraries Privacy Notices and Procedures

1501 Kincaid Street Eugene, OR 97403 P: 541-346-3053 F: 541-346-3485

• Visit us on Facebook
• Visit us on Twitter
• Visit us on Youtube
• Visit us on Instagram
• Report a Concern
• Nondiscrimination and Title IX
• Accessibility
• Privacy Policy
• Find People

The Sheridan Libraries

• Write a Literature Review
• Sheridan Libraries
• Find This link opens in a new window
• Evaluate This link opens in a new window

Get Organized

• Lit Review Prep Use this template to help you evaluate your sources, create article summaries for an annotated bibliography, and a synthesis matrix for your lit review outline.

Synthesize your Information

Synthesize: combine separate elements to form a whole.

Synthesis Matrix

A synthesis matrix helps you record the main points of each source and document how sources relate to each other.

After summarizing and evaluating your sources, arrange them in a matrix or use a citation manager to help you see how they relate to each other and apply to each of your themes or variables.  

By arranging your sources by theme or variable, you can see how your sources relate to each other, and can start thinking about how you weave them together to create a narrative.

• Step-by-Step Approach
• Example Matrix from NSCU
• Matrix Template
• << Previous: Summarize
• Next: Integrate >>
• Last Updated: Sep 26, 2023 10:25 AM
• URL: https://guides.library.jhu.edu/lit-review

How to Synthesize Written Information from Multiple Sources

Shona McCombes

Content Manager

B.A., English Literature, University of Glasgow

Shona McCombes is the content manager at Scribbr, Netherlands.

Learn about our Editorial Process

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

On This Page:

When you write a literature review or essay, you have to go beyond just summarizing the articles you’ve read – you need to synthesize the literature to show how it all fits together (and how your own research fits in).

Synthesizing simply means combining. Instead of summarizing the main points of each source in turn, you put together the ideas and findings of multiple sources in order to make an overall point.

At the most basic level, this involves looking for similarities and differences between your sources. Your synthesis should show the reader where the sources overlap and where they diverge.

Unsynthesized Example

Franz (2008) studied undergraduate online students. He looked at 17 females and 18 males and found that none of them liked APA. According to Franz, the evidence suggested that all students are reluctant to learn citations style. Perez (2010) also studies undergraduate students. She looked at 42 females and 50 males and found that males were significantly more inclined to use citation software ( p < .05). Findings suggest that females might graduate sooner. Goldstein (2012) looked at British undergraduates. Among a sample of 50, all females, all confident in their abilities to cite and were eager to write their dissertations.

Synthesized Example

Studies of undergraduate students reveal conflicting conclusions regarding relationships between advanced scholarly study and citation efficacy. Although Franz (2008) found that no participants enjoyed learning citation style, Goldstein (2012) determined in a larger study that all participants watched felt comfortable citing sources, suggesting that variables among participant and control group populations must be examined more closely. Although Perez (2010) expanded on Franz’s original study with a larger, more diverse sample…

Step 1: Organize your sources

After collecting the relevant literature, you’ve got a lot of information to work through, and no clear idea of how it all fits together.

Before you can start writing, you need to organize your notes in a way that allows you to see the relationships between sources.

One way to begin synthesizing the literature is to put your notes into a table. Depending on your topic and the type of literature you’re dealing with, there are a couple of different ways you can organize this.

Summary table

A summary table collates the key points of each source under consistent headings. This is a good approach if your sources tend to have a similar structure – for instance, if they’re all empirical papers.

Each row in the table lists one source, and each column identifies a specific part of the source. You can decide which headings to include based on what’s most relevant to the literature you’re dealing with.

For example, you might include columns for things like aims, methods, variables, population, sample size, and conclusion.

For each study, you briefly summarize each of these aspects. You can also include columns for your own evaluation and analysis.

The summary table gives you a quick overview of the key points of each source. This allows you to group sources by relevant similarities, as well as noticing important differences or contradictions in their findings.

Synthesis matrix

A synthesis matrix is useful when your sources are more varied in their purpose and structure – for example, when you’re dealing with books and essays making various different arguments about a topic.

Each column in the table lists one source. Each row is labeled with a specific concept, topic or theme that recurs across all or most of the sources.

Then, for each source, you summarize the main points or arguments related to the theme.

The purposes of the table is to identify the common points that connect the sources, as well as identifying points where they diverge or disagree.

Step 2: Outline your structure

Now you should have a clear overview of the main connections and differences between the sources you’ve read. Next, you need to decide how you’ll group them together and the order in which you’ll discuss them.

For shorter papers, your outline can just identify the focus of each paragraph; for longer papers, you might want to divide it into sections with headings.

There are a few different approaches you can take to help you structure your synthesis.

If your sources cover a broad time period, and you found patterns in how researchers approached the topic over time, you can organize your discussion chronologically .

That doesn’t mean you just summarize each paper in chronological order; instead, you should group articles into time periods and identify what they have in common, as well as signalling important turning points or developments in the literature.

If the literature covers various different topics, you can organize it thematically .

That means that each paragraph or section focuses on a specific theme and explains how that theme is approached in the literature.

Source Used with Permission: The Chicago School

If you’re drawing on literature from various different fields or they use a wide variety of research methods, you can organize your sources methodologically .

That means grouping together studies based on the type of research they did and discussing the findings that emerged from each method.

If your topic involves a debate between different schools of thought, you can organize it theoretically .

That means comparing the different theories that have been developed and grouping together papers based on the position or perspective they take on the topic, as well as evaluating which arguments are most convincing.

Step 3: Write paragraphs with topic sentences

What sets a synthesis apart from a summary is that it combines various sources. The easiest way to think about this is that each paragraph should discuss a few different sources, and you should be able to condense the overall point of the paragraph into one sentence.

This is called a topic sentence , and it usually appears at the start of the paragraph. The topic sentence signals what the whole paragraph is about; every sentence in the paragraph should be clearly related to it.

A topic sentence can be a simple summary of the paragraph’s content:

“Early research on [x] focused heavily on [y].”

For an effective synthesis, you can use topic sentences to link back to the previous paragraph, highlighting a point of debate or critique:

“Several scholars have pointed out the flaws in this approach.” “While recent research has attempted to address the problem, many of these studies have methodological flaws that limit their validity.”

By using topic sentences, you can ensure that your paragraphs are coherent and clearly show the connections between the articles you are discussing.

As you write your paragraphs, avoid quoting directly from sources: use your own words to explain the commonalities and differences that you found in the literature.

Don’t try to cover every single point from every single source – the key to synthesizing is to extract the most important and relevant information and combine it to give your reader an overall picture of the state of knowledge on your topic.

Step 4: Revise, edit and proofread

Like any other piece of academic writing, synthesizing literature doesn’t happen all in one go – it involves redrafting, revising, editing and proofreading your work.

Checklist for Synthesis

•   Do I introduce the paragraph with a clear, focused topic sentence?
•   Do I discuss more than one source in the paragraph?
•   Do I mention only the most relevant findings, rather than describing every part of the studies?
•   Do I discuss the similarities or differences between the sources, rather than summarizing each source in turn?
•   Do I put the findings or arguments of the sources in my own words?
•   Is the paragraph organized around a single idea?
•   Is the paragraph directly relevant to my research question or topic?
•   Is there a logical transition from this paragraph to the next one?

Further Information

How to Synthesise: a Step-by-Step Approach

Help…I”ve Been Asked to Synthesize!

Learn how to Synthesise (combine information from sources)

How to write a Psychology Essay

Literature Reviews

• 5. Synthesize your findings
• Getting started
• Types of reviews
• 1. Define your research question
• 2. Plan your search
• 3. Search the literature
• 4. Organize your results

How to synthesize

Approaches to synthesis.

• 6. Write the review
• Artificial intelligence (AI) tools
• Thompson Writing Studio This link opens in a new window
• Need to write a systematic review? This link opens in a new window

Contact a Librarian

Ask a Librarian

In the synthesis step of a literature review, researchers analyze and integrate information from selected sources to identify patterns and themes. This involves critically evaluating findings, recognizing commonalities, and constructing a cohesive narrative that contributes to the understanding of the research topic.

Here are some examples of how to approach synthesizing the literature:

💡 By themes or concepts

🕘 Historically or chronologically

📊 By methodology

These organizational approaches can also be used when writing your review. It can be beneficial to begin organizing your references by these approaches in your citation manager by using folders, groups, or collections.

Create a synthesis matrix

A synthesis matrix allows you to visually organize your literature.

Topic: ______________________________________________

Topic: Chemical exposure to workers in nail salons

• << Previous: 4. Organize your results
• Next: 6. Write the review >>
• Last Updated: May 8, 2024 8:12 AM
• URL: https://guides.library.duke.edu/lit-reviews

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

Literature Review Basics

• What is a Literature Review?
• Synthesizing Research
• Using Research & Synthesis Tables
• Additional Resources

Synthesis: What is it?

First, let's be perfectly clear about what synthesizing your research isn't :

• - It isn't  just summarizing the material you read
• - It isn't  generating a collection of annotations or comments (like an annotated bibliography)
• - It isn't  compiling a report on every single thing ever written in relation to your topic

When you  synthesize  your research, your job is to help your reader understand the current state of the conversation on your topic, relative to your research question.  That may include doing the following:

• - Selecting and using representative work on the topic
• - Identifying and discussing trends in published data or results
• - Identifying and explaining the impact of common features (study populations, interventions, etc.) that appear frequently in the literature
• - Explaining controversies, disputes, or central issues in the literature that are relevant to your research question
• - Identifying gaps in the literature, where more research is needed
• - Establishing the discussion to which your own research contributes and demonstrating the value of your contribution

Essentially, you're telling your reader where they are (and where you are) in the scholarly conversation about your project.

Synthesis: How do I do it?

Synthesis, step by step.

This is what you need to do  before  you write your review.

• Identify and clearly describe your research question (you may find the Formulating PICOT Questions table at  the Additional Resources tab helpful).
• Collect sources relevant to your research question.
• Organize and describe the sources you've found -- your job is to identify what  types  of sources you've collected (reviews, clinical trials, etc.), identify their  purpose  (what are they measuring, testing, or trying to discover?), determine the  level of evidence  they represent (see the Levels of Evidence table at the Additional Resources tab ), and briefly explain their  major findings . Use a Research Table to document this step.
• Study the information you've put in your Research Table and examine your collected sources, looking for  similarities  and  differences . Pay particular attention to  populations ,   methods  (especially relative to levels of evidence), and  findings .
• Analyze what you learn in (4) using a tool like a Synthesis Table. Your goal is to identify relevant themes, trends, gaps, and issues in the research.  Your literature review will collect the results of this analysis and explain them in relation to your research question.

Analysis tips

• - Sometimes, what you  don't  find in the literature is as important as what you do find -- look for questions that the existing research hasn't answered yet.
• - If any of the sources you've collected refer to or respond to each other, keep an eye on how they're related -- it may provide a clue as to whether or not study results have been successfully replicated.
• - Sorting your collected sources by level of evidence can provide valuable insight into how a particular topic has been covered, and it may help you to identify gaps worth addressing in your own work.
• << Previous: What is a Literature Review?
• Next: Using Research & Synthesis Tables >>
• Last Updated: Sep 26, 2023 12:06 PM
• URL: https://usi.libguides.com/literature-review-basics

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

512 Accesses

1 Citations

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Open access
• Published: 11 August 2009

Methods for the synthesis of qualitative research: a critical review

• Elaine Barnett-Page 1 &
• James Thomas 1  

BMC Medical Research Methodology volume  9 , Article number:  59 ( 2009 ) Cite this article

196k Accesses

1065 Citations

40 Altmetric

Metrics details

In recent years, a growing number of methods for synthesising qualitative research have emerged, particularly in relation to health-related research. There is a need for both researchers and commissioners to be able to distinguish between these methods and to select which method is the most appropriate to their situation.

A number of methodological and conceptual links between these methods were identified and explored, while contrasting epistemological positions explained differences in approaches to issues such as quality assessment and extent of iteration. Methods broadly fall into 'realist' or 'idealist' epistemologies, which partly accounts for these differences.

Methods for qualitative synthesis vary across a range of dimensions. Commissioners of qualitative syntheses might wish to consider the kind of product they want and select their method – or type of method – accordingly.

Peer Review reports

The range of different methods for synthesising qualitative research has been growing over recent years [ 1 , 2 ], alongside an increasing interest in qualitative synthesis to inform health-related policy and practice [ 3 ]. While the terms 'meta-analysis' (a statistical method to combine the results of primary studies), or sometimes 'narrative synthesis', are frequently used to describe how quantitative research is synthesised, far more terms are used to describe the synthesis of qualitative research. This profusion of terms can mask some of the basic similarities in approach that the different methods share, and also lead to some confusion regarding which method is most appropriate in a given situation. This paper does not argue that the various nomenclatures are unnecessary, but rather seeks to draw together and review the full range of methods of synthesis available to assist future reviewers in selecting a method that is fit for their purpose. It also represents an attempt to guide the reader through some of the varied terminology to spring up around qualitative synthesis. Other helpful reviews of synthesis methods have been undertaken in recent years with slightly different foci to this paper. Two recent studies have focused on describing and critiquing methods for the integration of qualitative research with quantitative [ 4 , 5 ] rather than exclusively examining the detail and rationale of methods for the synthesis of qualitative research. Two other significant pieces of work give practical advice for conducting the synthesis of qualitative research, but do not discuss the full range of methods available [ 6 , 7 ]. We begin our Discussion by outlining each method of synthesis in turn, before comparing and contrasting characteristics of these different methods across a range of dimensions. Readers who are more familiar with the synthesis methods described here may prefer to turn straight to the 'dimensions of difference' analysis in the second part of the Discussion.

Overview of synthesis methods

Meta-ethnography.

In their seminal work of 1988, Noblit and Hare proposed meta-ethnography as an alternative to meta-analysis [ 8 ]. They cited Strike and Posner's [ 9 ] definition of synthesis as an activity in which separate parts are brought together to form a 'whole'; this construction of the whole is essentially characterised by some degree of innovation, so that the result is greater than the sum of its parts. They also borrowed from Turner's theory of social explanation [ 10 ], a key tenet of which was building 'comparative understanding' [[ 8 ], p22] rather than aggregating data.

To Noblit and Hare, synthesis provided an answer to the question of 'how to "put together" written interpretive accounts' [[ 8 ], p7], where mere integration would not be appropriate. Noblit and Hare's early work synthesised research from the field of education.

Three different methods of synthesis are used in meta-ethnography. One involves the 'translation' of concepts from individual studies into one another, thereby evolving overarching concepts or metaphors. Noblit and Hare called this process reciprocal translational analysis (RTA). Refutational synthesis involves exploring and explaining contradictions between individual studies. Lines-of-argument (LOA) synthesis involves building up a picture of the whole (i.e. culture, organisation etc) from studies of its parts. The authors conceptualised this latter approach as a type of grounded theorising.

Britten et al [ 11 ] and Campbell et al [ 12 ] have both conducted evaluations of meta-ethnography and claim to have succeeded, by using this method, in producing theories with greater explanatory power than could be achieved in a narrative literature review. While both these evaluations used small numbers of studies, more recently Pound et al [ 13 ] conducted both an RTA and an LOA synthesis using a much larger number of studies (37) on resisting medicines. These studies demonstrate that meta-ethnography has evolved since Noblit and Hare first introduced it. Campbell et al claim to have applied the method successfully to non-ethnographical studies. Based on their reading of Schutz [ 14 ], Britten et al have developed both second and third order constructs in their synthesis (Noblit and Hare briefly allude to the possibility of a 'second level of synthesis' [[ 8 ], p28] but do not demonstrate or further develop the idea).

In a more recent development, Sandelowski & Barroso [ 15 ] write of adapting RTA by using it to ' integrate findings interpretively, as opposed to comparing them interpretively' (p204). The former would involve looking to see whether the same concept, theory etc exists in different studies; the latter would involve the construction of a bigger picture or theory (i.e. LOA synthesis). They also talk about comparing or integrating imported concepts (e.g. from other disciplines) as well as those evolved 'in vivo'.

Grounded theory

Kearney [ 16 ], Eaves [ 17 ] and Finfgeld [ 18 ] have all adapted grounded theory to formulate a method of synthesis. Key methods and assumptions of grounded theory, as originally formulated and subsequently refined by Glaser and Strauss [ 19 ] and Strauss and Corbin [ 20 , 21 ], include: simultaneous phases of data collection and analysis; an inductive approach to analysis, allowing the theory to emerge from the data; the use of the constant comparison method; the use of theoretical sampling to reach theoretical saturation; and the generation of new theory. Eaves cited grounded theorists Charmaz [ 22 ] and Chesler [ 23 ], as well as Strauss and Corbin [ 20 ], as informing her approach to synthesis.

Glaser and Strauss [ 19 ] foresaw a time when a substantive body of grounded research should be pushed towards a higher, more abstract level. As a piece of methodological work, Eaves undertook her own synthesis of the synthesis methods used by these authors to produce her own clear and explicit guide to synthesis in grounded formal theory. Kearney stated that 'grounded formal theory', as she termed this method of synthesis, 'is suited to study of phenomena involving processes of contextualized understanding and action' [[ 24 ], p180] and, as such, is particularly applicable to nurses' research interests.

As Kearney suggested, the examples examined here were largely dominated by research in nursing. Eaves synthesised studies on care-giving in rural African-American families for elderly stroke survivors; Finfgeld on courage among individuals with long-term health problems; Kearney on women's experiences of domestic violence.

Kearney explicitly chose 'grounded formal theory' because it matches 'like' with 'like': that is, it applies the same methods that have been used to generate the original grounded theories included in the synthesis – produced by constant comparison and theoretical sampling – to generate a higher-level grounded theory. The wish to match 'like' with 'like' is also implicit in Eaves' paper. This distinguishes grounded formal theory from more recent applications of meta-ethnography, which have sought to include qualitative research using diverse methodological approaches [ 12 ].

• Thematic Synthesis

Thomas and Harden [ 25 ] have developed an approach to synthesis which they term 'thematic synthesis'. This combines and adapts approaches from both meta-ethnography and grounded theory. The method was developed out of a need to conduct reviews that addressed questions relating to intervention need, appropriateness and acceptability – as well as those relating to effectiveness – without compromising on key principles developed in systematic reviews. They applied thematic synthesis in a review of the barriers to, and facilitators of, healthy eating amongst children.

Free codes of findings are organised into 'descriptive' themes, which are then further interpreted to yield 'analytical' themes. This approach shares characteristics with later adaptations of meta-ethnography, in that the analytical themes are comparable to 'third order interpretations' and that the development of descriptive and analytical themes using coding invoke reciprocal 'translation'. It also shares much with grounded theory, in that the approach is inductive and themes are developed using a 'constant comparison' method. A novel aspect of their approach is the use of computer software to code the results of included studies line-by-line, thus borrowing another technique from methods usually used to analyse primary research.

Textual Narrative Synthesis

Textual narrative synthesis is an approach which arranges studies into more homogenous groups. Lucas et al [ 26 ] comment that it has proved useful in synthesising evidence of different types (qualitative, quantitative, economic etc). Typically, study characteristics, context, quality and findings are reported on according to a standard format and similarities and differences are compared across studies. Structured summaries may also be developed, elaborating on and putting into context the extracted data [ 27 ].

Lucas et al [ 26 ] compared thematic synthesis with textual narrative synthesis. They found that 'thematic synthesis holds most potential for hypothesis generation' whereas textual narrative synthesis is more likely to make transparent heterogeneity between studies (as does meta-ethnography, with refutational synthesis) and issues of quality appraisal. This is possibly because textual narrative synthesis makes clearer the context and characteristics of each study, while the thematic approach organises data according to themes. However, Lucas et al found that textual narrative synthesis is 'less good at identifying commonality' (p2); the authors do not make explicit why this should be, although it may be that organising according to themes, as the thematic approach does, is comparatively more successful in revealing commonality.

Paterson et al [ 28 ] have evolved a multi-faceted approach to synthesis, which they call 'meta-study'. The sociologist Zhao [ 29 ], drawing on Ritzer's work [ 30 ], outlined three components of analysis, which they proposed should be undertaken prior to synthesis. These are meta-data-analysis (the analysis of findings), meta-method (the analysis of methods) and meta-theory (the analysis of theory). Collectively, these three elements of analysis, culminating in synthesis, make up the practice of 'meta-study'. Paterson et al pointed out that the different components of analysis may be conducted concurrently.

Paterson et al argued that primary research is a construction; secondary research is therefore a construction of a construction. There is need for an approach that recognises this, and that also recognises research to be a product of its social, historical and ideological context. Such an approach would be useful in accounting for differences in research findings. For Paterson et al, there is no such thing as 'absolute truth'.

Meta-study was developed to study the experiences of adults living with a chronic illness. Meta-data-analysis was conceived of by Paterson et al in similar terms to Noblit and Hare's meta-ethnography (see above), in that it is essentially interpretive and seeks to reveal similarities and discrepancies among accounts of a particular phenomenon. Meta-method involves the examination of the methodologies of the individual studies under review. Part of the process of meta-method is to consider different aspects of methodology such as sampling, data collection, research design etc, similar to procedures others have called 'critical appraisal' (CASP [ 31 ]). However, Paterson et al take their critique to a deeper level by establishing the underlying assumptions of the methodologies used and the relationship between research outcomes and methods used. Meta-theory involves scrutiny of the philosophical and theoretical assumptions of the included research papers; this includes looking at the wider context in which new theory is generated. Paterson et al described meta-synthesis as a process which creates a new interpretation which accounts for the results of all three elements of analysis. The process of synthesis is iterative and reflexive and the authors were unwilling to oversimplify the process by 'codifying' procedures for bringing all three components of analysis together.

Meta-narrative

Greenhalgh et al [ 32 ]'s meta-narrative approach to synthesis arose out of the need to synthesise evidence to inform complex policy-making questions and was assisted by the formation of a multi-disciplinary team. Their approach to review was informed by Thomas Kuhn's The Structure of Scientific Revolutions [ 33 ], in which he proposed that knowledge is produced within particular paradigms which have their own assumptions about theory, about what is a legitimate object of study, about what are legitimate research questions and about what constitutes a finding. Paradigms also tend to develop through time according to a particular set of stages, central to which is the stage of 'normal science', in which the particular standards of the paradigm are largely unchallenged and seen to be self-evident. As Greenhalgh et al pointed out, Kuhn saw paradigms as largely incommensurable: 'that is, an empirical discovery made using one set of concepts, theories, methods and instruments cannot be satisfactorily explained through a different paradigmatic lens' [[ 32 ], p419].

Greenhalgh et al synthesised research from a wide range of disciplines; their research question related to the diffusion of innovations in health service delivery and organisation. They thus identified a need to synthesise findings from research which contains many different theories arising from many different disciplines and study designs.

Based on Kuhn's work, Greenhalgh et al proposed that, across different paradigms, there were multiple – and potentially mutually contradictory – ways of understanding the concept at the heart of their review, namely the diffusion of innovation. Bearing this in mind, the reviewers deliberately chose to select key papers from a number of different research 'paradigms' or 'traditions', both within and beyond healthcare, guided by their multidisciplinary research team. They took as their unit of analysis the 'unfolding "storyline" of a research tradition over time' [[ 32 ], p417) and sought to understand diffusion of innovation as it was conceptualised in each of these traditions. Key features of each tradition were mapped: historical roots, scope, theoretical basis; research questions asked and methods/instruments used; main empirical findings; historical development of the body of knowledge (how have earlier findings led to later findings); and strengths and limitations of the tradition. The results of this exercise led to maps of 13 'meta-narratives' in total, from which seven key dimensions, or themes, were identified and distilled for the synthesis phase of the review.

Critical Interpretive Synthesis

Dixon-Woods et al [ 34 ] developed their own approach to synthesising multi-disciplinary and multi-method evidence, termed 'critical interpretive synthesis', while researching access to healthcare by vulnerable groups. Critical interpretive synthesis is an adaptation of meta-ethnography, as well as borrowing techniques from grounded theory. The authors stated that they needed to adapt traditional meta-ethnographic methods for synthesis, since these had never been applied to quantitative as well as qualitative data, nor had they been applied to a substantial body of data (in this case, 119 papers).

Dixon-Woods et al presented critical interpretive synthesis as an approach to the whole process of review, rather than to just the synthesis component. It involves an iterative approach to refining the research question and searching and selecting from the literature (using theoretical sampling) and defining and applying codes and categories. It also has a particular approach to appraising quality, using relevance – i.e. likely contribution to theory development – rather than methodological characteristics as a means of determining the 'quality' of individual papers [ 35 ]. The authors also stress, as a defining characteristic, critical interpretive synthesis's critical approach to the literature in terms of deconstructing research traditions or theoretical assumptions as a means of contextualising findings.

Dixon-Woods et al rejected reciprocal translational analysis (RTA) as this produced 'only a summary in terms that have already been used in the literature' [[ 34 ], p5], which was seen as less helpful when dealing with a large and diverse body of literature. Instead, Dixon-Woods et al adopted a lines-of-argument (LOA) synthesis, in which – rejecting the difference between first, second and third order constructs – they instead developed 'synthetic constructs' which were then linked with constructs arising directly from the literature.

The influence of grounded theory can be seen in particular in critical interpretive synthesis's inductive approach to formulating the review question and to developing categories and concepts, rejecting a 'stage' approach to systematic reviewing, and in selecting papers using theoretical sampling. Dixon-Woods et al also claim that critical interpretive synthesis is distinct in its 'explicit orientation towards theory generation' [[ 34 ], p9].

Ecological Triangulation

Jim Banning is the author of 'ecological triangulation' or 'ecological sentence synthesis', applying this method to the evidence for what works for youth with disabilities. He borrows from Webb et al [ 36 ] and Denzin [ 37 ] the concept of triangulation, in which phenomena are studied from a variety of vantage points. His rationale is that building an 'evidence base' of effectiveness requires the synthesis of cumulative, multi-faceted evidence in order to find out 'what intervention works for what kind of outcomes for what kind of persons under what kind of conditions' [[ 38 ], p1].

Ecological triangulation unpicks the mutually interdependent relationships between behaviour, persons and environments. The method requires that, for data extraction and synthesis, 'ecological sentences' are formulated following the pattern: 'With this intervention, these outcomes occur with these population foci and within these grades (ages), with these genders ... and these ethnicities in these settings' [[ 39 ], p1].

Framework Synthesis

Brunton et al [ 40 ] and Oliver et al [ 41 ] have applied a 'framework synthesis' approach in their reviews. Framework synthesis is based on framework analysis, which was outlined by Pope, Ziebland and Mays [ 42 ], and draws upon the work of Ritchie and Spencer [ 43 ] and Miles and Huberman [ 44 ]. Its rationale is that qualitative research produces large amounts of textual data in the form of transcripts, observational fieldnotes etc. The sheer wealth of information poses a challenge for rigorous analysis. Framework synthesis offers a highly structured approach to organising and analysing data (e.g. indexing using numerical codes, rearranging data into charts etc).

Brunton et al applied the approach to a review of children's, young people's and parents' views of walking and cycling; Oliver et al to an analysis of public involvement in health services research. Framework synthesis is distinct from the other methods outlined here in that it utilises an a priori 'framework' – informed by background material and team discussions – to extract and synthesise findings. As such, it is largely a deductive approach although, in addition to topics identified by the framework, new topics may be developed and incorporated as they emerge from the data. The synthetic product can be expressed in the form of a chart for each key dimension identified, which may be used to map the nature and range of the concept under study and find associations between themes and exceptions to these [ 40 ].

'Fledgling' approaches

There are three other approaches to synthesis which have not yet been widely used. One is an approach using content analysis [ 45 , 46 ] in which text is condensed into fewer content-related categories. Another is 'meta-interpretation' [ 47 ], featuring the following: an ideographic rather than pre-determined approach to the development of exclusion criteria; a focus on meaning in context; interpretations as raw data for synthesis (although this feature doesn't distinguish it from other synthesis methods); an iterative approach to the theoretical sampling of studies for synthesis; and a transparent audit trail demonstrating the trustworthiness of the synthesis.

In addition to the synthesis methods discussed above, Sandelowski and Barroso propose a method they call 'qualitative metasummary' [ 15 ]. It is mentioned here as a new and original approach to handling a collection of qualitative studies but is qualitatively different to the other methods described here since it is aggregative; that is, findings are accumulated and summarised rather than 'transformed'. Metasummary is a way of producing a 'map' of the contents of qualitative studies and – according to Sandelowski and Barroso – 'reflect [s] a quantitative logic' [[ 15 ], p151]. The frequency of each finding is determined and the higher the frequency of a particular finding, the greater its validity. The authors even discuss the calculation of 'effect sizes' for qualitative findings. Qualitative metasummaries can be undertaken as an end in themselves or may serve as a basis for a further synthesis.

Dimensions of difference

Having outlined the range of methods identified, we now turn to an examination of how they compare with one another. It is clear that they have come from many different contexts and have different approaches to understanding knowledge, but what do these differences mean in practice? Our framework for this analysis is shown in Additional file 1 : dimensions of difference [ 48 ]. We have examined the epistemology of each of the methods and found that, to some extent, this explains the need for different methods and their various approaches to synthesis.

Epistemology

The first dimension that we will consider is that of the researchers' epistemological assumptions. Spencer et al [ 49 ] outline a range of epistemological positions, which might be organised into a spectrum as follows:

Subjective idealism : there is no shared reality independent of multiple alternative human constructions

Objective idealism : there is a world of collectively shared understandings

Critical realism : knowledge of reality is mediated by our perceptions and beliefs

Scientific realism : it is possible for knowledge to approximate closely an external reality

Naïve realism : reality exists independently of human constructions and can be known directly [ 49 , 45 , 46 ].

Thus, at one end of the spectrum we have a highly constructivist view of knowledge and, at the other, an unproblematized 'direct window onto the world' view.

Nearly all of positions along this spectrum are represented in the range of methodological approaches to synthesis covered in this paper. The originators of meta-narrative synthesis, critical interpretive synthesis and meta-study all articulate what might be termed a 'subjective idealist' approach to knowledge. Paterson et al [ 28 ] state that meta-study shies away from creating 'grand theories' within the health or social sciences and assume that no single objective reality will be found. Primary studies, they argue, are themselves constructions; meta-synthesis, then, 'deals with constructions of constructions' (p7). Greenhalgh et al [ 32 ] also view knowledge as a product of its disciplinary paradigm and use this to explain conflicting findings: again, the authors neither seek, nor expect to find, one final, non-contestable answer to their research question. Critical interpretive synthesis is similar in seeking to place literature within its context, to question its assumptions and to produce a theoretical model of a phenomenon which – because highly interpretive – may not be reproducible by different research teams at alternative points in time [[ 34 ], p11].

Methods used to synthesise grounded theory studies in order to produce a higher level of grounded theory [ 24 ] appear to be informed by 'objective idealism', as does meta-ethnography. Kearney argues for the near-universal applicability of a 'ready-to-wear' theory across contexts and populations. This approach is clearly distinct from one which recognises multiple realities. The emphasis is on examining commonalities amongst, rather than discrepancies between, accounts. This emphasis is similarly apparent in most meta-ethnographies, which are conducted either according to Noblit and Hare's 'reciprocal translational analysis' technique or to their 'lines-of-argument' technique and which seek to provide a 'whole' which has a greater explanatory power. Although Noblit and Hare also propose 'refutational synthesis', in which contradictory findings might be explored, there are few examples of this having been undertaken in practice, and the aim of the method appears to be to explain and explore differences due to context, rather than multiple realities.

Despite an assumption of a reality which is perhaps less contestable than those of meta-narrative synthesis, critical interpretive synthesis and meta-study, both grounded formal theory and meta-ethnography place a great deal of emphasis on the interpretive nature of their methods. This still supposes a degree of constructivism. Although less explicit about how their methods are informed, it seems that both thematic synthesis and framework synthesis – while also involving some interpretation of data – share an even less problematized view of reality and a greater assumption that their synthetic products are reproducible and correspond to a shared reality. This is also implicit in the fact that such products are designed directly to inform policy and practice, a characteristic shared by ecological triangulation. Notably, ecological triangulation, according to Banning, can be either realist or idealist. Banning argues that the interpretation of triangulation can either be one in which multiple viewpoints converge on a point to produce confirming evidence (i.e. one definitive answer to the research question) or an idealist one, in which the complexity of multiple viewpoints is represented. Thus, although ecological triangulation views reality as complex, the approach assumes that it can be approximately knowable (at least when the realist view of ecological triangulation is adopted) and that interventions can and should be modelled according to the products of its syntheses.

While pigeonholing different methods into specific epistemological positions is a problematic process, we do suggest that the contrasting epistemologies of different researchers is one way of explaining why we have – and need – different methods for synthesis.

Variation in terms of the extent of iteration during the review process is another key dimension. All synthesis methods include some iteration but the degree varies. Meta-ethnography, grounded theory and thematic synthesis all include iteration at the synthesis stage; both framework synthesis and critical interpretive synthesis involve iterative literature searching – in the case of critical interpretive synthesis, it is not clear whether iteration occurs during the rest of the review process. Meta-narrative also involves iteration at every stage. Banning does not mention iteration in outlining ecological triangulation and neither do Lucas or Thomas and Harden for thematic narrative synthesis.

It seems that the more idealist the approach, the greater the extent of iteration. This might be because a large degree of iteration does not sit well with a more 'positivist' ideal of procedural objectivity; in particular, the notion that the robustness of the synthetic product depends in part on the reviewers stating up front in a protocol their searching strategies, inclusion/exclusion criteria etc, and being seen not to alter these at a later stage.

Quality assessment

Another dimension along which we can look at different synthesis methods is that of quality assessment. When the approaches to the assessment of the quality of studies retrieved for review are examined, there is again a wide methodological variation. It might be expected that the further towards the 'realism' end of the epistemological spectrum a method of synthesis falls, the greater the emphasis on quality assessment. In fact, this is only partially the case.

Framework synthesis, thematic narrative synthesis and thematic synthesis – methods which might be classified as sharing a 'critical realist' approach – all have highly specified approaches to quality assessment. The review in which framework synthesis was developed applied ten quality criteria: two on quality and reporting of sampling methods, four to the quality of the description of the sample in the study, two to the reliability and validity of the tools used to collect data and one on whether studies used appropriate methods for helping people to express their views. Studies which did not meet a certain number of quality criteria were excluded from contributing to findings. Similarly, in the example review for thematic synthesis, 12 criteria were applied: five related to reporting aims, context, rationale, methods and findings; four relating to reliability and validity; and three relating to the appropriateness of methods for ensuring that findings were rooted in participants' own perspectives. Studies which were deemed to have significant flaws were excluded and sensitivity analyses were used to assess the possible impact of study quality on the review's findings. Thomas and Harden's use of thematic narrative synthesis similarly applied quality criteria and developed criteria additional to those they found in the literature on quality assessment, relating to the extent to which people's views and perspectives had been privileged by researchers. It is worth noting not only that these methods apply quality criteria but that they are explicit about what they are: assessing quality is a key component in the review process for both of these methods. Likewise, Banning – the originator of ecological triangulation – sees quality assessment as important and adapts the Design and Implementation Assessment Device (DIAD) Version 0.3 (a quality assessment tool for quantitative research) for use when appraising qualitative studies [ 50 ]. Again, Banning writes of excluding studies deemed to be of poor quality.

Greenhalgh et al's meta-narrative review [ 32 ] modified a range of existing quality assessment tools to evaluate studies according to validity and robustness of methods; sample size and power; and validity of conclusions. The authors imply, but are not explicit, that this process formed the basis for the exclusion of some studies. Although not quite so clear about quality assessment methods as framework and thematic synthesis, it might be argued that meta-narrative synthesis shows a greater commitment to the concept that research can and should be assessed for quality than either meta-ethnography or grounded formal theory. The originators of meta-ethnography, Noblit and Hare [ 8 ], originally discussed quality in terms of quality of metaphor, while more recent use of this method has used amended versions of CASP (the Critical Appraisal Skills Programme tool, [ 31 ]), yet has only referred to studies being excluded on the basis of lack of relevance or because they weren't 'qualitative' studies [ 8 ]. In grounded theory, quality assessment is only discussed in terms of a 'personal note' being made on the context, quality and usefulness of each study. However, contrary to expectation, meta-narrative synthesis lies at the extreme end of the idealism/realism spectrum – as a subjective idealist approach – while meta-ethnography and grounded theory are classified as objective idealist approaches.

Finally, meta-study and critical interpretive synthesis – two more subjective idealist approaches – look to the content and utility of findings rather than methodology in order to establish quality. While earlier forms of meta-study included only studies which demonstrated 'epistemological soundness', in its most recent form [ 51 ] this method has sought to include all relevant studies, excluding only those deemed not to be 'qualitative' research. Critical interpretive synthesis also conforms to what we might expect of its approach to quality assessment: quality of research is judged as the extent to which it informs theory. The threshold of inclusion is informed by expertise and instinct rather than being articulated a priori.

In terms of quality assessment, it might be important to consider the academic context in which these various methods of synthesis developed. The reason why thematic synthesis, framework synthesis and ecological triangulation have such highly specified approaches to quality assessment may be that each of these was developed for a particular task, i.e. to conduct a multi-method review in which randomised controlled trials (RCTs) were included. The concept of quality assessment in relation to RCTs is much less contested and there is general agreement on criteria against which quality should be judged.

Problematizing the literature

Critical interpretive synthesis, the meta-narrative approach and the meta-theory element of meta-study all share some common ground in that their review and synthesis processes include examining all aspects of the context in which knowledge is produced. In conducting a review on access to healthcare by vulnerable groups, critical interpretive synthesis sought to question 'the ways in which the literature had constructed the problematics of access, the nature of the assumptions on which it drew, and what has influenced its choice of proposed solutions' [[ 34 ], p6]. Although not claiming to have been directly influenced by Greenhalgh et al's meta-narrative approach, Dixon-Woods et al do cite it as sharing similar characteristics in the sense that it critiques the literature it reviews.

Meta-study uses meta-theory to describe and deconstruct the theories that shape a body of research and to assess its quality. One aspect of this process is to examine the historical evolution of each theory and to put it in its socio-political context, which invites direct comparison with meta-narrative synthesis. Greenhalgh et al put a similar emphasis on placing research findings within their social and historical context, often as a means of seeking to explain heterogeneity of findings. In addition, meta-narrative shares with critical interpretive synthesis an iterative approach to searching and selecting from the literature.

Framework synthesis, thematic synthesis, textual narrative synthesis, meta-ethnography and grounded theory do not share the same approach to problematizing the literature as critical interpretive synthesis, meta-study and meta-narrative. In part, this may be explained by the extent to which studies included in the synthesis represented a broad range of approaches or methodologies. This, in turn, may reflect the broadness of the review question and the extent to which the concepts contained within the question are pre-defined within the literature. In the case of both the critical interpretive synthesis and meta-narrative reviews, terminology was elastic and/or the question formed iteratively. Similarly, both reviews placed great emphasis on employing multi-disciplinary research teams. Approaches which do not critique the literature in the same way tend to have more narrowly-focused questions. They also tend to include a more limited range of studies: grounded theory synthesis includes grounded theory studies, meta-ethnography (in its original form, as applied by Noblit and Hare) ethnographies. The thematic synthesis incorporated studies based on only a narrow range of qualitative methodologies (interviews and focus groups) which were informed by a similarly narrow range of epistemological assumptions. It may be that the authors of such syntheses saw no need for including such a critique in their review process.

Similarities and differences between primary studies

Most methods of synthesis are applicable to heterogeneous data (i.e. studies which use contrasting methodologies) apart from early meta-ethnography and synthesis informed by grounded theory. All methods of synthesis state that, at some level, studies are compared; many are not so explicit about how this is done, though some are. Meta-ethnography is one of the most explicit: it describes the act of 'translation' where terms and concepts which have resonance with one another are subsumed into 'higher order constructs'. Grounded theory, as represented by Eaves [ 17 ], is undertaken according to a long list of steps and sub-steps, includes the production of generalizations about concepts/categories, which comes from classifying these categories. In meta-narrative synthesis, comparable studies are grouped together at the appraisal phase of review.

Perhaps more interesting are the ways in which differences between studies are explored. Those methods with a greater emphasis on critical appraisal may tend (although this is not always made explicit) to use differences in method to explain differences in finding. Meta-ethnography proposes 'refutational synthesis' to explain differences, although there are few examples of this in the literature. Some synthesis methods – for example, thematic synthesis – look at other characteristics of the studies under review, whether types of participants and their context vary, and whether this can explain differences in perspective.

All of these methods, then, look within the studies to explain differences. Other methods look beyond the study itself to the context in which it was produced. Critical interpretive synthesis and meta-study look at differences in theory or in socio-economic context. Critical interpretive synthesis, like meta-narrative, also explores epistemological orientation. Meta-narrative is unique in concerning itself with disciplinary paradigm (i.e. the story of the discipline as it progresses). It is also distinctive in that it treats conflicting findings as 'higher order data' [[ 32 ], p420], so that the main emphasis of the synthesis appears to be on examining and explaining contradictions in the literature.

Going 'beyond' the primary studies

Synthesis is sometimes defined as a process resulting in a product, a 'whole', which is more than the sum of its parts. However, the methods reviewed here vary in the extent to which they attempt to 'go beyond' the primary studies and transform the data. Some methods – textual narrative synthesis, ecological triangulation and framework synthesis – focus on describing and summarising their primary data (often in a highly structured and detailed way) and translating the studies into one another. Others – meta-ethnography, grounded theory, thematic synthesis, meta-study, meta-narrative and critical interpretive synthesis – seek to push beyond the original data to a fresh interpretation of the phenomena under review. A key feature of thematic synthesis is its clear differentiation between these two stages.

Different methods have different mechanisms for going beyond the primary studies, although some are more explicit than others about what these entail. Meta-ethnography proposes a 'Line of Argument' (LOA) synthesis in which an interpretation is constructed to both link and explain a set of parts. Critical interpretive synthesis based its synthesis methods on those of meta-ethnography, developing an LOA using what the authors term 'synthetic constructs' (akin to 'third order constructs' in meta-ethnography) to create a 'synthesising argument'. Dixon-Woods et al claim that this is an advance on Britten et al's methods, in that they reject the difference between first, second and third order constructs.

Meta-narrative, as outlined above, focuses on conflicting findings and constructs theories to explain these in terms of differing paradigms. Meta study derives questions from each of its three components to which it subjects the dataset and inductively generates a number of theoretical claims in relation to it. According to Eaves' model of grounded theory [ 17 ], mini-theories are integrated to produce an explanatory framework. In ecological triangulation, the 'axial' codes – or second level codes evolved from the initial deductive open codes – are used to produce Banning's 'ecological sentence' [ 39 ].

The synthetic product

In overviewing and comparing different qualitative synthesis methods, the ultimate question relates to the utility of the synthetic product: what is it for? It is clear that some methods of synthesis – namely, thematic synthesis, textual narrative synthesis, framework synthesis and ecological triangulation – view themselves as producing an output that is directly applicable to policy makers and designers of interventions. The example of framework synthesis examined here (on children's, young people's and parents' views of walking and cycling) involved policy makers and practitioners in directing the focus of the synthesis and used the themes derived from the synthesis to infer what kind of interventions might be most effective in encouraging walking and cycling. Likewise, the products of the thematic synthesis took the form of practical recommendations for interventions (e.g. 'do not promote fruit and vegetables in the same way in the same intervention'). The extent to which policy makers and practitioners are involved in informing either synthesis or recommendation is less clear from the documents published on ecological triangulation, but the aim certainly is to directly inform practice.

The outputs of synthesis methods which have a more constructivist orientation – meta-study, meta-narrative, meta-ethnography, grounded theory, critical interpretive synthesis – tend to look rather different. They are generally more complex and conceptual, sometimes operating on the symbolic or metaphorical level, and requiring a further process of interpretation by policy makers and practitioners in order for them to inform practice. This is not to say, however, that they are not useful for practice, more that they are doing different work. However, it may be that, in the absence of further interpretation, they are more useful for informing other researchers and theoreticians.

Looking across dimensions

After examining the dimensions of difference of our included methods, what picture ultimately emerges? It seems clear that, while similar in some respects, there are genuine differences in approach to the synthesis of what is essentially textual data. To some extent, these differences can be explained by the epistemological assumptions that underpin each method. Our methods split into two broad camps: the idealist and the realist (see Table 1 for a summary). Idealist approaches generally tend to have a more iterative approach to searching (and the review process), have less a priori quality assessment procedures and are more inclined to problematize the literature. Realist approaches are characterised by a more linear approach to searching and review, have clearer and more well-developed approaches to quality assessment, and do not problematize the literature.

Mapping the relationships between methods

What is interesting is the relationship between these methods of synthesis, the conceptual links between them, and the extent to which the originators cite – or, in some cases, don't cite – one another. Some methods directly build on others – framework synthesis builds on framework analysis, for example, while grounded theory and constant comparative analysis build on grounded theory. Others further develop existing methods – meta-study, critical interpretive synthesis and meta-narrative all adapt aspects of meta-ethnography, while also importing concepts from other theorists (critical interpretive synthesis also adapts grounded theory techniques).

Some methods share a clear conceptual link, without directly citing one another: for example, the analytical themes developed during thematic synthesis are comparable to the third order interpretations of meta-ethnography. The meta-theory aspect of meta-study is echoed in both meta-narrative synthesis and critical interpretive synthesis (see 'Problematizing the literature, above); however, the originators of critical interpretive synthesis only refer to the originators of meta-study in relation to their use of sampling techniques.

While methods for qualitative synthesis have many similarities, there are clear differences in approach between them, many of which can be explained by taking account of a given method's epistemology.

However, within the two broad idealist/realist categories, any differences between methods in terms of outputs appear to be small.

Since many systematic reviews are designed to inform policy and practice, it is important to select a method – or type of method – that will produce the kind of conclusions needed. However, it is acknowledged that this is not always simple or even possible to achieve in practice.

The approaches that result in more easily translatable messages for policy-makers and practitioners may appear to be more attractive than the others; but we do need to take account lessons from the more idealist end of the spectrum, that some perspectives are not universal.

Dixon-Woods M, Agarwhal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Pol. 2005, 10 (1): 45-53b. 10.1258/1355819052801804.

Article   Google Scholar  

Barbour RS, Barbour M: Evaluating and synthesizing qualitative research: the need to develop a distinctive approach. J Eval Clin Pract. 2003, 9 (2): 179-186. 10.1046/j.1365-2753.2003.00371.x.

Article   PubMed   Google Scholar  

Mays N, Pope C, Popay J: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Pol. 2005, 10 (Suppl 1): 6-20. 10.1258/1355819054308576.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Shaw RL, Smith J, Sutton A, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006, 6: 27-44. 10.1177/1468794106058867.

Pope C, Mays N, Popay J: Synthesizing Qualitative and Quantitative Health Evidence: a Guide to Methods. 2007, Maidenhead: Open University Press

Google Scholar  

Thorne S, Jenson L, Kearney MH, Noblit G, Sandelowski M: Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004, 14: 1342-1365. 10.1177/1049732304269888.

Centre for Reviews and Dissemination: Systematic Reviews. CRD's Guidance for Undertaking Reviews in Health Care. 2008, York: CRD

Noblit GW, Hare RD: Meta-Ethnography: Synthesizing Qualitative Studies. 1988, London: Sage

Book   Google Scholar  

Strike K, Posner G: Types of synthesis and their criteria. Knowledge Structure and Use. Edited by: Ward S, Reed L. 1983, Philadelphia: Temple University Press

Turner S: Sociological Explanation as Translation. 1980, New York: Cambridge University Press

Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R: Using meta-ethnography to synthesis qualitative research: a worked example. J Health Serv Res. 2002, 7: 209-15. 10.1258/135581902320432732.

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J: Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003, 65: 671-84. 10.1016/S0277-9536(02)00064-3.

Pound P, Britten N, Morgan M, Yardley L, Pope C, Daker-White G, Campbell R: Resisting medicines: a synthesis of qualitative studies of medicine taking. Soc Sci Med. 2005, 61: 133-155. 10.1016/j.socscimed.2004.11.063.

Schutz A: Collected Paper. 1962, The Hague: Martinus Nijhoff, 1:

Sandelowski M, Barroso J: Handbook for Synthesizing Qualitative Research. 2007, New York: Springer Publishing Company

Kearney MH: Enduring love: a grounded formal theory of women's experience of domestic violence. Research Nurs Health. 2001, 24: 270-82. 10.1002/nur.1029.

Article   CAS   Google Scholar  

Eaves YD: A synthesis technique for grounded theory data analysis. J Adv Nurs. 2001, 35: 654-63. 10.1046/j.1365-2648.2001.01897.x.

Article   CAS   PubMed   Google Scholar  

Finfgeld D: Courage as a process of pushing beyond the struggle. Qual Health Res. 1999, 9: 803-814. 10.1177/104973299129122298.

Glaser BG, Strauss AL: The Discovery of Grounded Theory: Strategies for Qualitative Research. 1967, New York: Aldine De Gruyter

Strauss AL, Corbin J: Basics of Qualitative Research: Grounded Theory Procedures and Techniques. 1990, Newbury Park, CA: Sage

Strauss AL, Corbin J: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 1998, Thousand Oaks, CA: Sage

Charmaz K: The grounded theory method: an explication and interpretation. Contemporary Field Research: A Collection of Readings. Edited by: Emerson RM. 1983, Waveland Press: Prospect Heights, IL, 109-126.

Chesler MA: Professionals' Views of the Dangers of Self-Help Groups: Explicating a Grounded Theoretical Approach. 1987, [Michigan]: Department of Sociology, University of Michigan, Ann Arbour Centre for Research on Social Organisation, Working Paper Series

Kearney MH: Ready-to-wear: discovering grounded formal theory. Res Nurs Health. 1988, 21: 179-186. 10.1002/(SICI)1098-240X(199804)21:2<179::AID-NUR8>3.0.CO;2-G.

Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Meth. 2008, 8: 45-10.1186/1471-2288-8-45.

Lucas PJ, Arai L, Baird , Law C, Roberts HM: Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Meth. 2007, 7 (4):

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health research. J Epidemiol Community H. 2004, 58: 794-800. 10.1136/jech.2003.014829.

Paterson BL, Thorne SE, Canam C, Jillings C: Meta-Study of Qualitative Health Research. A Practical Guide to Meta-Analysis and Meta-Synthesis. 2001, Thousand Oaks, CA: Sage Publications

Zhao S: Metatheory, metamethod, meta-data-analysis: what, why and how?. Sociol Perspect. 1991, 34: 377-390.

Ritzer G: Metatheorizing in Sociology. 1991, Lexington, MA: Lexington Books

CASP (Critical Appraisal Skills Programme). date unknown, [ http://www.phru.nhs.uk/Pages/PHD/CASP.htm ]

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005, 61: 417-30. 10.1016/j.socscimed.2004.12.001.

Kuhn TS: The Structure of Scientific Revolutions. 1962, Chicago: University of Chicago Press

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Meth. 2006, 6 (35):

Gough D: Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Applied and Practice-based Research. Edited by: Furlong J, Oancea A. 2007, Special Edition of Research Papers in Education, 22 (2): 213-228.

Webb EJ, Campbell DT, Schwartz RD, Sechrest L: Unobtrusive Measures. 1966, Chicago: Rand McNally

Denzin NK: The Research Act: a Theoretical Introduction to Sociological Methods. 1978, New York: McGraw-Hill

Banning J: Ecological Triangulation. [ http://mycahs.colostate.edu/James.H.Banning/PDFs/Ecological%20Triangualtion.pdf ]

Banning J: Ecological Sentence Synthesis. [ http://mycahs.colostate.edu/James.H.Banning/PDFs/Ecological%20Sentence%20Synthesis.pdf ]

Brunton G, Oliver S, Oliver K, Lorenc T: A Synthesis of Research Addressing Children's, Young People's and Parents' Views of Walking and Cycling for Transport. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

Oliver S, Rees R, Clarke-Jones L, Milne R, Oakley A, Gabbay J, Stein K, Buchanan P, Gyte G: A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008, 11: 72-84. 10.1111/j.1369-7625.2007.00476.x.

Pope C, Ziebland S, Mays N: Qualitative research in health care: analysing qualitative data. BMJ. 2000, 320: 114-116. 10.1136/bmj.320.7227.114.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ritchie J, Spencer L: Qualitative data analysis for applied policy research. Analysing Qualitative Data. Edited by: Bryman A, Burgess R. 1993, London: Routledge, 173-194.

Miles M, Huberman A: Qualitative Data Analysis. 1984, London: Sage

Evans D, Fitzgerald M: Reasons for physically restraining patients and residents: a systematic review and content analysis. Int J Nurs Stud. 2002, 39: 739-743. 10.1016/S0020-7489(02)00015-9.

Suikkala A, Leino-Kilpi H: Nursing student-patient relationships: a review of the literature from 1984–1998. J Adv Nurs. 2000, 33: 42-50. 10.1046/j.1365-2648.2001.01636.x.

Weed M: 'Meta-interpretation': a method for the interpretive synthesis of qualitative research. Forum: Qual Soc Res. 2005, 6: Art 37-

Gough D, Thomas J: Dimensions of difference in systematic reviews. [ http://www.ncrm.ac.uk/RMF2008/festival/programme/sys1 ]

Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. 2003, London: Government Chief Social Researcher's Office

Banning J: Design and Implementation Assessment Device (DIAD) Version 0.3: A response from a qualitative perspective. [ http://mycahs.colostate.edu/James.H.Banning/PDFs/Design%20and%20Implementation%20Assessment%20Device.pdf ]

Paterson BL: Coming out as ill: understanding self-disclosure in chronic illness from a meta-synthesis of qualitative research. Reviewing Research Evidence for Nursing Practice. Edited by: Webb C, Roe B. 2007, [Oxford]: Blackwell Publishing Ltd, 73-83.

Chapter   Google Scholar  

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/9/59/prepub

Download references

Acknowledgements

The authors would like to acknowledge the helpful contributions of the following in commenting on earlier drafts of this paper: David Gough, Sandy Oliver, Angela Harden, Mary Dixon-Woods, Trisha Greenhalgh and Barbara L. Paterson. We would also like to thank the peer reviewers: Helen J Smith, Rosaline Barbour and Mark Rodgers for their helpful reviews. The methodological development was supported by the Department of Health (England) and the ESRC through the Methods for Research Synthesis Node of the National Centre for Research Methods (NCRM). An earlier draft of this paper currently appears as a working paper on the National Centre for Research Methods' website http://www.ncrm.ac.uk/ .

Author information

Authors and affiliations.

Social Science Research Unit, Evidence for Policy and Practice Information and Co-ordinating (EPPI-) Centre, 18 Woburn Square, London, WC1H 0NS, UK

Elaine Barnett-Page & James Thomas

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Elaine Barnett-Page .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

Both authors made substantial contributions, with EBP taking a lead on writing and JT on the analytical framework. Both authors read and approved the final manuscript.

Electronic supplementary material

12874_2009_375_moesm1_esm.doc.

Additional file 1: Dimensions of difference. Ranging from subjective idealism through objective idealism and critical realism to scientific realism to naïve realism (DOC 46 KB)

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Barnett-Page, E., Thomas, J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol 9 , 59 (2009). https://doi.org/10.1186/1471-2288-9-59

Download citation

Received : 09 March 2009

Accepted : 11 August 2009

Published : 11 August 2009

DOI : https://doi.org/10.1186/1471-2288-9-59

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Narrative Synthesis
• Theoretical Sampling
• Qualitative Synthesis
• Order Construct

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: May 9, 2024 11:05 AM
• URL: https://libguides.usc.edu/writingguide

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

• Research article
• Open access
• Published: 08 January 2021

Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis

• Raabia Sattar   ORCID: orcid.org/0000-0002-1003-9772 1 , 2 ,
• Rebecca Lawton 1 , 2 ,
• Maria Panagioti 3 &
• Judith Johnson 1 , 2  

BMC Health Services Research volume  21 , Article number:  50 ( 2021 ) Cite this article

36k Accesses

104 Citations

41 Altmetric

Metrics details

Qualitative synthesis approaches are increasingly used in healthcare research. One of the most commonly utilised approaches is meta-ethnography. This is a systematic approach which synthesises data from multiple studies to enable new insights into patients’ and healthcare professionals’ experiences and perspectives. Meta-ethnographies can provide important theoretical and conceptual contributions and generate evidence for healthcare practice and policy. However, there is currently a lack of clarity and guidance surrounding the data synthesis stages and process.

This paper aimed to outline a step-by-step method for conducting a meta-ethnography with illustrative examples.

A practical step-by-step guide for conducting meta-ethnography based on the original seven steps as developed by Noblit & Hare (Meta-ethnography: Synthesizing qualitative studies.,1998) is presented. The stages include getting started, deciding what is relevant to the initial interest, reading the studies, determining how the studies are related, translating the studies into one another, synthesising the translations and expressing the synthesis.

We have incorporated adaptations and developments from recent publications. Annotations based on a previous meta-ethnography are provided. These are particularly detailed for stages 4–6, as these are often described as being the most challenging to conduct, but with the most limited amount of guidance available.

Meta-ethnographic synthesis is an important and increasingly used tool in healthcare research, which can be used to inform policy and practice. The guide presented clarifies how the stages and processes involved in conducting a meta-synthesis can be operationalised.

Peer Review reports

The range of different methods for synthesising qualitative research has grown in recent years [ 1 ]. There are now a number of different qualitative synthesis methods including qualitative meta-synthesis, narrative synthesis, thematic synthesis, interpretative synthesis, grounded theory and meta-ethnography. A qualitative synthesis is defined as ‘any methodology whereby study findings are systematically interpreted through a series of expert judgements to represent the meaning of the collected work’ [ 2 ] .. In a qualitative synthesis the findings of qualitative studies are pooled [ 2 ] . The use of some types of qualitative syntheses allow for the inclusion of mixed-methods and quantitative research studies alongside qualitative studies. A qualitative synthesis approach can be used to examine the available literature, and review and integrate the primary research studies related to a specific question or phenomenon, to reveal deeper insights or explanations that would not be possible from a single study [ 3 ]. The various qualitative synthesis approaches mentioned above differ in their purposes, philosophical traditions and whether they primarily aggregate or re-interpret the study findings [ 4 , 5 ]. Meta-ethnography is an inductive, interpretative approach upon which most interpretative qualitative synthesis methods are based [ 6 ] and is the most commonly utilised qualitative synthesis approach in healthcare research [ 7 ].

Meta-ethnography is particularly suited to developing conceptual models and theories [ 8 ]. This method of qualitative synthesis is often chosen over alternative approaches as it is more suitable for the development of analytical rather than descriptive findings [ 9 ]. A meta-ethnography differs from other qualitative synthesis approaches as the reviewer re-interprets the conceptual data (themes, concepts or metaphors) created by the authors of the primary study whilst taking into account the primary data (participant quotes) using a unique translation synthesis method in order to transcend the findings of individual study accounts and create higher order themes [ 10 , 11 ]. Meta-ethnographic reviews offer greater description of methods and higher order interpretation compared to conventional narrative literature reviews [ 12 ]. In health sciences, meta-ethnographies can be used to generate evidence for healthcare and policy [ 13 ]. A meta-ethnographic synthesis approach is suitable when researchers are interested in conceptual or theoretical understandings of a particular phenomenon. Unlike some qualitative synthesis approaches which allow the inclusion of mixed-methods design studies (such as thematic synthesis and interpretative synthesis), a meta-ethnographic approach enables only the inclusion of qualitative studies. A meta-ethnography can include multiple study designs, whereas other approaches such as grounded theory require only the inclusion of similar study approaches [ 14 ].

Although meta-ethnography is a widely used qualitative literature synthesis method within healthcare research, it is poorly demarcated and there is a lack of clarity surrounding the description of the data analysis process. A number of reviews have used this approach [ 15 , 16 , 17 , 18 , 19 , 20 ] but do not provide a fully rigorous description of the stages involved in the analysis process. Given the value of qualitative meta-synthesis in integrating the findings from multiple studies into a higher conceptual level, it is important to provide detailed guidance on each of the steps involved in conducting a meta-ethnography. This paper aims to fill this gap by outlining a step-by-step method for conducting meta-ethnography. We describe our interpretation of each of the seven steps outlined by Noblit & Hare [ 10 ] who first proposed this approach. We also incorporate adaptations and developments by recent researchers [ 21 ] and provide annotations where applicable to assist in describing the stages involved.

The worked example we are using is a published meta-ethnography (reference and author names omitted for author anonymity during peer review). Where applicable, illustrative examples from this review are provided alongside the each phase to demonstrate the process.

Within a meta-ethnographic synthesis, the process of translation is key and unique. It is defined as ‘comparing the metaphors and concepts in one account with the metaphors and concepts in others’ [ 10 ]. A meta-ethnography should involve a reciprocal and refutational translation, where possible combined with a line of argument synthesis [ 22 , 23 ]. Reciprocal translation occurs when concepts in one study can incorporate those of another [ 22 ], whereas a refutational translation explains and explores differences, exceptions, incongruities and inconsistencies [ 1 , 22 , 24 ]. Reviewers often overlook refutational translation [ 24 , 25 ]; however studies may refute each other [ 26 , 27 ] or concepts within studies may refute one another [ 27 , 28 ]. Therefore it may be possible to conduct both a reciprocal and refutational translation [ 22 ]. A line of argument synthesis is not an alternative to conducting a translation but is viewed as the next stage of analysis [ 23 ]. A line of argument synthesis is the translation of accounts that interpret different aspects of the same phenomenon under study, which results in producing a whole that is greater than the sum of its individual parts [ 10 , 15 ]. Although Noblit & Hare [ 10 ] describe meta-ethnography as a seven step process, it is important to acknowledge that this process is iterative and the phases are not discrete but may overlap and run in parallel [ 10 ]. A meta-ethnography reporting tool, eMERGE has very recently been developed, and provides a framework for reviewers to follow when reporting the important aspects of a meta-ethnography [ 22 ].

In order to identify relevant literature to inform the present guide, we searched for articles which described an evaluation or discussed methodological issues in conducting a meta-ethnography or provided guidance for reporting a meta-ethnography. We then scanned the reference lists of relevant articles to identify further relevant literature. We also drew on the results from two recent systematic reviews [ 23 , 29 ]. As such, while the searches conducted for the present article were not systematic, the guide reflects recent methodological recommendations in the wider methodological literature. All relevant articles were read and recommendations were noted; where any disagreement between authors of papers was apparent, guidance which was based on systematic reviews of the evidence rather than individual reflections was prioritised.

Doing a meta-ethnographic synthesis: a step-by-step guide with illustrated examples

Phase 1: getting started.

The initial stage requires the authors to identify an area of interest [ 10 ]. The reviewers need to consider if a synthesis of the topic is required and whether a qualitative synthesis and the meta-ethnographic approach fits with the research question [ 30 ]. E.g. A meta-ethnographic synthesis approach is suitable when researchers are interested in conceptual or theoretical understandings of a particular phenomenon. It is also important to determine whether there is a large and growing body of qualitative research in this area, and whether synthesizing qualitative findings can contribute valuable knowledge to the existing literature [ 31 ]. As proposed by Campbell and colleagues [ 32 ], we emphasize, that at this stage, it is important to establish a team of researchers who have different approaches, opinions and the key skills to conduct the meta-ethnography, as this will add rigour to the meta-ethnographic review.

We were interested in the disclosure of adverse events within healthcare; specifically in the perceptions and experiences of patients and healthcare professionals relating to these events. We were aware of the large and growing body of qualitative research in this area. Our searches revealed that there was no qualitative synthesis specific to the experiences of adverse event disclosure. We believed that synthesizing the views, attitudes and experiences of both groups (patients and healthcare professionals) would enable us to understand what patients require from the disclosure conversation and what healthcare professionals currently offer. Our motivation for synthesizing the body of qualitative evidence was to inform future disclosure interventions which were acceptable to patients and practical for healthcare professionals to deliver. Synthesizing qualitative findings can make valuable knowledge accessible to healthcare professionals and policy makers [ 31 ].

Phase 2: deciding what is relevant to the initial interest

Once you have chosen your topic of interest, phase 2 involves the following steps: a) defining the focus of the synthesis, b) selecting studies for inclusion in the synthesis and locating relevant studies, c) developing inclusion and exclusion criteria and d) quality assessment of the included studies [ 12 ].

2a. Defining the focus of the synthesis

An important decision involves deciding whether to include all the studies within your chosen area of interest. It is necessary to find a balance between a review which has a broad scope, and a focus which will yield a manageable number of studies. The scope of a meta-ethnography is more restricted compared to other qualitative narrative reviews. This is due to the avoidance of making gross generalisations across disparate fields [ 10 , 26 ]. There is currently no agreement to how many studies should be included in the synthesis. Some researchers argue that synthesizing a large number of studies may interfere with the ability to produce a useful interpretative output and could result in an aggregative synthesis [ 23 ]. Synthesizing too few studies can result in underdeveloped theories/concepts [ 24 , 28 ]. A large number of studies have varied from 40 [ 32 ] to over 100 [ 24 ]. The volume of data, rather than just the number of studies is important and team size and resources will affect the ability to manage this data [ 22 ]. It is recognised that focusing on a particular aspect of your chosen topic interest and excluding certain aspects may result in some papers being overlooked. However it is important to make this choice to ensure that you have manageable number of studies [ 12 ].

Our systematic review question focused on ‘The views and experiences of patients and healthcare professionals on the disclosure of adverse events’. We focused on studies which examined the views and experiences of patients (and/or family members, members of the general public) and healthcare professionals. We found that qualitative research in the area of adverse event disclosure was limited. As this was an under-researched area, we were able to include all the available qualitative studies in this research area (enabling us to include both patients’ and healthcare professionals’ views on adverse event disclosure).

Phase 2b: locating relevant studies

The second important step involves locating potentially relevant qualitative studies by conducting a systematic search of the literature. In order to conduct a systematic search, a well-constructed and comprehensive search strategy needs to be developed. Qualitative searches can yield a large number of search results, which can be daunting and time consuming to screen. One of the ways to make your search strategy more specific is through the use of qualitative search filters. Empirically tested search filters for qualitative studies have been developed [ 33 , 34 , 35 ]. However it is possible that some of the potentially relevant studies may be missed when using such filters. Decisions regarding your search strategy and screening depend on your aims and resources available. We advise the use of a librarian for reaching decisions on the content of searches. Multiple databases are utilised to locate relevant articles and this can be further supplemented by hand searching. This is important as it can locate relevant articles which are not indexed or inaccurately indexed, and minimises the risk of missing relevant studies [ 24 ].

Some argue that a more purposive sampling approach may be more appropriate [ 36 , 37 ], which aims to provide a holistic interpretation of a phenomenon, where the extent of searching is driven by the need to reach theoretical saturation rather than to identify all eligible studies [ 24 , 38 ]. Detailed information on purposive sampling technique is available [ 24 , 28 ]. Also, to avoid the potential problem of having too few descriptively or conceptually-rich studies, knowledge-building and theory-generating systematic reviewers can conduct expansive searches of the literature [ 28 ]. We do not describe here how to conduct a systematic search of the literature, however there are a number of papers which describe this process [ 39 , 40 , 41 ].

We searched five electronic databases, and our search strategy included a combination of the three major concepts (disclosure, safety incident and experience). We also supplemented the database searches by hand searching relevant journals and reference lists. We chose not to apply qualitative filters in order to capture all the possible relevant articles.

Phase 2c: decisions to include studies

A number of factors should be considered when deciding whether to include or exclude studies from the synthesis. An important consideration is the expertise of the review authors and the time available to complete the review [ 36 ]. Reviewers should consider the likelihood of excluding valuable insights on the basis of quality, and the contribution of these studies to the development and interpretation of findings. Would excluding such studies affect the coherence of qualitative synthesized findings? [ 36 ]. Also, an important consideration is the nature of the primary data which is available to synthesise [ 23 ]. Including predominantly thin descriptive data can be problematic as it is difficult to further interpret data which lacks depth [ 23 ]. Conceptually rich data which is explanatory, or rich descriptive data which provides sufficient detail to be further developed is suitable for meta-ethnography. Therefore selecting studies based on this suitability is one of the approaches reviewers should consider. Further discussion on decisions to include studies is available [ 36 ].

Phase 2d: quality appraisal

There is a lack of agreement surrounding the use of quality appraisal for qualitative studies [ 30 ]. Some researchers argue there are difficulties with quality appraisal as some aspects of qualitative research are difficult to appraise and therefore depend on subjective judgement [ 5 ]. Although this debate continues, we argue that at least some quality appraisal of studies needs to be considered to give an indication of the credibility of the included studies. Critically appraising the studies and assigning numerical scores to indicate level of quality is also useful as it can be used as a way to order the studies for analysis. Previous published qualitative reviews have either used the highest scoring paper as the ‘index study’ [ 15 ] or have arranged all the papers in chronological order by date, and used the most recently published paper as the ‘index study’ [ 42 ]. One of the limitations is of assigning numerical scores using CASP and the use the highest scoring as an index study is that it focuses on the methodological rather than conceptual strength. Other reviewers have chosen a ‘conceptually rich’ index account [ 43 , 44 ] however it is unclear how this ‘conceptually rich’ index account should be selected. The different ways of ordering study accounts has yet to be formally empirically compared and there is no guidance for reviewers [ 23 ]. However the order could affect the synthesis output [ 11 , 12 , 24 , 45 ]. There are different perspectives to the use of tools in the quality assessment of qualitative research [ 46 ]. Some recommend the exclusion of studies based on a low-quality assessment and others refute this view and suggest that such tools may not truly assess the meaningfulness and potential impact of qualitative findings [ 47 ]. However, we believe that these checklists can equip novice qualitative researchers with the resources to evaluate qualitative research efficiently.

Two common and widely used quality assessment tools are the Critical Appraisal Skills Programme (CASP) and the Qualitative Assessment and Review Instrument (JBI-QARI). The Critical Appraisal Skills Checklist (CASP) provides detailed instructions and decision rules on how to interpret the criteria [ 48 ]. This checklist contains a number of questions which help the reviewer to assess the rigour, credibility and relevance of each study [ 49 , 50 , 51 , 52 ]. All studies are critically appraised and each study is assigned a numerical score out of ten, where a higher score is correlated to a higher quality [ 15 ]. The two studies ranked with the highest scores are used as index studies, and can be used as the first studies from which concepts are translated into other studies and therefore shaping the analysis [ 12 ]. Similarly, the Qualitative Assessment and Review Instrument (JBI-QARI) is a 10 item checklist which assesses the methodological quality of a study, and determines the extent to which a study has addressed the possibility of bias in its design, conduct and analysis [ 53 ]. Some researchers provide guidelines for determining and excluding studies which have major methodological flaws [ 54 ]. However, it can be argued that excluding studies based on quality criteria may result in the exclusion of insightful studies. GRADE-CERQual is a recently developed approach which provides guidance for assessing how much confidence to place in findings from systematic reviews of qualitative research [ 55 ]. The application of GRADE-CERQual can be helpful for appraising the overall quality of the qualitative synthesis [ 55 ] but a quality appraisal of primary studies is required before applying the CERQual tool.

We used the CASP checklist to assess the quality of included studies. We chose to use the CASP as it propagates a systematic process through which the strengths and weaknesses of a research study can be identified [ 56 ]. The CASP guidelines are easy to follow, especially for novice researchers [ 56 ]. We made a decision in advance not to exclude studies with low quality scores. We believed that although some authors may have failed to describe the methods in sufficient detail for us to determine that quality criteria had been met, lack of reporting did not necessarily mean it was poorly conducted research [ 12 ]. We did however use the quality rating of the studies in our synthesis approach. The study ranked with the highest score was used as the ‘index study’ and was the first study from which concepts were translated into other studies and therefore shaping the analysis [ 12 ].

Phase 3: Reading the studies

It is during this phase where the synthesis process begins. First, this involves repeatedly reading the included studies and familiarising yourself with the key concepts and metaphors. It is important at this stage to become as familiar as possible with the content and detail of the included studies. A concept is defined as ‘having some analytical or conceptual power, unlike more descriptive themes [ 26 ]. It is important to acknowledge that reading the studies is not a discrete phase; reading occurs throughout the synthesis process. The notion of first, second order and third order constructs [ 26 ] are useful in distinguishing the ‘data’ of the meta-ethnography which are defined in Table  1 below.

Once you have read through the chosen studies, the next step involves extracting the ‘raw data’ from the studies for the synthesis. The raw data for a meta-ethnographic synthesis are the first and second order constructs [ 29 , 31 ]. The data needs to be extracted from each of the studies, which can be done by using a standardised data extraction form [ 11 ]. Alternative ways to extract data include creating a list of metaphors and themes [ 32 ] or coding concepts in Nvivo; a software programme for the analysis of qualitative data [ 31 ]. The data should be extracted verbatim, so there is no risk of losing important data [ 12 ] and to preserve the original terminology used by the primary authors. However, some authors of a previous meta-ethnography chose to record summaries of primary author interpretations due to the large number of studies included in their synthesis [ 12 ]. However, one of the drawbacks of recording such summaries is that there is the risk of potentially losing important detail.

It is essential at this stage to extract information on study characteristics for each study, using a separate data extraction form as it provides context for interpretations and explanation of each study [ 57 ]. This includes information on study sample, data collection methods, data analysis methods, study outcomes and study conclusions.

We have provided an example of a data extraction table we used to extract the raw data (Fig.  1 ).

Example of a data extraction table

Phase 4: determining how the studies are related

During this stage, the relationships between the key concepts from the different papers need to be considered. A concept is described as a ‘ meaningful idea that develops by comparing particular instances’ [ 29 , 45 ]. It is also important that concepts explain and do not only describe the data [ 29 , 45 ] as one of the aims of qualitative analysis is to develop concepts which help to understand an experience and not just describe it [ 58 ]. In order to consider the relationship between concepts from the different studies, you are required to look across the studies for common and recurring concepts. This can be done by creating a list of the themes [ 10 ]. These are then juxtaposed against each other to examine the relationships between the key concepts and metaphors these themes reflect and to identify common and recurring concepts. From this list, the themes from the different studies are then clustered into relevant categories, where we grouped common concepts from studies according to the common underlying metaphors, an approach which has previously been used [ 12 , 31 , 59 ]. During this phase it is essential to examine the contextual data about each study. This includes settings, aims and focuses. These newly formed categories are labelled using terminology which encompasses all the relevant concepts they contain. This phase is likely to be iterative, and clusters may be revised through discussions within the review team of how they are related and by making reference to the original text.

Other authors have used diagrams [ 11 , 32 ] or coding using qualitative analysis software [ 31 ]. The use of lists or tables in phase 4 is useful when synthesising a small number of studies, however such an approach would be unwieldy when there are hundreds of concepts, whereas coding in NVivo is efficient [ 23 ]. However, the recording of links between concepts within primary studies may be difficult when using NVivo [ 23 ].

During this phase, for our review we created a list of the themes from each paper (Fig.  2 ) listed under each study name. As we had included both healthcare professional and patient studies, we also labelled whether the study had included patients, healthcare professionals or both groups.

List of key metaphors/concepts from each study

The next step involved reducing the themes from the different studies into relevant categories (Fig.  3 ).

Reducing themes into relevant categories

It is important to note that the category labels you create during this phase are not the higher third order constructs, but are descriptive labels. The third order constructs are developed within the next two phases. However, the data within each category forms the basis of reciprocal translation or refutational synthesis in the next stages. This approach can work well when you have a manageable number of studies, however this can prove to be challenging when you have a larger number of studies. In previous meta-ethnographies where a large number of studies have been included, a thematic analysis of themes was carried out instead [ 12 ].

Phase 5: translating the studies into one another

The original method of meta-ethnography suggests that this phase involves ‘comparing the metaphors and concepts in one account with the metaphors and concepts in others’ [ 10 ]. However, despite a number of meta-ethnographies being conducted, it is unclear how this should be done and how this phase of the analysis should be recorded. To address this lack of clarity, we will now outline below one way in which this can be done. During this phase, each concept from each paper is compared with all the other papers to check for the presence or absence of commonality. Doing this highlights the similarities and differences between the concepts and metaphors and allows the researcher to organise them into further conceptual categories, which results in the development of the higher third order constructs.

This phase is approached by arranging the studies either chronologically [ 32 ] from the highest scoring paper to the lowest scoring paper (where the scores are generated during the quality appraisal process [ 15 ]. Arranging the studies chronologically is advised when you are including a large number of papers over a large time span [ 12 , 29 ]. The order in which studies are compared may influence the synthesis, as earlier papers will have a strong influence on the subsequent development of ideas [ 60 ]. The reviewer first starts by summarising the themes and concepts from paper 1. Summarising involves comparing and contrasting the concepts taking into account study contexts. They then summarise the themes and concepts from paper 2, commenting first on what is similar with paper 1 and then what paper 2 may add to paper 1 or where its findings diverge from paper 1 [ 12 , 29 ]. Next, paper 3 is summarised, considering what is similar to papers 1 and 2, and then noting any areas of divergence and anything that paper 3 adds to the knowledge offered in papers 1 and 2. This process continues until you have synthesised all the papers and produces a synthesis of the primary author interpretations (see Fig.  4 ) which are useful in aiding with the development of the third order constructs in the next stage.

Primary data synthesis of the primary author interpretations

Examining the key concepts within and across the studies is similar to the method of constant comparison [ 29 ]. During this phase, it is important to refer back to the table of study characteristics you recorded earlier, (country, sample, recruitment method, gender, publication date etc.) to use as a context for the comparisons [ 15 ] as well the full papers. This process can also be supported by creating a translations table, as this is a useful way to display this level of synthesis [ 61 ] (see Fig.  5 for an example of a translations table). Maintaining a personal journal during this phase of the analysis can help to ensure that the researcher is aware of their position from a theoretical point of view [ 62 ]. Discussing the key concepts and their meanings with team members can result in collaborative interpretations.

Example of a translations table

We conducted two separate syntheses; one for the views of patients and one for healthcare professionals, and conducted a line of argument synthesis of all the included studies, therefore we found it useful to have two separate translation tables; one for each group. Part of the translation table for healthcare professionals is shown in Fig. 5 (see example of table below).

Phase 6: Synthesising the translations

This phase is described by [ 10 ] as ‘making the whole into something more than the parts alone imply’. However, similar to Phase 5, there has been no clear guidance on how to carry out this phase. During this phase, the studies are now viewed as a ‘whole’ with the aim of developing a framework [ 29 , 31 ]. When writing about how the studies are related, reviewers can present this in a narrative and/or diagrammatic form [ 30 ]. Phase 6 can be broken down into the following two stages; (a) reciprocal and refutational synthesis and (b) line of argument synthesis.

(A) Reciprocal and refutational synthesis

This stage of the synthesis involves deciding whether the studies are sufficiently similar in their focus to allow for a reciprocal translation synthesis. Alternatively, the studies may refute each other in which case a refutational synthesis is conducted. It is possible to conduct both types of synthesis to discuss similar accounts (reciprocal translation synthesis) and also explore any contradictions between the studies (refutational synthesis) [ 23 ]. Generally, reciprocal translation syntheses are conducted more frequently in reviews than refutational syntheses and guidance on how to conduct a refutational synthesis is currently limited [ 23 ]. Below we first discuss how to carry out a reciprocal translation and then describe the way a refutational synthesis can be conducted. Referring to the translations table of data developed in the stages above allows reviewers to establish the relationship between the studies and consider how to approach a reciprocal and refutational synthesis.

Reciprocal translation

It is during this phase where the shared themes across the studies are summarised by juxtaposing the first and second order constructs. This leads to the generation of new concepts which provide a fuller account of the given phenomenon and resolve any contradictions [ 63 ]. These are known as the original third order constructs developed by the review authors and provide a new understanding of the phenomena [ 15 ]. To put briefly, this can be achieved by reading the primary data synthesis (Fig. 4 ) alongside the translations table (Fig. 5 ) and drawing out the main points to form the reciprocal translations and therefore developing the third order constructs. It is important to constantly check the summary and third order constructs you are developing against the translations table to ensure it is consistent with the original data.

Refutational synthesis

There are limited published examples of refutational synthesis [ 25 , 45 ] as reviewers often focus on reciprocal translations [ 25 ]. Also reviewers may conduct a refutational synthesis, but not label it as such [ 23 ]. There are two published examples of refutational synthesis [ 43 , 64 ]. This is not surprising given the lack of guidance available on how to conduct a refutational synthesis. The purpose of a refutational synthesis is to explore and explain the differences, exceptions, incongruities and inconsistencies in concepts across the studies [ 1 , 24 ]. Refutational synthesis focuses on identifying, understanding and reconciling the contradictions, rather than developing concepts around the similarities. Similar to reciprocal translation, reviewers are required to refer back to the primary data synthesis and translations table in order to develop third order constructs. The contradictions between the concepts across the studies may be explained by differences in participants, settings or study design. During this phase, it is helpful to refer back to the study characteristics table as this can help provide context for interpretations and explanations [ 57 ]. It has been suggested that a refutational translation can be approached by placing two refutational concepts at either end of a continuum and proceed by analysing the differences between the concepts [ 22 , 28 ]. In order to express the refutational findings, a narrative can be created so that the findings ‘are placed into context’ [ 28 ].

(B) A lines of argument synthesis

A lines of argument synthesis can then be created from the third order constructs, which involves ‘making a whole into something more than the parts alone imply’ (known as higher order interpretations) [ 10 ]. A lines of argument synthesis means that there is an ‘interpretation of the relationship between the themes, which further emphasises a key concept that may be hidden within individual studies in order to discover the whole from a set of parts’ [ 10 ]. This is classed as a further higher level of interpretative synthesis, and provides scope for developing new insights.

A lines of argument synthesis is achieved by constant comparison of the concepts and developing a ‘grounded theory that puts the similarities and differences between the studies into interpretative order’ [ 10 ]. Practically, reviewers can approach this phase by reading through the reciprocal translations and noting down the similarities and differences between each of the third-order constructs. These notes can then be used to construct interpretations of how each third order construct relates to the others in the analysis. These relationships can then be represented using a diagram to aid understanding. Each of the reviewers can carry out this stage independently, and merge their findings as a team to produce the final line of argument synthesis. Diagrams can be used to develop the line of argument synthesis and it is suggested that discussions between team members are vital to this process [ 29 , 30 ]. A lines of argument synthesis can be a useful way to bring together and explain the perspectives of two or more different groups and interpreting the relationship between the themes. This is particularly relevant for research in healthcare, where often the views of one or more groups are examined on a phenomenon (e.g. patients and healthcare professionals). An example of a line of argument synthesis from the worked example is presented in Fig.  6 .

Example of a line of argument synthesis developed

We conducted separate reciprocal translations for the first- and second-order constructs relating to patients and healthcare professionals, resulting in third order constructs which related to solely either patients or professionals. Therefore, the synthesis process for our review consisted of three steps- (1) reciprocal translations of the patient studies to understand patients’ views and experiences of disclosure, (2) reciprocal translations of the healthcare professional studies to explore healthcare professionals views and experiences on disclosure and (3) a line of argument synthesis which contributed to the identification of both the key elements of an ideal disclosure desired by patients and the facilitators for healthcare professionals which can increase the likelihood of this taking place. We initially considered a refutational translation instead of a line-of-argument synthesis, but it was apparent during the synthesis that the concepts from the patient and healthcare professional studies were not contradictory in nature; rather they described alternate perspectives of the same phenomenon. Therefore we believed a line of argument synthesis was the most appropriate for the aim of our synthesis. During this stage of the analysis, we found it helpful to place all the third order constructs in a table to enable visual comparison (see Table  2 ).

The third order constructs should be theoretically rich. In our synthesis, although we found that the data we were dealing with was descriptive, it was rich descriptive data. This therefore provided us with sufficient detail to further interpret this and develop third order constructs [ 23 ]. The third order constructs we developed reflected the data we were dealing with, but allowed us to produce higher levels of analysis. Reviewers should take caution when dealing descriptive data. They need determine whether it is ‘thin descriptive data’ which could be problematic to further interpret due to lack of depth, or ‘rich descriptive data’ which can provide sufficient detail to be further interpreted [ 23 ].

Phase 7: expressing the synthesis

Reviewers should follow the eMERGE reporting guidance when writing up the synthesis [ 22 ] and the PRISMA guidelines may be used alongside this if systematic searches are conducted as many journals may require a PRISMA diagram [ 65 ]. In addition to these standard reporting methods as described by the eMERGE guidance [ 22 ] the final phase can be broken down into the following three stages; (a) summary of findings, (b) strengths, limitations & reflexivity and (c) recommendations and conclusions (refer to [ 22 ] where this phase is described in further detail).

Meta-ethnography is an evolving approach to synthesising qualitative research and is being increasingly used in healthcare research [ 29 ]. A meta-ethnographic approach offers a greater description of methods and higher-order interpretation (an overarching explanation of a phenomenon that goes beyond what the study parts alone imply), compared to a conventional narrative literature review [ 12 ]. The use of this approach can assist in generating evidence for healthcare staff, researchers and policy-makers. Although this approach is being used by numerous reviewers, transparency on how each of the stages should be conducted is still poor and there is a lack of clarity surrounding the exact stages reviewers utilise to reach their final synthesis [ 23 ]. The ultimate aim of qualitative research synthesis in healthcare is to contribute towards improvements in patient care and experience, as well as improving the processes for healthcare professionals involved [ 39 ]. In order for a meta-ethnography syntheses to be considered to be of high quality and useful, the meta-ethnographic approach needs to be rigorous and consistent. Therefore, a clear understanding of the steps included in a meta-ethnography is vital to produce a synthesis which is rigorous and comprehensive. Poorly reported methods of meta-ethnography can also make it challenging, particularly for early career qualitative researchers to conduct this synthesis. Therefore, we have provided a practical step-by-step guide to assist reviewers with conducting a meta-ethnographic synthesis of qualitative research. High quality qualitative research synthesis should not end with the final write up and further research needs to focus on how the impact of qualitative research can be maximised to improve healthcare.

Like any other method, the meta-ethnographic approach is not without its limitations. Within a meta-ethnography, although reviewers provide a synthesis, this is only one interpretation and as qualitative synthesis is subjective, several alternative interpretations are likely to be possible [ 66 ]. The subjective nature of a meta-ethnography may also affect the representativeness of the synthesis findings. To develop this guide, we searched for articles in a number of ways which is described in detail in the methods section. However, as a systematic literature search was not conducted to identify articles for the development of this guide, there is the potential that this may have resulted in the exclusion of some articles. Whilst we have provided guidance on how to conduct a meta-ethnographic synthesis, it is important to note that this is a flexible guide, which researchers can utilise and adapt the stages, according to their own research questions and the phenomenon under study. Some of the steps and challenges described in this guide hold true for systematic reviews in general. However, this guide aimed to offer practical step-by-step guidance on how to conduct meta-ethnography for even those researchers who may not be experienced in conducting systematic reviews as well as being unfamiliar with a meta-ethnographic approach. This guide was developed to assist with conducting a meta-ethnography within healthcare research. Although this guide would be potentially useful beyond healthcare research, there might be additional challenges and considerations in other research fields which may not be fully captured in this guide.

Conclusions

There was previously a lack of step-by-step guide to meta-ethnography conduct. In this paper, we have filled this gap by providing a practical step-by-step guide for conducting meta-ethnography based on the original seven steps as developed by Noblit & Hare [ 10 ]. We have incorporated adaptations and developments by recent publications and we provide detailed annotations, particularly for stages 4–6 which are often described as being the most challenging to conduct, yet the least amount of guidance is provided for conducting these stages. We have described each stage in relation to one of the previous meta-ethnographies we have conducted to aid understanding, and allows the reader to follow on from one step to the next easily.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59.

Article   PubMed   PubMed Central   Google Scholar  

Bearman M, Dawson P. Qualitative synthesis and systematic review in health professions education. Med Educ. 2013;47(3):252–60.

Article   PubMed   Google Scholar  

Erwin EJ, Brotherson MJ, Summers JA. Understanding qualitative metasynthesis: issues and opportunities in early childhood intervention research. J Early Interv. 2011;33(3):186–200.

Article   Google Scholar  

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a review of published reports. Qual Res. 2007;7:375–422.

Paterson BL. “It looks great but how do I know if it fits?”: an introduction to Meta-synthesis research. Synthesizing Qual Res: Choosing Right Approach. 2011;16:1–20.

Google Scholar  

Hannes K, Macaitis K. A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qual Res. 2012;12:402–42.

France EF, Wells M, Lang H, Williams B. Why, when and how to update a meta-ethnography qualitative synthesis. Syst Rev. 2016;5:44.

Daker-White G, Hays R, McSharry J, Giles S, Cheraghi-Sohi S, Rhodes P, Sanders C. Blame the patient, blame the doctor or blame the system? A meta-synthesis of qualitative studies of patient safety in primary care. PLoS One. 2015;10:128–329.

Article   CAS   Google Scholar  

Noblit GW, Hare RD. Meta-ethnography: Synthesizing qualitative studies. California: Sage Publications Ltd; 1988.

Book   Google Scholar  

Malpass A, Shaw A, Sharp D, Walter F, Feder G, Ridd M, Kessler D. “Medication career” or “moral career”? The two sides of managing antidepressants: a meta-ethnography of patients’ experience of antidepressants. Soc Sci Med. 2009;68:154–68.

Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol. 2008;8:21.

Classen S, Alvarez L. Evidence-informed reviews- moving occupational therapy practice and science forward. OTJR: Occup, ParticipationHealth. 2015;35:199–203.

Soilemezi D, Linceviciute S. Synthesizing qualitative research: reflections and lessons learnt by two new reviewers. Int J Qual Methods. 2018;17(1):1–14.

Scott DA, Grant SM. A meta-ethnography of the facilitators and barriers to successful implementation of patient complaints processes in healthcare settings. Health Expect. 2018;21:508–17.

Elmir R, Schmied V. A meta-ethnographic synthesis of fathers′ experiences of complicated births that are potentially traumatic. Midwifery . 2016;32:66–74.

Cullinan S, O’Mahony D, Fleming A, Byrne S. A meta-synthesis of potentially inappropriate prescribing in older patients. Drugs Aging. 2014;31:631–8.

Purc-Stephenson RJ, Thrasher C. Nurses’ experiences with telephone triage and advice: a meta-ethnography. J Adv Nurs. 2010;66:482–94.

Toye F, Seers K, Barker KL. Meta-ethnography to understand healthcare professionals’ experience of treating adults with chronic non-malignant pain. BMJ Open. 2017;7:e018411.

Rubio-Valera M, Pons-Vigués M, Martínez-Andrés M, Moreno-Peral P, Berenguera A, Fernández A. Barriers and facilitators for the implementation of primary prevention and health promotion activities in primary care: a synthesis through meta-ethnography. PLoS One. 2014;9:e89554.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Lee RP, Hart RI, Watson RM, Rapley T. Qualitative synthesis in practice: some pragmatics of meta-ethnography. Qual Res. 2015;15:334–50.

France EF, Cunningham M, Ring N, Uny I, Duncan EA, Jepson RG, Maxwell M, Roberts RJ, Turley RL, Booth A, Britten N. Improving reporting of meta-ethnography: the eMERGe reporting guidance. BMC Med Res Methodol. 2019;19:25.

France EF, Uny I, Ring N, Turley RL, Maxwell M, Duncan EA, Jepson RG, Roberts RJ, Noyes J. A methodological systematic review of meta-ethnography conduct to articulate the complex analytical phases. BMC Med Res Methodol. 2019;19:35.

Booth A, Carroll C, Ilott I, Low LL, Cooper K. Desperately seeking dissonance: identifying the disconfirming case in qualitative evidence synthesis. Qual Health Res. 2013;23:126–41.

Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M. Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004;13:1342–65.

Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R. Using meta ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy. 2002;7:209–15.

Bondas T, Hall EO. Challenges in approaching metasynthesis research. Qual Health Res. 2007;17:113–21.

Finfgeld-Connett D. Metasynthesis findings: potential versus reality. Qual Health Res. 2014;24:1581–91.

Cahill M, Robinson K, Pettigrew J, Galvin R, Stanley M. Qualitative synthesis: a guide to conducting a meta-ethnography. Br J Occup Ther. 2018;81:129–37.

Toye F, Seers K, Allcock N, Briggs M, Carr E, Andrews J, Barker K. ‘Trying to pin down jelly’-exploring intuitive processes in quality assessment for meta-ethnography. BMC Med Res Methodol. 2013;13:46.

Toye F, Seers K, Allcock N, Briggs M, Carr E, Barker K. Meta-ethnography 25 years on: challenges and insights for synthesising a large number of qualitative studies. BMC Med Res Methodol. 2014;14:80.

Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, Yardley L, Pope C, Donovan J. Evaluating meta ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess. 2011;15:1–164.

Article   CAS   PubMed   Google Scholar  

Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative. Medinfo. 2004;107:311.

Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res. 2007;17:705–10.

McKibbon KA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof. 2006;29:440–54.

Noyes J, Popay J, Pearson A, Hannes K. 20 qualitative research and Cochrane reviews. In: Cochrane handbook for systematic reviews of interventions, vol. 571; 2008.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6:27–44.

Benoot C, Hannes K, Bilsen J. The use of purposeful sampling in a qualitative evidence synthesis: a worked example on sexual adjustment to a cancer trajectory. BMC Med Res Methodol. 2016;16:21.

Atkinson LZ, Cipriani A. How to carry out a literature search for a systematic review: a practical guide. BJPsych Adv. 2018;24:74–82.

Dissemination CR. CRD’s guidance for undertaking reviews in health care. Layerthorpe: University of York, Centre for Reviews and Dissemination; 2009.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Syst Rev. 2012;1:19.

Flemming K, Graham H, Heirs M, Fox D, Sowden A. Smoking in pregnancy: a systematic review of qualitative research of women who commence pregnancy as smokers. J Adv Nurs. 2013;69:1023–36.

Garside R. A comparison of methods for the systematic review of qualitative research: two examples using meta-ethnography and meta-study; 2008. p. 61.

Nye E, Melendez-Torres GJ, Bonell C. Origins, methods and advances in qualitative meta-synthesis. Rev Educ. 2016;4:57–79.

France EF, Ring N, Thomas R, Noyes J, Maxwell M, Jepson R. A methodological systematic review of what’s wrong with meta-ethnography reporting. BMC Med Res Methodol. 2014;14:119.

Majid U, Vanstone M. Appraising qualitative research for evidence syntheses: a compendium of quality appraisal tools. Qual Health Res. 2018;28:2115–31.

Melia KM. Recognizing quality in qualitative research. California: Sage Publications Ltd; 2010.

Programme Critical Appraisal Skills. Qualitative research checklist. Oxford: Critical Appraisal Skills Programme; 2013. https://casp-uk.net/casp-tools-checklists/ . Accessed 15 Feb 2019.

Kitto SC, Chesters J, Grbich C. Quality in qualitative research. Med J Aust. 2008;188:243–6.

Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. Bmj. 2008;7:337.

Kuper A, Reeves S, Levinson W. An introduction to reading and appraising qualitative research. Bmj. 2008;337:288.

Mays N, Pope C. Assessing quality in qualitative research. Bmj. 2000;320:50–2.

Article   CAS   PubMed   PubMed Central   Google Scholar  

The Joanna Briggs Institute Critical Appraisal tools for use in JBI Systematic Reviews. Checklist for qualitative research: The Joanna Briggs Institute; 2017. https://joannabriggs.org/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Qualitative_Research2017_0.pdf . Accessed 10 Mar 2019.

Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. BMJ Qual Saf. 2004;13:223–5.

Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, Bohren MA, Tunçalp Ö, Colvin CJ, Garside R, Carlsen B. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation Sci. 2018;13:2.

Singh J. Critical appraisal skills programme. J Pharmacol Pharmacother. 2013;4:76.

Feast A, Orrell M, Charlesworth G, Poland F, Featherstone K, Melunsky N, Moniz-Cook E. Using meta-ethnography to synthesize relevant studies: capturing the bigger picture in dementia with challenging behavior within families. London: SAGE Publications Ltd; 2018.

Seale C. Quality in qualitative research. Qual Inq. 1999;5:465–78.

Erasmus E. The use of street-level bureaucracy theory in health policy analysis in low-and middle-income countries: a meta-ethnographic synthesis. Health Policy Plan. 2014;29:iii70–8.

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003;56:671–84.

Coventry PA, Small N, Panagioti M, Adeyemi I, Bee P. Living with complexity; marshalling resources: a systematic review and qualitative meta-synthesis of lived experience of mental and physical multimorbidity. BMC Fam Pract. 2015;16:171.

Doyle LH. Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003;3:321–44.

Strauss A, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. California: Sage Publications Ltd; 1990.

Sleijpen M, Boeije HR, Kleber RJ, Mooren T. Between power and powerlessness: a meta-ethnography of sources of resilience in young refugees. Ethn Health. 2016;21(2):158–80.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9.

Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29.

Download references

Acknowledgements

This report is independent research funded by the National Institute for Health Research, Yorkshire and Humber Applied Research Collaborations. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care.

This research was funded by NIHR CLAHRC Yorkshire and Humber.

Author information

Authors and affiliations.

University of Leeds, Leeds, LS2 9JT, UK

Raabia Sattar, Rebecca Lawton & Judith Johnson

Bradford Institute for Health Research, Bradford, BD9 6RJ, UK

National Institute of Health Research for Primary Care Research, Manchester Academic Health Science Centre, University of Manchester, Manchester, M13 9PL, UK

Maria Panagioti

You can also search for this author in PubMed   Google Scholar

Contributions

RS, RL and JJ originated the idea for this guide to conducting a meta-ethnography. RS, RL and JJ conducted the original meta-ethnography from which annotations and examples are based on within this manuscript. All authors contributed to the development of the method, and read and approved the final manuscript. RS drafted the first version of this manuscript. RS, RL, JJ and MP made significant contributions to the ideas developed and presented in this manuscript.

Authors’ information

RS 1 is a PhD student (MSc, BSc Honours) based in the School of Psychology at the University of Leeds and Bradford Institute for Health Research. RL 2 (PhD, BA) is a Professor in Psychology of healthcare, based at the University of Leeds and Bradford Institute for Health Research. MP 3 (PhD, MSc, BSc Honours) is a senior research fellow based at the Centre for Primary Care, Manchester. JJ 4 (PhD, BSc Honours) is a clinical psychologist based at the University of Leeds and Bradford Institute for Health Research.

Corresponding author

Correspondence to Raabia Sattar .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sattar, R., Lawton, R., Panagioti, M. et al. Meta-ethnography in healthcare research: a guide to using a meta-ethnographic approach for literature synthesis. BMC Health Serv Res 21 , 50 (2021). https://doi.org/10.1186/s12913-020-06049-w

Download citation

Received : 10 June 2020

Accepted : 26 December 2020

Published : 08 January 2021

DOI : https://doi.org/10.1186/s12913-020-06049-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-ethnography
• Research methods
• Qualitative synthesis
• Qualitative health research

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Technical Support
• Find My Rep

You are here

Using Mixed Methods Research Synthesis for Literature Reviews

• Mieke Heyvaert - KU Leuven—University of Leuven
• Karin Hannes - KU Leuven—University of Leuven
• Patrick Onghena - KU Leuven, Belgium
• Description

This practical guide provides step-by-step instruction for conducting a mixed methods research synthesis (MMRS) that integrates both qualitative and quantitative evidence. The book progresses through a systematic, comprehensive approach to conducting an MMRS literature review to analyze and summarize the empirical evidence regarding a particular review question. Readers will benefit from discussion of the potential advantages of MMRS and guidance on how to avoid its potential pitfalls.

  Using Mixed Methods Research Synthesis for Literature Reviews is Volume 4 in the SAGE Mixed Methods Research Series. To learn more about each text in the series, please visit www.sagepub.com/mmrs .  

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

“This text absolutely offers an avenue for introducing new ways of getting at information in the literature and combining it for effective explanations of results using a chosen methodology.”

“Each step of the MMRS process is given adequate coverage…This is a great tool for students and novice researchers.”

“The consistent format of each chapter would provide students with a clear and ‘comforting’ process for learning the material.”

“[The practical tips] are very useful and demonstrate how the information can be utilized in a manner beyond the theoretical discussion of the process.”

“‘Using Mixed Methods Research Synthesis for Literature Review’ is a book whose time has come. It provides a thoughtful overview of research synthesis techniques that have been developed from different epistemological traditions. The authors show these traditions complement, rather than compete, with one another. They integrate and present these methods in a practical manner while keeping front and center the importance of the theories underlying the synthesis’ constituent research. They argue convincingly that we need to know not only if social policies and practices work but also why and how they work. The ability of social science to contribute to social policy and practice will benefit greatly from the approach to research synthesis contained herein.”

“Here Heyvaert, Hannes and Onghena unpack literature syntheses—both qualitative and quantitative—detailing ways to synthesize, segregate, integrate and even to convert qual findings to quan, or quan to qual.  An excellent resource and the next step in moving mixed methods forward!”

KEY FEATURES :

• Theoretical guidelines and practical strategies show readers how to conduct each step of an MMRS literature review process.
• Several worked examples of MMRS literature reviews published in diverse research fields showcase how to put the theory on the MMRS process into practice.
• Each chapter sets the stage with an outline and ends with summary points , challenges , questions for thought , and suggestions for further reading to enhance reader mastery.
• Numerous boxes contain concrete research examples and illustrations to further clarify important points.
• Practical Tip boxes in every chapter include “how-to” guidelines that demonstrate how information can be utilized in a manner beyond the theoretical discussion of the MMRS process.
• Exercises provide concrete activities related to conducting each step of the MMRS process for hands-on practice.  

Sample Materials & Chapters

For instructors, select a purchasing option, related products.

This title is also available on SAGE Research Methods , the ultimate digital methods library. If your library doesn’t have access, ask your librarian to start a trial .

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Methodology
• Open access
• Published: 19 April 2012

Realist synthesis: illustrating the method for implementation research

• Jo Rycroft-Malone 1 ,
• Brendan McCormack 2 ,
• Alison M Hutchinson 3 , 4 ,
• Kara DeCorby 5 ,
• Tracey K Bucknall 3 , 4 ,
• Bridie Kent 6 , 7 ,
• Alyce Schultz 8 ,
• Erna Snelgrove-Clarke 9 ,
• Cheryl B Stetler 10 ,
• Marita Titler 11 ,
• Lars Wallin 12 &
• Val Wilson 13  

Implementation Science volume  7 , Article number:  33 ( 2012 ) Cite this article

91k Accesses

380 Citations

83 Altmetric

Metrics details

Realist synthesis is an increasingly popular approach to the review and synthesis of evidence, which focuses on understanding the mechanisms by which an intervention works (or not). There are few published examples of realist synthesis. This paper therefore fills a gap by describing, in detail, the process used for a realist review and synthesis to answer the question ‘what interventions and strategies are effective in enabling evidence-informed healthcare?’ The strengths and challenges of conducting realist review are also considered.

The realist approach involves identifying underlying causal mechanisms and exploring how they work under what conditions. The stages of this review included: defining the scope of the review (concept mining and framework formulation); searching for and scrutinising the evidence; extracting and synthesising the evidence; and developing the narrative, including hypotheses.

Based on key terms and concepts related to various interventions to promote evidence-informed healthcare, we developed an outcome-focused theoretical framework. Questions were tailored for each of four theory/intervention areas within the theoretical framework and were used to guide development of a review and data extraction process. The search for literature within our first theory area, change agency, was executed and the screening procedure resulted in inclusion of 52 papers. Using the questions relevant to this theory area, data were extracted by one reviewer and validated by a second reviewer. Synthesis involved organisation of extracted data into evidence tables, theming and formulation of chains of inference, linking between the chains of inference, and hypothesis formulation. The narrative was developed around the hypotheses generated within the change agency theory area.

Conclusions

Realist synthesis lends itself to the review of complex interventions because it accounts for context as well as outcomes in the process of systematically and transparently synthesising relevant literature. While realist synthesis demands flexible thinking and the ability to deal with complexity, the rewards include the potential for more pragmatic conclusions than alternative approaches to systematic reviewing. A separate publication will report the findings of the review.

Peer Review reports

The range of approaches to research review and synthesis has been growing over recent years [ 1 ]. One approach that has been growing in popularity is realist synthesis. A realist review focuses on understanding and unpacking the mechanisms by which an intervention works (or fails to work), thereby providing an explanation, as opposed to a judgment about how it works [ 2 ]. The realist approach is fundamentally concerned with theory development and refinement [ 2 – 4 ], accounting for context as well as outcomes in the process of systematically and transparently synthesizing relevant literature [ 3 , 4 ]. Given the complex, multifaceted nature of strategies and interventions used to promote evidence-informed healthcare, and the current limited understanding of their mechanisms of action, the realist approach is particularly suited to the synthesis of evidence about complex implementation interventions. Whilst the use of this method is increasing and the evidence base growing [ 5 – 8 ], few published studies provide a detailed account of how it has been used. This paper therefore adds to the methodological evidence base about realist synthesis by describing application of the approach of an international project team (Realist Synthesis of Implementation Strategies (ReS-IS) to synthesise evidence about knowledge translation interventions for enabling evidence-informed healthcare.

Realist synthesis

Traditional systematic review approaches have been criticised for being too specific and inflexible [ 2 , 9 , 10 ]. This is an important consideration when examining the complexity of implementing health and social care interventions. The context of service delivery is complex, multi-faceted and dynamic, which arguably means that rarely would the same intervention work in the same way in different contexts. Consequently, conventional systematic review approaches to evaluating the evidence of whether interventions work (or not) often result in limited answers such as ‘to some extent’ and ‘sometimes’ [ 4 , 9 ].

Realist review has emerged as a strategy for synthesising evidence and focuses on providing explanations for why interventions may or may not work, in what contexts, how and in what circumstances [ 11 ]. For example, Greenhalgh et al. [ 6 ] undertook a realist review to supplement a Cochrane review of school feeding programmes. Whilst the Cochrane review provided evidence that feeding programmes work, it did not provide information about how they work and in what contexts. The findings from their realist review resulted in evidence regarding situations in which programmes may be more likely to be effective.

The realist approach is philosophically rooted in realism, which combines three social science principles: causal explanations are achievable; social reality is mainly an interpretative reality of social actors; and social actors evaluate their social reality [ 12 ]. Realism involves identifying underlying causal mechanisms and exploring how they work under what conditions [ 13 , 14 ]. This contextually bound approach to causality is represented as context + mechanism = outcome [ 15 ]. Therefore, it is an intuitively appealing approach to those trying to expose and unpack the complexities of contexts and interrelated mechanisms underlying implementation activity.

The aim of a realist synthesis is ‘…to articulate underlying programme theories and then to interrogate the existing evidence to find out whether and where these theories are pertinent and productive. Primary research is examined for its contribution to the developing theory…’ [ 4 ]. In the context of implementation interventions, which are usually multi-faceted and complex, when setting out to develop and implement an intervention there is always an underlying theory about how it should work: if we do X in this way, then it will bring about an improved outcome [ 2 , 4 , 9 ]. The logic underpinning the aim of uncovering underlying theories about interventions is that no deterministic theories can always explain or predict outcomes in every context [ 16 ]. Focussing on what it is about an intervention that makes it work (or not) in a given context should enable implementation researchers to work at the level of mechanisms of action. The premise is that in certain contexts individuals are likely (although not always certain) to make similar choices, and therefore particular contexts influence our choices such that reoccurring patterns emerge, i.e. , demi-regularities [ 4 ]. Realist review provides an approach to uncover the underlying theories that explain these demi-regularities by examining the interactions between mechanism, context, and outcome.

A realist synthesis follows similar stages to a traditional systematic review (Table 1 ), but with some notable differences:

The focus of the synthesis is derived from a negotiation between stakeholders and reviewers and therefore the extent of stakeholder involvement throughout the process is high.

The search and appraisal of evidence is purposive and theoretically driven with the aim of refining theory.

Multiple types of information and evidence can be included.

The process is iterative.

The findings from the synthesis focus on explaining to the reader why (or not) the intervention works and in what ways, to enable informed choices about further use and/or research [ 3 ].

The rest of this paper describes in detail the ReS-IS team’s approach to applying the realist synthesis method to a review of interventions that enable evidence-informed healthcare, including the strengths and challenges encountered in its use. A separate publication will describe the findings of the review and synthesis in detail.

Review purpose

Numerous interventions have been applied and tested to promote the use of research evidence in practice. Implementation of such interventions is often accompanied by complex strategies comprising support structures, resources and processes. While numerous systematic reviews have been conducted to determine the effectiveness of specific interventions [ 17 – 22 ], a systematic synthesis of the literature to examine the mechanisms by which such interventions work, and under what circumstances, has not been undertaken. Therefore the broad purpose of this review was to determine what interventions and strategies are effective in enabling evidence-informed healthcare. The specific purpose of the review was to establish what works, for whom, in what circumstances, and why with respect to interventions and strategies to enable evidence-informed healthcare. The purpose of the review was refined through stakeholder engagement at a knowledge utilisation colloquium meeting ( http://www.uofaweb.ualberta.ca/kusp/KU0Xarchive.cfm ). The stakeholders are a multi-disciplinary community of researchers, practitioners, and policy makers with expertise in knowledge translation. This community served as a stakeholder reference group that were consulted at key stages in the review process: questions formulation, tool development, and evidence synthesis, providing critique and challenge to the method and emerging findings.

Defining the scope of the review: Concept mining and theory formulation

This stage is fundamental to a realist synthesis because it provides the structure and framework for examining and synthesising diverse evidence [ 9 ]. The challenge of developing a framework for a realist synthesis is in finding a level of abstraction that allows reviewers to stand back from the detail and variation in the evidence, but that is also specific enough to meet the purpose of the review. For a realist synthesis, an intervention is a theory; because interventions are implemented on a hypothesis of if we do X in this way, then it will bring about an outcome. This stage involves ‘digging through’ the literature and drawing on experience to identify key terms, concepts and mid-range theories that provide some explanation about the subject of interest [ 4 ]. The resultant model must be outcome-focused because a realist synthesis is concerned with uncovering ‘what works’ within differing contextual configurations. For this review, concept mining and theory formulation was achieved through a mixture of face-to-face and virtual brain storming by the team familiar with the implementation and knowledge translation research literature.

A list of questions about interventions to promote knowledge use was developed and refined through extensive dialogue in small and larger group discussions. From these questions, the meaning of terms was clarified and concepts were identified. For example the meaning of terms such as ‘evidence-based,’ ‘knowledge,’ ‘strategy,’ and ‘intervention’ was discussed. It was agreed that an intervention was a broad concept, while strategies referred to mechanisms or approaches to achieve the intervention. Consensus was reached on the use of the term ‘evidence-informed,’ in preference to ‘evidence-based,’ on the grounds that decision making in practice, for good reasons, is not always research-based, but rather takes into account a range of factors in addition to empirical evidence [ 23 ]. We also developed a common understanding of terms related to realist synthesis (see Additional file 1 ).

The group then identified common concepts among the questions developed in the previous activity. Eventually four key concepts, i.e. , ‘theory areas’ emerged: change agency, technology, education and learning strategies, and systems change. These concepts were then framed to construct the theoretical framework, which underwent a number of iterations before declaration of the final framework. Figure 1 presents the final framework, which includes theory and contextual factors, dose and levels as central factors because of their importance in understanding implementation interventions, and outcomes is represented as an all-encompassing factor. The term ‘dose’ was used to describe how much of an intervention would bring about change. We recognise that the term dose does not necessarily fit well within the philosophy of realism, however in the context of this review we used it to ensure we paid attention to the relative strength of action of a mechanism within a particular context. The generic questions developed earlier were then customised to each of the four theory areas and these were used to guide the review and data extraction process (see Additional file 2 for foci and questions).

Review framework.

The theoretical framework is made visible in a realist synthesis through the ‘data extraction forms.’ This is a unique feature of realist synthesis in that, unlike traditional systematic reviews, a bespoke set of data extraction forms are developed based on the content of the theoretical framework (see Additional file 3 ). The theory area questions, described previously, provided the basis for development of these forms. In order to prevent misinterpretation of the data extraction questions, guidelines for using the forms were developed to explain each section and factors to consider when applying each question during extraction of information from an article.

Pawson et al. [ 9 ] caution that completely comprehensive reviews may be impossible and recommend that the programme theories to be inspected be agreed upon and prioritised. As the framework reveals, the scope of our work was potentially vast. Therefore, the ReS-IS team engaged in discussion with the wider knowledge utilisation community through presentations and discussions, and as a result adopted a pragmatic approach that resulted in prioritisation of one theory area within the framework, change agency, for a first review. The programme theory questions for this theory area were:

What impact do the characteristics of the change agent have on evidence-informed healthcare?

What is the overall impact of the change agent intervention on evidence-informed healthcare?

What impact does the interaction between the change agent and the setting have on evidence-informed healthcare?

These questions were framed as ‘what’ questions in order to help us determine what it is about the particular actions (mechanisms) of change agents that have an impact and connect themes to action. Overall our approach to analysis was concerned with understanding why mechanisms were having an effect (or not).

Search for, and appraisal of evidence

Search approach.

In realist synthesis the literature needs to be scrutinised to identify studies related to the targeted ‘programme theories’ [ 9 ]. This focus does not mean that the approach to search and appraise literature is any less rigorous or systematic than approaches used in traditional reviews. The search is purposive in that for each of the theory areas the group embarked on producing a list of relevant and related search terms. The final list of terms, in conjunction with relevant indexing terms, was used to guide the searches, which were conducted by two team members in consultation with their institution’s health science librarians (see Additional file 4 for search terms and strategy).

Six online databases were searched: Medline, CINAHL, Embase, PsycInfo, Sociological Abstracts, and Web of Science. Ovid was used to execute the search within a 10-year publication period, which was considered an appropriate timeframe in the search for intervention studies. As a quality measure, one group member reviewed the indexes of 14 journals that publish articles about knowledge utilization issues. A second group member determined that relevant papers from these journals were adequately indexed in the databases selected. Additionally, using their knowledge of the literature all team members reviewed the final reference list to ensure there were no obvious omissions. Although Pawson et al. [ 9 ] describe the relevance of snowballing and consultation with experts as part of the realist review process, we did not have the resources to extend our search beyond the databases described above, and therefore acknowledge we may have missed some key pieces of evidence.

Search results were saved as text files and downloaded into Reference Manager Professional Version 11.0. The content of the file was then backed-up to a secure server. Over 24,000 electronic references were returned from the change agents search strategies, which were screened through a process summarised in Figure 2 .

Screening process.

The preliminary screen was intentionally inclusive to capture all articles potentially relevant to the review purpose, including those outside of healthcare. We erred on the side of inclusion wherever a title appeared to be even potentially relevant to the change agent/agency concept or any of the terms of reference developed in relation to the change agency component of the theoretical framework. At this stage, all seemingly relevant papers were retrieved in full-text for a more detailed relevance test: ‘Is the evidence provided in this theory area good and relevant enough to be included in the synthesis.’

Using criteria for exclusion, in the next stage we undertook a preliminary screening of the article titles retrieved in the search and reduced the list of potentially relevant papers to 196. Second-level screening resulted in the exclusion of a further 144 papers, bringing the final number of papers relevant to this theory area to 52. This process was managed by two team members, in consultation with the wider group as appropriate. Our consistent reference point was the review framework and related theory area questions. Therefore decisions about what was eventually included and excluded was managed through discussions about whether the papers were of direct relevance to interventions that had been evaluated within a healthcare context.

The primary reasons for exclusion of papers at this level included:

They were not research based (using the broadest definition of research, i.e. , demonstrating a systematic approach to inquiry).

They were purely an anecdotal account.

They were not relevant to healthcare, e.g. , were based on an unrelated field such as banking (we did not have resources to undertake a review of the wider literature).

To test the usability and functionality of the data extraction form and to promote a consistent approach to data extraction, the tool was pre-tested, by all group members, on two purposefully selected articles. Pre-testing resulted in some minor modifications to the instrument prior to commencement of the review process. The group divided into five subgroups and the references related to the change agent theory area were divided evenly across the subgroups for appraisal.

In contrast to other review processes, in a pure realist synthesis no literature is excluded (unless it does not relate to any of the theory areas). Because a paper is excluded in one theory area does not mean that it is necessarily excluded in other theory areas. Therefore, exclusion criteria at this level need to be explicit and a clear rationale needs to be documented for each article that is excluded. Consistent with Pawson’s suggestions, in this review the test for inclusion was: Is the evidence provided in this theory area ‘good and relevant enough’ to be included (consider issues of sample size, data collection, data analysis, and claims made)?

Discrepancies in opinions about the relevance of articles were resolved through discussion amongst the review group. Questions about relevance were guided by the theory area questions, and how particular pieces of evidence did or did not inform these. Final decisions erred on inclusiveness, rather than exclusion.

Extraction and synthesis

Data extraction.

If considered relevant, data were extracted from the article and then peer reviewed and checked by a second member of the subgroup. Even though the initial review wave comprised articles related to the change agent theory area, questions relating to each theory were applied during the extraction process because many of the articles reported aspects that were also relevant to the other theory areas. As the contents of the review’s theoretical framework were embedded in data extraction forms, these provided a template to ‘interrogate’ the papers. When extracting data, if an article did not include information relevant to a question in the form, the extractor recorded ‘Not reported.’ Direct quotations from the article were often most informative and were accompanied by the page number from which the quote was taken. Data pertaining to the target population and discipline for each study were also extracted.

The aim of the data extraction process is to populate the evaluative framework with evidence. Therefore, once all subgroups had completed data extraction for the articles deemed relevant to the review purpose, the content from each group’s data extraction tables was amalgamated to form a single data extraction table including all articles addressing change agency.

Data synthesis

The basic task of the synthesis process is to refine the programme theory; i.e. , to determine what works, for whom, in what circumstances, in what respects and why. Pawson et al. ’ publications provide little guidance on how to approach data synthesis. They suggest that synthesis should focus on four dimensions: questioning the integrity of a theory, adjudicating between competing theories, considering the same theory in comparative settings, or comparing the ‘official’ theory with actual practice [ 4 , 9 ].

In this review we developed an approach to synthesis based on the principles of realist evaluation [ 15 ], including the following steps:

Organisation of extracted data into evidence tables,

Theming by individual reviewers,

Comparison of reviewers’ themes for a specific article and formulation of chains of inference from the identified themes,

Linking of the chains of inference, and tracking and linking of articles

Hypothesis formulation.

This process is summarised in Additional file 5 .

At a face-to-face meeting, sub-groups undertook a process of analysis, focussed on addressing the three programme theory questions (see Figure 1 and ‘Defining The Scope’ section above).

Each question was assigned to a subgroup, and its members’ independently themed relevant data extracted from each article. The subgroup then collated the themes identified by each of the members. At this point, emerging findings were challenged, and contrary examples sought. From here the subgroup members identified chains of inference. A chain of inference is a connection that can be made across articles based on the themes identified. Over a period of time, through virtual and face-to-face meetings, chains of interference were developed and refined. This process was both inductive and deductive. We did not get to the stage of retroduction within this review, which could therefore be viewed as a limitation of the review process reported here. Further work would need to be undertaken on the hypotheses to make better inferences about generative mechanisms.

The group then formulated hypotheses regarding the chains of inference. Thus, themes were linked to chains of inference, which were then linked to a hypothesis. Further, all papers were explicitly linked to chains of inference and hypotheses. Additional file 5 includes full details of the stages of synthesis, and Additional file 6 presents full details of the hypotheses linked to themes, chains of inference, and papers. Tables 2 and 3 provide a summary of these stages.

Development of narrative

Typically the writing of the review follows the theoretical framework model developed. For this review, the narrative was organised according to hypotheses generated within the Change Agency theory area, with the data for each theme linked in two ways; to one another within each hypothesis, and also across hypotheses. Whilst the findings from this theory area will be presented in a separate publication, the following presents a summary to illustrate our approach to analysis and synthesis.

Data from extraction tables were summarised and organised into theory area and questions related tables, for example, in relation to the question ‘what impact do the characteristics of the change agents have on knowledge utilization,’ the table would include extracts of data, and the link back to the source papers.

These data were then themed, those themes were challenged and contrary evidence sought. In relation to the characteristics of change agents a number of emerging issues emerged that could be considered as relevant conditions for change agency within an opinion leader role for example, such as confidence, years of experience, level of qualification (educated to post-graduate level), and willingness to work collaboratively.

Looking for chains of inference (connections) across extracted data and themes. Through an iterative process, connections were looked for across data/themes to build up a cumulative picture. For example, were the opinion leader conditions evident in papers about other change agent roles—this resulted in a larger list factors, including, for example attitude, expert knowledge, gender, age, and cultural compatibility (see Table 2 for more examples), leadership, being embedded, tailoring, partnerships, influence, culture, support, and resources.

Hypotheses formation (mechanism, context, outcome chains). The output from steps three and four resulted in a cumulative picture of potential mechanisms, contexts and outcome chains (hypotheses), which could be linked back to source evidence (see Table 3 for an example). Our review indicates that, for example, change agents who are adequately supported and resourced (context) who role model the practices they espouse (mechanism) may impact more positively on achieving evidence-informed healthcare (outcome).

The hypotheses acted as synthesised statements of findings against which the previous stages of analysis could be presented. The narrative was developed around each hypotheses, and summarised the nature of the context, mechanism and outcome links, and the characteristics of the evidence underpinning them

Pawson et al . [ 9 ] do not suggest that ‘recommendations’ per se are developed from realist synthesis, as the purpose is not to determine ‘best’ practice, but to describe the relationships between interventions and the contexts in which those interventions occur. They do however suggest that stakeholders are engaged both in the process of ‘validating’ the emerging findings and in dissemination activities. To that end, our review approach and the developing findings were shared with a community of knowledge utilisation researchers and practitioners at an annual colloquium in 2009. This process helped to refine the focus and presentation of the narrative from the programme theory area. It also validated our view that some new insights about change agency had emerged from this review process, which will be reported in a separate paper.

There are few published examples of realist syntheses and those that exist do not include a detailed account of the approach used because authors tend to focus on the dissemination of findings within publications [ 6 , 7 , 24 – 27 ]. This lack of information about application of the realist approach is unhelpful to a novice realist reviewer. To fill this gap, this paper presents in some detail the approach we took to conduct a realist synthesis of evidence about the effect of change agency on evidence-informed healthcare. Change agency is a complex implementation intervention, which made realist synthesis an appropriate approach for unpacking its effects within different contexts and groups. However, undertaking this review was not without its challenges, not least because of the practicalities of working as an unfunded and geographically dispersed group.

Our approach deviated in some respects from the approach recommended by Pawson et al. For example, Pawson et al . do not advocate a comprehensive literature search, or double reviews and data extraction. In this sense we developed a hybrid approach that was fundamentally rooted in realist synthesis philosophy and principles ( i.e. , theory led, purposive, iterative, stakeholder involvement), but which also drew on some of the practices of traditional systematic reviewing.

One of the strengths of realist review is the approach’s firm roots in philosophy and social sciences [ 2 , 9 ]. Rather than being a method or formula, it is a ‘logic of enquiry’ [ 9 ], which enables a flexible, all-embracing approach to explanation of what works for whom in what circumstances and in what respects. Rather than controlling for real life events, realist synthesis provides a framework for working with and untangling the complexity of real-life implementation. This allows for an equal focus on what works, as much as what does not work, in an attempt to learn from failures and maximise learning across policy, disciplinary and organisational boundaries. Furthermore, realist synthesis is inherently stakeholder driven, which facilitates engagement and the inclusion of multiple perspectives.

The strengths of realist synthesis underpin its limitations. Realist synthesis is premised on a set of principles rather than a formula, and whilst this allows for flexibility and inclusivity, it means that the findings from a review are theoretically transferable. For example, it follows that if the appraisal and data extraction needs to be bespoke to the particular review questions that arise from the theoretical framework, these will be different for each review. Furthermore, given that the fundamental interest in realist synthesis is about finding out what works in what contexts, the recommendations one can make will not be generalisable. Rather a realist review results in findings that are theoretically transferable; ideas (‘theories’) that can be tested in different contexts, with different stakeholders.

Pawson et al . [ 2 ] suggest that realist syntheses are not for novices. Unlike a Cochrane review, for example, which relies on standardised protocols and tools, the demands on a realist synthesiser are different. For example, quality assurance within realist synthesis is dependent on reviewers’ explicitness and reflexivity. During this review, we kept a log of the process and decisions made throughout the process, which we developed into a technical report. In addition, we undertook a more formal reflective process during the review because members of the group had varying experiences of realist review. This involved reflecting on questions about what was going well, what was going less well, as well as engaging in group learning activities. Throughout the review process, we had large and small group discussions that provided the opportunity for building in checks and balances, and for explicating processes. In turn, this requires a high level of expertise in reasoning, research methods and quality appraisal, and expertise in the subject area. The complexity of the realist synthesis approach means that it is time-consuming and human resource intensive, and for those reasons a potentially expensive endeavour.

Realist synthesis is a new but emerging approach to evidence review. In this paper, we have described our use and development of the approach. It is particularly appropriate for unpacking the impact of complex interventions because it works on the premise that one needs to understand how interventions work in different contexts, and why. It is not an easy option. Realist review demands much of the reviewer, including an ability to think flexibly and deal with complexity. There is not one prescribed approach to doing a realist synthesis; rather, there is a set of principles that the reviewer must particularise to the issue being explored whilst being sympathetic to the philosophy of realism. This presents unique challenges, but with it, the opportunity to develop more pragmatically insightful conclusions than those produced by some other approaches to systematic reviewing.

Dixon-Woods M, Agarwhal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.

Article   PubMed   Google Scholar  

Pawson R, Greenhalgh T, Harvey G, Walshe K: Realist review - a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005, 10: 21-34. 10.1258/1355819054308530.

Rycroft-Malone J, McCormack B, DeCorby K, Hutchinson A: Realist Synthesis: Method and Examples. The research process in nursing. research process in nursing. Edited by: Gerrish KL. 2010, A. Oxford, UK: Wiley Blackwell, 3

Google Scholar  

Pawson R: Evidence-based Policy. A Realist Perspective. 2006, London: Sage

Book   Google Scholar  

Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009, 9: 59-10.1186/1471-2288-9-59.

Article   PubMed   PubMed Central   Google Scholar  

Greenhalgh T, Kristjansson E, Robinson V: Realist review to understand the efficacy of school feeding programmes. Br Med J. 2007, 335: 858-861. 10.1136/bmj.39359.525174.AD.

Article   Google Scholar  

Wong G, Greenhalgh T, Pawson R: Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010, 10: http://www.biomedcentral.com/1472-6920/10/12

Pawson R, Owen L, Wong G: Known knowns, known unknowns, unknown unknowns: The predicament of evidence based policy. Am J Eval. 2011, 6:

Pawson R, Greenhalgh T, Harvey G, Walshe K: Realist synthesis: an introduction. RMP Methods Paper 2/2004. 2004, Manchester, UK: ESRC Research Methods Programme, University of Manchester

McCormack B, Wright J, Dewer B, Harvey G, Ballintine K: A realist synthesis of evidence relating to practice development: methodology and methods. Pract Dev Health Care. 2007, 6: 5-24. 10.1002/pdh.210.

Greenhalgh T, Wong G, Westhorp G, Pawson R: Protocol – realist and meta narrative evidence synthesis: Evolving standards (RAMESES). Protocol – realist and meta narrative evidence synthesis: Evolving standards (RAMESES) BMC Med Res Methodol. 2011, 11:

Delanty G: Social Science: beyond constructivism and realism. 1997, Buckingham: Open University Press

McEvoy P, Richards D: Critical realism: a way forward for evaluation research in nursing?. J Adv Nurs. 2003, 43: 411-420. 10.1046/j.1365-2648.2003.02730.x.

Wilson V, McCormack B: Critical realism as emancipatory action: the case for realistic evaluation in practice development. Nurs Philos. 2006, 7: 45-57. 10.1111/j.1466-769X.2006.00248.x.

Pawson R, Tilley N: Realistic Evaluation. 1997, London, England: Sage

Pawson R: Nothing as practical as a good theory. Evaluation. 2003, 9: 471-490. 10.1177/1356389003094007.

Thomas L, Cullum N, McColl E, Rousseau N, Soutter J, Steen N: Guidelines in professions allied to medicine (Cochrane Review). Cochrane Database Syst Rev. 2000, 2: CD000349-

PubMed   Google Scholar  

O'Brien MA, Freemantle N, Oxman AD, Wolf F, Davis DA, Herrin J: Continuing education meetings and workshops: Effects of professional practice and health care outcomes (Cochrane Review). Cochrane Database Syst Rev. 2001, 2: CD003030-

Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8: 1-72.

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA: Changing provider behaviour. An overview of systematic reviews of interventions. Med Care. 2001, 38: II-2-II-45.

Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thompson MA: Getting research findings into practice: closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. Br Med J. 1998, 317: 465-468. 10.1136/bmj.317.7156.465.

Article   CAS   Google Scholar  

Thompson DS, Estabrooks CA, Scott-Findlay S, Moore K, Wallin L: Interventions aimed at increasing research use in nursing: A systematic review. Implement Sci. 2007, 2: 15-10.1186/1748-5908-2-15.

Rycroft-Malone J, Seers K, Titchen A, Harvey G, Kitson A, McCormack B: What counts as evidence in evidence-based practice?. J Adv Nurs. 2004, 47: 81-90. 10.1111/j.1365-2648.2004.03068.x.

Leeman J, Chang YK, Lee EJ, Voils CI, Crandell J, Sandelowski M: Implementation of antiretroviral therapy adherence interventions: a realist synthesis of evidence. J Adv Nurs. 2010, 66: 1915-1930.

Walshe C, Luker KA: District nurses' role in palliative care provision: A realist review. Int J Nurs Stud. 2010, 47: 1167-1183. 10.1016/j.ijnurstu.2010.04.006.

Mazzocato P, Savage C, Brommels M, Aronsson H, Thor J: Lean thinking in healthcare: a realist review of the literature. Qual Saf Health Care. 2010, 19: 376-382. 10.1136/qshc.2009.037986.

O'Campo P, Kirst M, Tsamis C, Chambers C, Ahmad F: Implementing successful intimate partner violence screening programs in health care settings: Evidence generated from a realist-informed systematic review. Soc Sci Med. 2011, 72: 855-866. 10.1016/j.socscimed.2010.12.019.

Download references

Acknowledgements

Thanks to the Knowledge Utilisation Colloquium community for providing a critical commentary at various stages of this work.

Author information

Authors and affiliations.

School of Healthcare Sciences, Bangor University, Fron Heulog, Ffriddoedd Road, Bangor, UK

Jo Rycroft-Malone

Institute of Nursing Research/School of Nursing, University of Ulster, Shore Road, Newtownabbey, Co. Antrim, Northern Ireland

Brendan McCormack

School of Nursing and Midwifery, Deakin University, Burwood Highway, Melbourne, Australia

Alison M Hutchinson & Tracey K Bucknall

Cabrini-Deakin Centre for Nursing Research, Cabrini Health, Wattletree Road, Melbourne, Australia

School of Nursing, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada

Kara DeCorby

School of Nursing and Midwifery, Deakin University, Melbourne, Australia

Bridie Kent

Eastern Health Nursing Research Unit, 5 Arnold Street, Box Hill, Melbourne, Australia

EBP Concepts, Alyce A. Schultz & Associates, LLC, 5747 W Drake Court, Chandler, AZ, 85226, USA

Alyce Schultz

School of Nursing, Dalhousie University, 5869 University Avenue, Halifax, Canada

Erna Snelgrove-Clarke

Independent Consultant, Massachusetts, & Health Services Department, Boston University School of Public Health Boston, 321 Middle Street Amherst, Massachusetts, USA

Cheryl B Stetler

School of Nursing, University of Michigan, 400 North Ingalls Building, Ann Arbor, Michigan, MI, 48109-5482, USA

Marita Titler

Department of Neurobiology, Care Sciences and Society, Division of Nursing, Karolinska Institutet, Stockholm, Sweden

Lars Wallin

Faculty of Nursing, Midwifery and Health, University of Technology Sydney, Building 10, 235-253 Jones Street, Ultimo, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jo Rycroft-Malone .

Additional information

Competing interests.

Bridie Kent is an Associate Editor for Implementation Science; all decisions on this manuscript were made by other editors.

Authors’ contributions

BM and JRM led the project. All authors participated in defining the scope of the review. KD and ESC executed the search. The appraisal of evidence and data extraction were undertaken by all authors. All authors were involved in the analysis process. AMH led the documentation of the study process. KD led the development of the narrative. JRM wrote the first draft of the paper; BM, AH, and KD commented on it. All authors provided feedback on various drafts, and read and approved the final manuscript.

Electronic supplementary material

Additional file 1: terms related to realist synthesis. (doc 30 kb), additional file 2: review foci and questions. (doc 30 kb), additional file 3: data extraction form. (doc 68 kb), additional file 4: search terms and strategy. (doc 55 kb), additional file 5: stages of synthesis. (doc 48 kb), additional file 6: hypotheses linked to themes, chains of inference and papers. (doc 44 kb), authors’ original submitted files for images.

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Rycroft-Malone, J., McCormack, B., Hutchinson, A.M. et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci 7 , 33 (2012). https://doi.org/10.1186/1748-5908-7-33

Download citation

Received : 26 July 2011

Accepted : 05 April 2012

Published : 19 April 2012

DOI : https://doi.org/10.1186/1748-5908-7-33

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Knowledge Translation
• Change Agency
• Programme Theory
• Data Extraction Form
• Knowledge Utilisation

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

1. Introduction

2. results and discussion, 3. conclusions, 4. related literature, supporting information.

research papers \(\def\hfill{\hskip 5em}\def\hfil{\hskip 3em}\def\eqno#1{\hfil {#1}}\)

Analysis of COF-300 synthesis: probing degradation processes and 3D electron diffraction structure

a XStruct, Department of Chemistry, Ghent University, Krijgslaan 281–S3, 9000 Ghent, Belgium, b COMOC – Center for Ordered Materials, Organometallics and Catalysis – Department of Chemistry, Ghent University, Krijgslaan 281–S3, 9000 Ghent, Belgium, c Rigaku Corporation, Haijima, Tokyo, Japan, and d Rigaku Europe SE, Neu-Isenburg, Germany * Correspondence e-mail: [email protected]

Although COF-300 is often used as an example to study the synthesis and structure of (3D) covalent organic frameworks (COFs), knowledge of the underlying synthetic processes is still fragmented. Here, an optimized synthetic procedure based on a combination of linker protection and modulation was applied. Using this approach, the influence of time and temperature on the synthesis of COF-300 was studied. Synthesis times that were too short produced materials with limited crystallinity and porosity, lacking the typical pore flexibility associated with COF-300. On the other hand, synthesis times that were too long could be characterized by loss of crystallinity and pore order by degradation of the tetrakis(4-aminophenyl)methane (TAM) linker used. The presence of the degradation product was confirmed by visual inspection, Raman spectroscopy and X-ray photoelectron spectroscopy (XPS). As TAM is by far the most popular linker for the synthesis of 3D COFs, this degradation process might be one of the reasons why the development of 3D COFs is still lagging compared with 2D COFs. However, COF crystals obtained via an optimized procedure could be structurally probed using 3D electron diffraction (3DED). The 3DED analysis resulted in a full structure determination of COF-300 at atomic resolution with satisfying data parameters. Comparison of our 3DED-derived structural model with previously reported single-crystal X-ray diffraction data for this material, as well as parameters derived from the Cambridge Structural Database, demonstrates the high accuracy of the 3DED method for structure determination. This validation might accelerate the exploitation of 3DED as a structure determination technique for COFs and other porous materials.

Keywords: 3D electron diffraction ; 3DED ; microcrystal electron diffraction ; microED ; covalent organic frameworks ; Cambridge Structural Database ; porous organic solids ; crystallization and crystal growth .

CCDC reference: 2321626

In Figs. S6–S8, the effect of time on I 65°C (as discussed earlier) is compared with the effect on the other samples. Note that, as expected, I RT is slower to form a crystalline material compared with I 65°C due to the reduced error correction at room temperature, with reflections appearing after 1 d, and fully developed crystallinity after 5 d. However, the pore structure never fully establishes, as indicated by the broad, late and small second step in the N 2 -sorption isotherm. The appearance of crystallinity in C 65°C is even more delayed, with no crystalline reflections observed after 1 d of reaction time, indicating the superiority of the intermediate-assisted procedure. Here, maximal crystallinity is observed after 5 d, as peaks start to broaden significantly after 7 d. Surprisingly, the best N 2 -sorption behaviour was observed for the 7 d sample, indicating that the relationship between crystallinity and porosity is not always straightforward. Finally, using the conditions of C RT, we were unable to form any crystalline material, even after 7 d of reaction time. We also checked if the scale of the synthesis had any influence on the material. Therefore, a sample (I 65°C ×5) was prepared in an identical way to I 65°C but with every quantity used multiplied by 5. The resulting PXRD patterns and the N 2 -sorption isotherms are presented in Fig. S9 and show no significant influence on the crystallinity and a small decrease of porosity (with a BET surface area of 1180 m 2  g −1 and V p of 0.71 obtained for I 65°C ×5).

The response of COF-300 to an intermediate-assisted synthesis protocol was studied by careful evaluation of the evolution of both crystallinity and porosity as functions of reaction time and temperature. Kinetic studies among four different synthesis conditions revealed three distinct stages in the synthesis of COF-300, namely a network build-up phase at short synthetic times (≤1 d) with low crystallinity and no pore flexibility, followed by an optimal stage (3 d) characterized by high crystallinity and porosity before partial breakdown by TAM degradation (≥5 d). This degradation process could be confirmed in both control experiments as well as the obtained COF materials and can easily be estimated by the observation of magenta-coloured reaction mixtures. As a pronounced influence of this degradation reaction on both crystallinity and porosity was observed and most 3D COFs are based on the TAM linker, knowledge of TAM degradation in a acidic environment is of utmost importance for the synthesis of high-quality 3D COFs. Knowledge of this degradation process might help to increase the synthetic toolbox for 3D COFs (which are mainly based on the TAM linker), which is still lacking compared with 2D COFs. However, using the optimized conditions, a reliable crystal structure of COF-300 could be readily obtained via 3DED analysis, indicating single crystallinity of the synthesized materials. The structure model obtained showed high completeness and comparable resolution and R values. Comparison with an SCXRD structure model as well as with data for similar chemical functionalities in the CSD database showed no significant differences, supporting that 3DED is a reliable and fast technique for the structure solution of COFs. As SCXRD structure solution is hardly possible and PXRD models often show ambiguity in structure determination, 3DED might play an important role in the future of COFs with better accessibility of 3DED diffraction equipment and improving dynamic refinement algorithms.

Crystal structure: contains datablock 1. DOI: https://doi.org/10.1107/S2052252524003713/vq5005sup1.cif

Structure factors: contains datablock 1. DOI: https://doi.org/10.1107/S2052252524003713/vq5005sup2.hkl

Supporting Information - revised - highlighted. DOI: https://doi.org/10.1107/S2052252524003713/vq5005sup3.pdf

Acknowledgements

The authors thank Karen Leus for the XPS measurements and Dieter Buyst for the solid-state NMR measurements.

Funding information

LB acknowledges Ghent University (UGent) for funding. PVDV acknowledges financial support through UGent concerted action (grant No. 01G01017) and the Fonds Wetenschappelijk Onderzoek (FWO)–Vlaanderen project (grant Nos. 3G020521 awarded to PVDV; 1275221N awarded to SB and KVH). Gas sorption and powder X-ray diffraction were made possible through UGent (grant Nos. 01B00215; BOF20/BAS/015 awarded to PVDV). The spectrometer electronics, magnet and accessories used for solid-state NMR measurements, including the BBI and high-gradient diffusion probe, were funded by the Hercules foundation (grant No. AUGE/09/2006); the solid-state (CP-MAS) and HR-MAS expansion were made possible by FWO (grant No. I006920N).

This is an open-access article distributed under the terms of the Creative Commons Attribution (CC-BY) Licence , which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are cited.