search strategy of the literature review

Literature Review: Developing a search strategy

• Traditional or narrative literature reviews
• Scoping Reviews
• Systematic literature reviews
• Annotated bibliography
• Keeping up to date with literature
• Finding a thesis
• Evaluating sources and critical appraisal of literature
• Managing and analysing your literature
• Further reading and resources

From research question to search strategy

Keeping a record of your search activity

Good search practice could involve keeping a search diary or document detailing your search activities (Phelps et. al. 2007, pp. 128-149), so that you can keep track of effective search terms, or to help others to reproduce your steps and get the same results. 

This record could be a document, table or spreadsheet with:

• The names of the sources you search and which provider you accessed them through - eg Medline (Ovid), Web of Science (Thomson Reuters). You should also include any other literature sources you used.
• how you searched (keyword and/or subject headings)
• which search terms you used (which words and phrases)
• any search techniques you employed (truncation, adjacency, etc)
• how you combined your search terms (AND/OR). Check out the Database Help guide for more tips on Boolean Searching.
• The number of search results from each source and each strategy used. This can be the evidence you need to prove a gap in the literature, and confirms the importance of your research question.

A search planner may help you to organise you thoughts prior to conducting your search. If you have any problems with organising your thoughts prior, during and after searching please contact your Library  Faculty Team   for individual help.

• Literature search - a librarian's handout to introduce tools, terms and techniques Created by Elsevier librarian, Katy Kavanagh Web, this document outlines tools, terms and techniques to think about when conducting a literature search.
• Search planner

Literature search cycle

Diagram text description

This diagram illustrates the literature search cycle. It shows a circle in quarters. Top left quarter is identify main concepts with rectangle describing how to do this by identifying:controlled vocabulary terms, synonyms, keywords and spelling. Top right quarter select library resources to search and rectangle describing resources to search library catalogue relevant journal articles and other resource. Bottom right corner of circle search resources and in rectangle consider using boolean searching proximity searching and truncated searching techniques. Bottom left quarter of circle review and refine results. In rectangle evaluate results, rethink keywords and create alerts.

Have a search framework

Search frameworks are mnemonics which can help you focus your research question. They are also useful in helping you to identify the concepts and terms you will use in your literature search.

PICO is a search framework commonly used in the health sciences to focus clinical questions.  As an example, you work in an aged care facility and are interested in whether cranberry juice might help reduce the common occurrence of urinary tract infections.  The PICO framework would look like this:

Now that the issue has been broken up to its elements, it is easier to turn it into an answerable research question: “Does cranberry juice help reduce urinary tract infections in people living in aged care facilities?”

Other frameworks may be helpful, depending on your question and your field of interest. PICO can be adapted to PICOT (which adds T ime) or PICOS (which adds S tudy design), or PICOC (adding C ontext).

For qualitative questions you could use

• SPIDER : S ample,  P henomenon of  I nterest,  D esign,  E valuation,  R esearch type  

For questions about causes or risk,

• PEO : P opulation,  E xposure,  O utcomes

For evaluations of interventions or policies, 

• SPICE: S etting,  P opulation or  P erspective,  I ntervention,  C omparison,  E valuation or
• ECLIPSE: E xpectation,  C lient group,  L ocation,  I mpact,  P rofessionals,  SE rvice 

See the University of Notre Dame Australia’s examples of some of these frameworks. 

You can also try some PICO examples in the National Library of Medicine's PubMed training site: Using PICO to frame clinical questions.

Contact Your Faculty Team Librarian

Faculty librarians are here to provide assistance to students, researchers and academic staff by providing expert searching advice, research and curriculum support.

• Faculty of Arts & Education team
• Faculty of Business, Justice & Behavioural Science team
• Faculty of Science team

Further reading

• << Previous: Annotated bibliography
• Next: Keeping up to date with literature >>
• Last Updated: Jun 3, 2024 9:25 AM
• URL: https://libguides.csu.edu.au/review

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

205k Accesses

205 Citations

117 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to write a search strategy for your systematic review

Home | Blog | How To | How to write a search strategy for your systematic review

Practical tips to write a search strategy for your systematic review

With a great review question and a clear set of eligibility criteria already mapped out, it’s now time to plan the search strategy. The medical literature is vast. Your team plans a thorough and methodical search, but you also know that resources and interest in the project are finite. At this stage it might feel like you have a mountain to climb.

The bottom line? You will have to sift through some irrelevant search results to find the studies that you need for your review. Capturing a proportion of irrelevant records in your search is necessary to ensure that it identifies as many relevant records as possible. This is the trade-off of precision versus sensitivity and, because systematic reviews aim to be as comprehensive as possible, it is best to favour sensitivity – more is more.

By now, the size of this task might be sounding alarm bells. The good news is that a range of techniques and web-based tools can help to make searching more efficient and save you time. We’ll look at some of them as we walk through the four main steps of searching for studies:

• Decide where to search
• Write and refine the search
• Run and record the search
• Manage the search results

Searching is a specialist discipline and the information given here is not intended to replace the advice of a skilled professional. Before we look at each of the steps in turn, the most important systematic reviewer pro-tip for searching is:

 Pro Tip – Talk to your librarian and do it early!

1. decide where to search .

It’s important to come up with a comprehensive list of sources to search so that you don’t miss anything potentially relevant. In clinical medicine, your first stop will likely be the databases MEDLINE , Embase , and CENTRAL . Depending on the subject of the review, it might also be appropriate to run the search in databases that cover specific geographical regions or specialist areas, such as traditional Chinese medicine.

In addition to these databases, you’ll also search for grey literature (essentially, research that was not published in journals). That’s because your search of bibliographic databases will not find relevant information if it is part of, for example:

• a trials register
• a study that is ongoing
• a thesis or dissertation
• a conference abstract.

Over-reliance on published data introduces bias in favour of positive results. Studies with positive results are more likely to be submitted to journals, published in journals, and therefore indexed in databases. This is publication bias and systematic reviews seek to minimise its effects by searching for grey literature.

2. Write and refine the search 

Search terms are derived from key concepts in the review question and from the inclusion and exclusion criteria that are specified in the protocol or research plan.

Keywords will be searched for in the title or abstract of the records in the database. They are often truncated (for example, a search for therap* to find therapy, therapies, therapist). They might also use wildcards to allow for spelling variants and plurals (for example, wom#n to find woman and women). The symbols used to perform truncation and wildcard searches vary by database.

Index terms  

Using index terms such as MeSH and Emtree in a search can improve its performance. Indexers with subject area expertise work through databases and tag each record with subject terms from a prespecified controlled vocabulary.

This indexing can save review teams a lot of time that would otherwise be spent sifting through irrelevant records. Using index terms in your search, for example, can help you find the records that are actually about the topic of interest (tagged with the index term) but ignore those that contain only a brief mention of it (not tagged with the index term).

Indexers assign terms based on a careful read of each study, rather than whether or not the study contains certain words. So the index terms enable the retrieval of relevant records that cannot be captured by a simple search for the keyword or phrase.

Use a combination

Relying solely on index terms is not advisable. Doing so could miss a relevant record that for some reason (indexer’s judgment, time lag between a record being listed in a database and being indexed) has not been tagged with an index term that would enable you to retrieve it. Good search strategies include both index terms and keywords.

Let’s see how this works in a real review! Figure 2 shows the search strategy for the review ‘Wheat flour fortification with iron and other micronutrients for reducing anaemia and improving iron status in populations’. This strategy combines index terms and keywords using the Boolean operators AND, OR, and NOT. OR is used first to reach as many records as possible before AND and NOT are used to narrow them down.

• Lines 1 and 2: contain MeSH terms (denoted by the initial capitals and the slash at the end).
• Line 3: contains truncated keywords (‘tw’ in this context is an instruction to search the title and abstract fields of the record).
• Line 4: combines the three previous lines using Boolean OR to broaden the search.
• Line 11: combines previous lines using Boolean AND to narrow the search.
• Lines 12 and 13: further narrow the search using Boolean NOT to exclude records of studies with no human subjects.

Writing a search strategy is an iterative process. A good plan is  to try out a new strategy and check that it has picked up the key studies that you would expect it to find based on your existing knowledge of the topic area. If it hasn’t, you can explore the reasons for this, revise the strategy, check it for errors, and try it again!

3. Run and record the search

Because of the different ways that individual databases are structured and indexed, a separate search strategy is needed for each database. This adds complexity to the search process, and it is important to keep a careful record of each search strategy as you run it. Search strategies can often be saved in the databases themselves, but it is a good idea to keep an offline copy as a back-up; Covidence allows you to store your search strategies online in your review settings.

The reporting of the search will be included in the methods section of your review and should follow the PRISMA guidelines. You can download a flow diagram from PRISMA’s website to help you log the number of records retrieved from the search and the subsequent decisions about the inclusion or exclusion of studies. The PRISMA-S extension provides guidance on reporting literature searches.

It is very important that search strategies are reproduced in their entirety (preferably using copy and paste to avoid typos) as part of the published review so that they can be studied and replicated by other researchers. Search strategies are often made available as an appendix because they are long and might otherwise interrupt the flow of the text in the methods section.

4. Manage the search results 

Once the search is done and you have recorded the process in enough detail to write up a thorough description in the methods section, you will move on to screening the results. This is an exciting stage in any review because it’s the first glimpse of what the search strategies have found. A large volume of results may be daunting but your search is very likely to have captured some irrelevant studies because of its high sensitivity, as we have already seen. Fortunately, it will be possible to exclude many of these irrelevant studies at the screening stage on the basis of the title and abstract alone 😅.

Search results from multiple databases can be collated in a single spreadsheet for screening. To benefit from process efficiencies, time-saving and easy collaboration with your team, you can import search results into a specialist tool such as Covidence. A key benefit of Covidence is that you can track decisions made about the inclusion or exclusion of studies in a simple workflow and resolve conflicting decisions quickly and transparently. Covidence currently supports three formats for file imports of search results:

• EndNote XML
• PubMed text format
• RIS text format

If you’d like to try this feature of Covidence but don’t have any data yet, you can download some ready-made sample data .

And you’re done!

There is a lot to think about when planning a search strategy. With practice, expert help, and the right tools your team can complete the search process with confidence.

This blog post is part of the Covidence series on how to write a systematic review.

Sign up for a free trial of Covidence today!

[1] Witt  KG, Hetrick  SE, Rajaram  G, Hazell  P, Taylor Salisbury  TL, Townsend  E, Hawton  K. Pharmacological interventions for self‐harm in adults . Cochrane Database of Systematic Reviews 2020, Issue 12. Art. No.: CD013669. DOI: 10.1002/14651858.CD013669.pub2. Accessed 02 February 2021

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

Systematic Reviews: Search strategy

• Types of literature review, methods, & resources
• Protocol and registration
• Search strategy
• Medical Literature Databases to search
• Study selection and appraisal
• Data Extraction/Coding/Study characteristics/Results
• Reporting the quality/risk of bias
• Manage citations using RefWorks This link opens in a new window
• GW Box file storage for PDF's This link opens in a new window

Development of search strategy: PRISMA Item 6

Begin by defining your research question. You should identify who and what are the population, interventions, comparisons, outcomes, study design (PICOS) and study characteristics you are interested in, these are the beginning of your inclusion & exclusion criteria you will use to decide what articles you wish to include in your review. Watch this video from Rosalind Franklin University, on how to apply the PICO mnemonic to identify your search terms. Read these practice scenarios  with example PICO formatted answers in the box below each scenario. Read the article Munn et al (2018) Table 1 to determine what type of literature review you want to do, and check the adjacent column titled Question Format to see what type of PICO mnemonic you ought to apply.  Then work up an example PICO for your question.

Write down your keyword concepts, documenting them on a spreadsheet, you might like to start a Logic Grid following the guidance suggested by the University of Adelaide. Identify the MEDLINE MeSH headings  used for your keywords - try the Yale MeSH Analyzer  into which you can copy and paste up to 20 PMID numbers from search results in PubMed/MEDLINE and generate a MeSH Analysis grid to quickly scan the MeSH headings that were used to index those journal articles. Distribute the spreadsheet with your keywords and search terms to your research team colleagues and seek feedback from them and from your librarian. It is likely your spreadsheet will grow as you read around the topic and find new or related additional keywords and concepts.

SOURCES FOR SEARCH STRATEGIES

Someone may have already carried out a review on your topic and these can be a good source for finding a search strategy, some review guidelines such as PRISMA require the author to include an example of the search strategy for at least one database including the search keywords used, and in most cases this will be the MEDLINE search keywords and MeSH terms.  

• PubMed lists >4,500  reviews that have followed the PRISMA format  and in general these will include an example of at least one database search.
• Another source for search strategies is the PROSPERO database of review protocols, type keywords into the search box to see if there are any reviews similar to your topic.
• Search the Dissertations and Theses Online database for completed Doctoral research. The peer-review process for a dissertation is different than for a published journal article, and may not have been subjected to independent scrutiny, however  a good PhD dissertation or thesis should always include a literature review and this can be a good source for ideas of resources to search.
• Search MEDLINE or PubMed and use the "Publication type" limit for systematic review to limit the results to just this type of review article. Or  search  PubMed Clinical Queries  using simple keywords and look in the center column of results for a list of recent systematic reviews.
• The websites of the  McMaster University Health Information Research Unit , and the  University of York Centre for Reviews and Dissemination , may list a pre-existing evidence-based database search filter that can identify studies on your topic.

STUDY DESIGNS TO LOOK FOR

To help you think about what type of clinical studies you may want to include in your review read this description by staff at Duke/UNC medical libraries of  the best study types for the type of question you are investigating .  Think about what type of study design best collects the type of data you want to find and compare in your analysis e.g. if you want to analyse observational data what would be best: a time-series cohort study that measured data at several uniform time intervals, or a spatial data study that recorded the geographic locations where the observations were made, or a cross-sectional study that collected data at only one point in time such as a census? If you want to analyse the effect of an intervention or treatment what would be best: a randomized controlled trial, or would a non-randomized quasi-experimental study design be better such as an interrupted time-series where samples from the same population are taken before and after the intervention?  To make a safety case certain types of study design are important to control for potential biases and confounding variables, however when you come to do your literature search you may find that to answer your question certain types of study design were not possible for reasons including ethics, time, cost, etc.  If you can't find the data you're looking for in journal articles, might it exist in other public or private commercial sources? Try searching some of the supplementary data sources described on the next page of this guide such as clinical trials registers. Consider contacting authors you know have worked on this, identify them by doing an author search in Pubmed or an affilitation search in Scopus. Ask your professional colleagues or post questions to internet forums or listservs for professional societies and working groups for people working on this topic.  

HOW TO REPORT YOUR SEARCH STRATEGY

From 2021 authors should now follow the PRISMA-S checklist which is a 16-item guidance checklist on how to report the search strategies used in each of the databases searched for your systematic review. Optionally authors may wish to upload to an institutional repository or provide the publisher with supplemental files containing the search strategy as described in the guidance article Rethlefsen, M.L., Kirtley, S., Waffenschmidt, S.  et al.  PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews.  Syst Rev   10,  39 (2021). https://doi.org/10.1186/s13643-020-01542-z

In the final report in the methods section the PRISMA checklist Item 6 the PICOS representing the types of study you were looking for and study eligibility criteria should be reported as:

• Participants/population
• Interventions/treatment
• Comparison (if any)
• Outcome measures sought & length of follow up
• Study eligibility criteria (e.g. what types of study were sought RCTs, Case studies, etc. in what language, published or unpublished, year of publication, who commissioned the study, who carried it out, etc.)
• << Previous: Protocol and registration
• Next: Medical Literature Databases to search >>

• Last Updated: May 8, 2024 11:07 AM
• URL: https://guides.himmelfarb.gwu.edu/systematic_review

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

Literature Reviews: systematic searching at various levels

• for assignments
• for dissertations / theses
• Search strategy and searching
• Boolean Operators

Search strategy template

• Screening & critiquing
• Citation Searching
• Google Scholar (with Lean Library)
• Resources for literature reviews
• Adding a referencing style to EndNote
• Exporting from different databases
• PRISMA Flow Diagram
• Grey Literature

You can map out your search strategy in whatever way works for you.

Some people like lists and so plan their search strategy out in a grid-box or table format. Some people are more visual and like to draw their strategy out using a mind-map approach (either on paper or using mind-mapping software). Some people use sticky notes or Trello or a spreadsheet.

If it works for you then as long as it enables you to search systematically and thoroughly there's no need to change the way you work. 

If your search strategies are not very developed, the method you use doesn't lead to a good search, then consider using one of the other methods to see if changing your approach helps.

• Search Strategy Document
• << Previous: Boolean Operators
• Next: Screening & critiquing >>
• Last Updated: Apr 12, 2024 11:57 AM
• URL: https://libguides.derby.ac.uk/literature-reviews

How to undertake a literature search: a step-by-step guide

Affiliation.

• 1 Literature Search Specialist, Library and Archive Service, Royal College of Nursing, London.
• PMID: 32279549
• DOI: 10.12968/bjon.2020.29.7.431

Undertaking a literature search can be a daunting prospect. Breaking the exercise down into smaller steps will make the process more manageable. This article suggests 10 steps that will help readers complete this task, from identifying key concepts to choosing databases for the search and saving the results and search strategy. It discusses each of the steps in a little more detail, with examples and suggestions on where to get help. This structured approach will help readers obtain a more focused set of results and, ultimately, save time and effort.

Keywords: Databases; Literature review; Literature search; Reference management software; Research questions; Search strategy.

• Databases, Bibliographic*
• Information Storage and Retrieval / methods*
• Nursing Research
• Review Literature as Topic*

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

National Guideline Centre (UK). Evidence review for targets: Hypertension in adults: diagnosis and management: Evidence review D. London: National Institute for Health and Care Excellence (NICE); 2019 Aug. (NICE Guideline, No. 136.)

Evidence review for targets: Hypertension in adults: diagnosis and management: Evidence review D.

Appendix b literature search strategies.

The literature searches for this review are detailed below and complied with the methodology outlined in Developing NICE guidelines: the manual 2014, updated 2017 .

For more detailed information, please see the Methodology Review.

B.1. Clinical search literature search strategy

Searches were constructed using a PICO framework where population (P) terms were combined with Intervention (I) and in some cases Comparison (C) terms. Outcomes (O) are rarely used in search strategies for interventions as these concepts may not be well described in title, abstract or indexes and therefore difficult to retrieve. Search filters were applied to the search where appropriate.

Table 10 Database date parameters and filters used

View in own window

Table 11: Medline (Ovid) search terms

Table 12 embase (ovid) search terms, table 13 cochrane library (wiley) search terms, b.2. health economics literature search strategy.

Health economic evidence was identified by conducting a broad search relating hypertension in adults population in NHS Economic Evaluation Database (NHS EED – this ceased to be updated after March 2015) and the Health Technology Assessment database (HTA) with no date restrictions. NHS EED and HTA databases are hosted by the Centre for Research and Dissemination (CRD). Additional searches were run on Medline and Embase for health economics, economic modelling and quality of life studies.

Table 14 Database date parameters and filters used

Table 15 medline (ovid) search terms, table 16 embase (ovid) search terms, table 17 nhs eed and hta (crd) search terms.

• Cite this Page National Guideline Centre (UK). Evidence review for targets: Hypertension in adults: diagnosis and management: Evidence review D. London: National Institute for Health and Care Excellence (NICE); 2019 Aug. (NICE Guideline, No. 136.) Appendix B, Literature search strategies.
• PDF version of this title (1.3M)

In this Page

• Clinical search literature search strategy
• Health Economics literature search strategy

Other titles in this collection

• NICE Evidence Reviews Collection

Related NICE guidance and evidence

• NICE Guideline 136: Hypertension in adults: diagnosis and management

Supplemental NICE documents

• Methods (PDF)
• Cost-effectiveness analysis: Treatment initiation threshold for people with stage 1 hypertension (PDF)

Recent Activity

• Literature search strategies - Evidence review for targets Literature search strategies - Evidence review for targets

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Literature Reviews & Search Strategies

• Defining the Literature Review
• Types of Literature Reviews
• Choosing Databases

Overview of Search Strategies

Search strategies, subject searching, example: iteratively developing + using keywords, demonstration: developing keywords from a question, demonstration: an advanced search.

• Organizing Your Literature
• Books: Research Design & Scholarly Writing
• Recommended Tutorials

There are many ways to find literature for your review, and we recommend that you use a combination of strategies - keeping in mind that you're going to be searching multiple times in a variety of ways, using different databases and resources. Searching the literature is not a straightforward, linear process - it's iterative (translation: you'll search multiple times, modifying your strategies as you go, and sometimes it'll be frustrating). 

• Known Item Searching
• Citation Jumping

Some form of a keyword search is the way most of us get at scholarly articles in database - it's a great approach! Make sure you're familiar with these librarian strategies to get the most out of your searches.

Figuring out the best keywords for your research topic/question is a process - you'll start with one or a few words and then shift, adapt, and expand them as you start finding source that describe the topic using other words. Your search terms are the bridge between known topics and the unknowns of your research question - so sometimes one specific word will be enough, sometimes you'll need several different words to describe a concept AND you'll need to connect that concept to a second (and/or third) concept.

The number and specificity of your search terms depend on your topic and the scope of your literature review.

Connect Keywords Using Boolean

Make the database work more.

...uses the asterisk (*) to end a word at its core, allowing you to retrieve many more documents containing variations of the search term.  Example: educat* will find educate, educates, education, educators, educating and more.

Phrase Searching

...is when you put quotations marks around two or more words, so that the database looks for those words in that exact order. Examples: "higher education," "public health" and "pharmaceutical industry."

Controlled Vocabulary

... is when you use the terms the database uses to describe what each article is about as search terms. Searching using controlled vocabularies is a great way to get at everything on a topic in a database.  

Databases and search engines are probably going to bring back a lot of results - more than a human can realistically go through. Instead of trying to manually read and sort them all, use the filters in each database to remove the stuff you wouldn't use anyway (ie it's outside the scope of your project).

To make sure you're consistent between searches and databases, write down the filters you're using.

A Few Filters to Try

Once you know you have a good article , there are a lot of useful parts to it - far beyond the content.

Not sure where to start? Try course readings and other required materials.

Useful Parts of a Good Article

Ways to use citations.

• Interactive Tutorial: Searching Cited and Citing Practice starting your search at an article and using the references to gather additional sources.

Your search results don't have to be frozen in the moment you search! There are a few things you can set up to keep your search going automatically.

Searching using subject headings is a comprehensive search strategy that requires some planning and topic knowledge. Work through this PubMed tutorial for an introduction to this important approach to searching.

Through these videos and the accompanying PDF, you'll see an example of starting with a potential research question and developing search terms through brainstorming and keyword searching.

• Slidedeck: Keywords and Advanced Search PowerPoint slides to accompany the two demonstration videos on developing keywords from a question, and doing an advanced search.
• << Previous: Choosing Databases
• Next: Organizing Your Literature >>
• Last Updated: Jun 14, 2023 11:18 AM
• URL: https://mcphs.libguides.com/litreviews

• University of Michigan Library
• Research Guides

Systematic Reviews

• Search Strategy
• Work with a Search Expert
• Covidence Review Software
• Types of Reviews
• Evidence in a Systematic Review
• Information Sources

Developing an Answerable Question

Creating a search strategy, identifying synonyms & related terms, keywords vs. index terms, combining search terms using boolean operators, a sr search strategy, search limits.

• Managing Records
• Selection Process
• Data Collection Process
• Study Risk of Bias Assessment
• Reporting Results
• For Search Professionals

Validated Search Filters

Depending on your topic, you may be able to save time in constructing your search by using specific search filters (also called "hedges") developed & validated by researchers in the Health Information Research Unit (HiRU) of McMaster University, under contract from the National Library of Medicine.  These filters can be found on

• PubMed’s Clinical Queries &  Health Services Research Queries pages
• Ovid Medline’s Clinical Queries  filters or here
• Embase  & PsycINFO
• EBSCOhost’s main search page for CINAHL (Clinical Queries category)
• HiRU’s Nephrology Filters page
• American U of Beirut, esp. for " humans" filters .
• Countway Library of Medicine methodology filters
• InterTASC Information Specialists' Sub-Group Search Filter Resource
• SIGN (Scottish Intercollegiate Guidelines Network) filters page

Why Create a Sensitive Search?

In many literature reviews, you try to balance the sensitivity of the search (how many potentially relevant articles you find) &  specificit y (how many definitely relevant articles  you find ), realizing that you will miss some.  In a systematic review, you want a very sensitive search:  you are trying to find any potentially relevant article.  A systematic review search will:

• contain many synonyms & variants of search terms
• use care in adding search filters
• search multiple resources, databases & grey literature, such as reports & clinical trials

PICO is a good framework to help clarify your systematic review question.

P -   Patient, Population or Problem: What are the important characteristics of the patients &/or problem?

I -  Intervention:  What you plan to do for the patient or problem?

C -  Comparison: What, if anything, is the alternative to the intervention?

O -  Outcome:  What is the outcome that you would like to measure?

Beyond PICO: the SPIDER tool for qualitative evidence synthesis.

5-SPICE: the application of an original framework for community health worker program design, quality improvement and research agenda setting.

A well constructed search strategy is the core of your systematic review and will be reported on in the methods section of your paper. The search strategy retrieves the majority of the studies you will assess for eligibility & inclusion. The quality of the search strategy also affects what items may have been missed.  Informationists can be partners in this process.

For a systematic review, it is important to broaden your search to maximize the retrieval of relevant results.

Use keywords:  How other people might describe a topic?

Identify the appropriate index terms (subject headings) for your topic.

• Index terms differ by database (MeSH, or  Medical Subject Headings ,   Emtree terms , Subject headings) are assigned by experts based on the article's content.
• Check the indexing of sentinel articles (3-6 articles that are fundamental to your topic).  Sentinel articles can also be used to  test your search results.

Include spelling variations (e.g., behavior, behaviour ).  

Both types of  search terms are useful & both should be used in your search.

Keywords help to broaden your results.  They will be searched for at least in journal titles, author names, article titles, & article abstracts.  They can also be tagged to search all text.

Index/subject terms  help to focus your search appropriately, looking for items that have had a specific term applied by an indexer.

Boolean operators let you combine search terms in specific ways to broaden or narrow your results.

An example of a search string for one concept in a systematic review.

In this example from a PubMed search, [mh] = MeSH &  [tiab] = Title/Abstract, a more focused version of a keyword search.

A typical database search limit allows you to narrow results so that you retrieve articles that are most relevant to your research question. Limit types vary by database & include:

• Article/publication type
• Publication dates

In a systematic review search, you should use care when applying limits, as you may lose articles inadvertently.  For more information, see, particularly regarding language & format limits.     Cochrane 2008 6.4.9

• Interlibrary Loan

Ask an Expert

Ask an expert about access to resources, publishing, grants, and more.

MD Anderson faculty and staff can also request a one-on-one consultation with a librarian or scientific editor.

• Library Calendar

Log in to the Library's remote access system using your MyID account.

• Library Home
• Research Guides

Literature Search Basics

Develop a search strategy.

• Define your search
• Decide where to search

What is a search strategy

Advanced search tips.

• Track and save your search
• Class Recording: Writing an Effective Narrative Review
• A search strategy includes  a combination of keywords, subject headings, and limiters (language, date, publication type, etc.)
• A search strategy should be planned out and practiced before executing the final search in a database.
• A search strategy and search results should be documented throughout the searching process.

What is a search strategy?

A search strategy is an organized combination of keywords, phrases, subject headings, and limiters used to search a database.

Your search strategy will include:

• keywords 
• boolean operators
• variations of search terms (synonyms, suffixes)
• subject headings 

Your search strategy  may  include:

• truncation (where applicable)
• phrases (where applicable)
• limiters (date, language, age, publication type, etc.)

A search strategy usually requires several iterations. You will need to test the strategy along the way to ensure that you are finding relevant articles. It's also a good idea to review your search strategy with your co-authors. They may have ideas about terms or concepts you may have missed.

Additionally, each database you search is developed differently. You will need to adjust your strategy for each database your search.  For instance, Embase is a European database, many of the medical terms are slightly different than those used in MEDLINE and PubMed.

Choose search terms

Start by writing down as many terms as you can think of that relate to your question. You might try  cited reference searching  to find a few good articles that you can review for relevant terms.

Remember than most terms or  concepts can be expressed in different ways.  A few things to consider:

• synonyms: "cancer" may be referred to as "neoplasms", "tumors", or "malignancy"
• abbreviations: spell out the word instead of abbreviating
• generic vs. trade names of drugs

Search for the exact phrase

If you want words to appear next to each other in an exact phrase, use quotation marks, eg “self-esteem”.

Phrase searching decreases the number of results you get. Most databases allow you to search for phrases, but check the database guide if you are unsure.

Truncation and wildcards

Many databases use an asterisk (*) as their truncation symbol  to find various word endings like singulars and plurals.  Check the database help section if you are not sure which symbol to use. 

"Therap*"

retrieves: therapy, therapies, therapist or therapists.

Use a wildcard (?) to find different spellings like British and American spellings.

"Behavio?r" retrieves behaviour and behavior.

Searching with subject headings

Database subject headings are controlled vocabulary terms that a database uses to describe what an article is about.

Using appropriate subject headings enhances your search and will help you to find more results on your topic. This is because subject headings find articles according to their subject, even if the article does not use your chosen key words.

You should combine both subject headings and keywords in your search strategy for each of the concepts you identify. This is particularly important if you are undertaking a systematic review or an in-depth piece of work

Subject headings may vary between databases, so you need to investigate each database separately to find the subject headings they use. For example, for MEDLINE you can use MeSH (Medical Subject Headings) and for Embase you can use the EMTREE thesaurus.

SEARCH TIP:  In Ovid databases, search for a known key paper by title, select the "complete reference" button to see which subject headings the database indexers have given that article, and consider adding relevant ones to your own search strategy.

Use Boolean logic to combine search terms

Boolean operators (AND, OR and NOT) allow you to try different combinations of search terms or subject headings.

Databases often show Boolean operators as buttons or drop-down menus that you can click to combine your search terms or results.

The main Boolean operators are:

OR is used to find articles that mention  either  of the topics you search for.

AND is used to find articles that mention  both  of the searched topics.

NOT excludes a search term or concept. It should be used with caution as you may inadvertently exclude relevant references.

For example, searching for “self-esteem NOT eating disorders” finds articles that mention self-esteem but removes any articles that mention eating disorders.

Adjacency searching 

Use adjacency operators to search by phrase or with two or more words in relation to one another. A djacency searching commands differ among databases. Check the database help section if you are not sure which searching commands to use. 

In Ovid Medline

"breast ADJ3 cancer" finds the word breast within three words of cancer, in any order.

This includes breast cancer or cancer of the breast.

Cited Reference Searching

Cited reference searching is a method to find articles that have been cited by other publications. 

Use cited reference searching to:

• find keywords or terms you may need to include in your search strategy
• find pivotal papers the same or similar subject area
• find pivotal authors in the same or similar subject area
• track how a topic has developed over time

Cited reference searching is available through these tools:

• Web of Science
• GoogleScholar
• << Previous: Decide where to search
• Next: Track and save your search >>
• Last Updated: Nov 29, 2022 3:34 PM
• URL: https://mdanderson.libguides.com/literaturesearchbasics

A Guide to Evidence Synthesis: 4. Write a Search Strategy

• Meet Our Team
• Our Published Reviews and Protocols
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 0. Develop a Protocol
• 1. Draft your Research Question
• 2. Select Databases
• 3. Select Grey Literature Sources
• 4. Write a Search Strategy
• 5. Register a Protocol
• 6. Translate Search Strategies
• 7. Citation Management
• 8. Article Screening
• 9. Risk of Bias Assessment
• 10. Data Extraction
• 11. Synthesize, Map, or Describe the Results
• Evidence Synthesis Institute for Librarians
• Open Access Evidence Synthesis Resources

Video: Databases and search strategies (3:40 minutes)

Writing a Search Strategy

It is recommended that you work with a librarian to help you design comprehensive search strategies across a variety of databases. Writing a successful search strategy takes an intimate knowledge of bibliographic databases.  

Using Boolean logic is an important component of writing a search strategy: 

• "AND" narrows the search, e.g.  children AND exercise
• "OR" broadens the search, e.g.  (children OR adolescents) AND (exercise OR diet) 
• "NOT" excludes terms, e.g.  exercise NOT diet 
• "*" at the root of a word finds all forms of that word, e.g.  (child* OR adolescen*) AND (exercise* OR diet*)
• parentheses ensure all terms will be searched together as a set 
• quotations around a phrase searches that exact phrase, e.g.  (child* OR adolescen* OR "young adult*") 

Evidence Synthesis Search Strategy Examples

Agriculture example: .

• Research question:  What are the strategies that farmer organizations use, and what impacts do those strategies have on small-scale producers in Sub Saharan Africa and India? 
• Key concepts from the question combined with AND:  (farmer organizations) AND (Sub-Saharan Africa OR India) 
• Protocol and search strategies for this question in CAB Abstracts, Scopus, EconLit, and grey literature
• Published scoping review for this question

Nutrition Example: 

• Research question:  What are the health benefits and safety of folic acid fortification of wheat and maize flour (i.e. alone or in combination with other micronutrients) on folate status and health outcomes in the overall population, compared to wheat or maize flour without folic acid (or no intervention)? 
• Key concepts from the question combined with AND:  (folic acid) AND (fortification) 
• Protocol on PROSPERO
• Published systematic review for this question with search strategies used in 14 databases

Search Strategy Template and Filters

• Human Studies Filter
• Randomized Controlled Trial Filters
• Other Methodology Search Filters

If you want to exclude animal studies from your search results, you may add a "human studies filter" to the end of your search strategy. This approach works best with databases that use Medical Subject Headings (MeSH) or other controlled vocabulary. You can see an example of how this was used in the MEDLINE(Ovid) search strategy of this published review (lines 13-14).

A simplified explanation of this filter can be seen below:

Add the following lines to the end of your search strategy to filter for randomized controlled trials. These are "validated search filters" meaning they have been tested for sensitivity and specificity, and the results of those tests have been published as a scientific article. The ISSG Search Filters Resource provides validated search filters for many other study design types. 

Highly Sensitive MEDLINE (via PubMed) Filter from Cochrane  

(randomized controlled trial [pt] OR controlled clinical trial [pt] OR randomized [tiab] OR placebo [tiab] OR drug therapy [sh] OR randomly [tiab] OR trial [tiab] OR groups [tiab])

Highly Sensitive MEDLINE (OVID) Filter from Cochrane 

((randomized controlled trial.pt. or controlled clinical trial.pt. or randomized.ab. or placebo.ab. or drug therapy.fs. or randomly.ab. or trial.ab. or groups.ab.) not (exp animals/ not humans.sh.)) ​

CINAHL Filter from Cochrane 

TX allocat* random* OR (MH "Quantitative Studies") OR (MH "Placebos") OR TX placebo* OR TX random* allocat* OR (MH "Random Assignment") OR TX randomi* control* trial* OR TX ( (singl* n1 blind*) OR (singl* n1 mask*) ) OR TX ( (doubl* n1 blind*) OR (doubl* n1 mask*) ) OR TX ( (tripl* n1 blind*) OR (tripl* n1 mask*) ) OR TX ( (trebl* n1 blind*) OR (trebl* n1 mask*) ) OR TX clinic* n1 trial* OR PT Clinical trial OR (MH "Clinical Trials+")

PsycINFO Filter from ProQuest:

SU.EXACT("Treatment Effectiveness Evaluation") OR SU.EXACT.EXPLODE("Treatment Outcomes") OR SU.EXACT("Placebo") OR SU.EXACT("Followup Studies") OR placebo* OR random* OR "comparative stud*" OR  clinical NEAR/3 trial* OR research NEAR/3 design OR evaluat* NEAR/3 stud* OR prospectiv* NEAR/3 stud* OR (singl* OR doubl* OR trebl* OR tripl*) NEAR/3 (blind* OR mask*)

Web Of Science (WoS) Filter from University of Alberta - Not Validated

TS= clinical trial* OR TS=research design OR TS=comparative stud* OR TS=evaluation stud* OR TS=controlled trial* OR TS=follow-up stud* OR TS=prospective stud* OR TS=random* OR TS=placebo* OR TS=(single blind*) OR TS=(double blind*)

Scopus Filter from Children's Mercy Kansas City

TITLE-ABS-KEY((clinic* w/1 trial*) OR (randomi* w/1 control*) OR (randomi* w/2 trial*) OR (random* w/1 assign*) OR (random* w/1 allocat*) OR (control* w/1 clinic*) OR (control* w/1 trial) OR placebo* OR (Quantitat* w/1 Stud*) OR (control* w/1 stud*) OR (randomi* w/1 stud*) OR (singl* w/1 blind*) or (singl* w/1 mask*) OR (doubl* w/1 blind*) OR (doubl* w/1 mask*) OR (tripl* w/1 blind*) OR (tripl* w/1 mask*) OR (trebl* w/1 blind*) OR (trebl* w/1 mask*)) AND NOT (SRCTYPE(b) OR SRCTYPE(k) OR SRCTYPE(p) OR SRCTYPE(r) OR SRCTYPE(d) OR DOCTYPE(ab) OR DOCTYPE(bk) OR DOCTYPE(ch) OR DOCTYPE(bz) OR DOCTYPE(cr) OR DOCTYPE(ed) OR DOCTYPE(er) OR DOCTYPE(le) OR DOCTYPE(no) OR DOCTYPE(pr) OR DOCTYPE(rp) OR DOCTYPE(re) OR DOCTYPE(sh))

Sources and more information:

• Cochrane Handbook for Systematic Reviews of Interventions
• Cochrane RCT Filters for Different Databases
• American University of Beirut University Libraries Search Filters / Hedges
• Methodology Search Filters by Study Design - Countway Library of Medicine, Harvard University Filters for RCTs, CCTs, Non-randomized/observational designs, and tests of diagnostic accuracy.
• Search Filters - American University of Beirut University Libraries Filters for RCTs, GUIDELINEs, systematic reviews, qualitative studies, etc.

Pre-generated queries in Scopus for the UN Sustainable Development Goals

Pre-written SDG search strategies available in Scopus 

Scopus, a multidisciplinary research database, provides pre-written search strategies to capture articles on topics about each of the 17 United Nations Sustainable Development Goals  (SDGs). These search strategies were updated in 2023 and are no longer available directly on "Advanced Document Search". To use these SDG search strategies:

• Go to the Elsevier 2023 Sustainable Development Goals (SDGs) Mapping page.
• Under Files , click on the SDG 2023 Queries folder.
• Download the .txt file for each pre-written search strategy you are interested in. You will need to know the number of the SDG of interest (e.g., SDG01.txt is for SDG 1: No Poverty). This .txt file will contain the entire search string for the SDG, already written in Scopus syntax .
• In Scopus , click on "Advanced Document Search".
• Copy and paste the pre-written SDG search strategy into the search field.

More about the Sustainable Development Goals: 

" The 2030 Agenda for Sustainable Development,  adopted by all United Nations Member States in 2015, provides a shared blueprint for peace and prosperity for people and the planet, now and into the future. At its heart are the 17 Sustainable Development Goals (SDGs), which are an urgent call for action by all countries - developed and developing - in a global partnership. They recognize that ending poverty and other deprivations must go hand-in-hand with strategies that improve health and education, reduce inequality, and spur economic growth – all while tackling climate change and working to preserve our oceans and forests."

Source:  https://sdgs.un.org/goals 

• << Previous: 3. Select Grey Literature Sources
• Next: 5. Register a Protocol >>
• Last Updated: Jun 6, 2024 3:35 PM
• URL: https://guides.library.cornell.edu/evidence-synthesis

Systematic Literature Review

• What is a Systematic Literature Review?
• Doing a Systematic Literature Review
• Developing a Search Strategy
• Useful Links and Resources

Searching for information

The Library Catalogue is good place to start looking for information. 

Many of the resources described here can be found in the Library. Follow the link below to begin your search

TU Dublin Library Catalogue

Developing a search strategy

• Search Strategy
• Types of information resources
• Where to find your information
• Getting the best out of your search
• Subject headings

Researchers conducting a  systematic literature review need to perform comprehensive searches to ensure they have retrieved all of the relevant information. Below is an overview of the steps involved in conducting a search for literature. For further information on conducting a comprehensive search, please see the Cochrane handbook .

Scope the topic. this will help to contextualise your topic within the broader context of your subject and also indicate the volume of literature available on your subject.  it also gives an opportunity to refine your search strategy to ensure an effective and accurate search..

Inclusion/Exclusion criteria  is used to identify the specific attributes of material that you want to include in the review.  For example, the type of study or population, etc.

Identify the concepts in your research question.  You can use a framework to help with this, such as PICO , SPIDER or another suitable method.

Identify keywords, synonyms and alternative keywords.  You can develop your synonyms and alternative keywords using relevant articles or papers that you are already aware of.  You will also need to use subject headings to ensure you do a comprehensive search (see subject headings tab). 

Combine searches using Boolean operators, OR and AND. Use OR to combine the keywords and subject headings, and use AND to retrieve material that includes all the concepts in your research question. See the  getting the best out of your searching  tab  for further details on truncation, phrased searching, etc. The idea of a search strategy is that it can be used across multiple sources, however, you may need to re-map your keywords to the subject headings of different databases.

Apply search filters.  Many databases allow researchers to limit search results through the use of filters or hedges. For example results can be limited by publication type, study type, etc.  However, these in-built filters may exclude relevant studies.  For this reason, and to maintain consistency across databases, researchers conducting systematic literature reviews use standard filters that can be added to searches using the AND operator. The filters are re-useable and shared widely.

• The InterTASC Information Specialists' Sub Group Search Filter Resources (ISSG)
• BMJ Clinical Evidence Study Design Search Filters

Sensitivity versus precision involves balancing your strategy between retrieving a large number of documents that might include some irrelevant material, against are a more precise strategy that might miss relevant material.

Review your strategy.   Make sure that your results are relevant, identify any keyword/concepts from your strategy that don't appear in your results.  Check retrieved material for any relevant keywords or subject headings that you may have omitted.

Document your search.   You need to record the details of your searches to ensure transparency and consistency. You can use a table or spreadsheet to do this. Include;

• search terms
• database/catalogue searched
• refinements or limitations (if used)
• date searched
• basic/advanced search
• number of records (hits)
• articles retained
• articles you can't access​

Manage your results. Use bibliographic management software (Endnote, Mendeley or Zotero) to remove duplicate records. 

Select material. Use the agreed criteria to select material.  This is done in stages, initially by screening the title, then the abstract and finally by full text.  This should be done by more than one person, to help reduce bias in the process.  Having more than one person involved also means that it can be easier to resolve queries regarding selection of individual papers, should a query arise.

Move on to the data extraction phase.   For further information see chapter 5 of the Cochrane handbook .

For further information on conducting searches for SLR, please see the following useful links:​

• Search Strategy for SLR (NUI Galway)
• Search Strategy for SLR (UCD)

There are many different types of information resources.  Students and researchers can use a combination of resources when completing assignments or research. Some of the most widely used sources are described below.

Textbooks  that you find in the library are usually written by experts.  Textbooks are structured for ease of use; they begin with simpler aspects of a topic before progressing towards more complicated aspects. Books often have useful glossaries and an index of subjects and authors.  They also include references and bibliographies, which can help to expand your search.

Conference Proceedings  can be very important for researchers because often, the first time research is published, is at a conference. Proceedings provide access to specialist and focused information.

Journals  are published at regular intervals, such as monthly, bi-monthly or even annually.  This means that journals can be a good source of up to date information.  They can be written by either academics, specialist researchers or professional practitioners.  Some journals are peer-reviewed, this means any material published in the journal has undergone an evaluation process to ensure the quality and validity of the information.  Journals can be highly focused on a specific subject and also include references.

Official Publications  are published by governments and government departments all over the world.  They can be a useful source of information on areas such as legal, education, finance, science, health and social policies.

Reference Material  includes subject directories, almanacs and encyclopedia.  They can provide useful definitions and can be a good sources of primary data (statistics, speeches, diaries). There is no evaluation or interpretation of the material provided, which allows the student or researcher to use the data for their own purpose and to draw their conclusions.

Standards  are an agreed set of procedures, processes or technical specifications that provide guidance across multiple disciplines and industries.

Social media,  such as blogs and twitter feeds can highlight key topics and discussions that are current and fast moving. Social media has the potential to help researchers to stay up to date with developments in different disciplines.

Theses ​  are final year research projects submitted by students completing degree and doctorate level qualifications. Theses are a valuable information source for researchers as they are highly focused on a particular topic, They will also contain a detailed literature review section, and have references and a bibliography.

Grey literature describes material and research that is not published by academic or commercial publishers.  It is often produced by professional bodies or organisations.  For example, theses, annual reports, technical guidelines, conference posters or government papers.

Other resources

​It is important to be aware of the many different sources of information that exist, so you can select the most suitable ones for your project or research. Below is a list of some of the most popular sources.

Library Catalogues  provide access to a combination of print and digital material and often cover multiple disciplines. Most library catalogues can be searched by anyone, although access to material in other institutions is subject to certain permissions.

Institutional repositories  provide access to the research outputs of individual academic institutions.  This includes PhD and other research theses.  OpenDOAR allows users to find repositories.  RIAN allows user to search all Irish repositories, including Arrow@TuDublin .

Databases​​ ​ ​  contain many different types of information resources, such as journal articles, books, conference proceedings, newspapers and reports.  The databases that are available through the library provide access to scholarly, and frequently, peer-reviewed material.  Academic databases are designed to help researchers, so many have advanced search features to facilitate accurate and focused searches.

Search Engines​​  are useful resources discovery tools.  They contain links to and information on a wide variety of subjects in various formats.  However, there is very little, if any, oversight or review of the material.  So any information taken from search engines must be thoroughly evaluated.  Although results may be largely relevance-based, some search engines manipulate how results are displayed for commercial purpose, so there is a chance that you are not seeing the most relevant information at the top of the results page. N.B. If you are doing a systematic literature review using Google, be sure to turn off personalisation by deleting your browser history, logging out of your account and removing all Google services through the "My Activity" page.  Otherwise, your results may be influenced by the search history and this may impact the relevance of your results.  Consider using other specialist search engines, such as BASE .

Subject Directories  provide access to curated, and sometimes annotated, subject resources.  Sites included in the directory are usually selected by an administrator according to set criteria.  The directory is navigated by clicking through options rather than a search box.  Directories contain less information than a search engine, but they can be useful for specific subject areas.  See the subject guide in your area for further information.

OpenGrey  is the system for information on grey literature in Europe and provides access to 700,000 bibliographical references of grey literature produced in Europe.

LENUS  provides access to the research output of many healthcare organisations in the Republic of Ireland.

Other Sources  available to researchers include government & EU websites, NGOs and other voluntary and professional organisations.

• Phrased searching

The default Boolean term for a database or search engine (unless otherwise stated) is AND.  For example, a search for -information technology- will be interpreted as information AND technology.  However, if you want to search for information on the topic - information technology, put inverted commas around both words, "information technology".  Phrased searching should decrease the number of results you retrieve but increase the relevance of those results.

• Wildcard & Truncation

Wildcards (?/#) allow you to search for the American and British spellings of words.  For example, a search for the word behavio?r, will include both behaviour and behavior. You can also use # to include plurals of words like man or men by using m#n.

Truncation (*) allows you to include variations of a word.  For example, teach*, will include teachers, teacher, teaching and teach.

• Subject descriptors/headings
• Snowballing
• Citation searching
• Hand-searching

​Involves searching for relevant material that may have been missed by databases.  For further guidelines on hand-searching see the Cochrane handbook .

Subject headings are assigned by an administrator when a document is added to the database.  The headings provide a consistent description of the subject content of the document. They are also known as descriptors or categories.  They are similar to a hashtag but are added by the publisher, not the user.  Subject headings are selected from a controlled vocabulary (a list of agreed or standard terms), which is maintained and updated by an administrator.

There are different ways of locating subject headings.  Some databases allow access through a list of subject headings, descriptors, categories or provide a searchable thesaurus.  The thesaurus will allow you to enter your keyword into the search box.  The list of results should provide the subject heading closest to your keyword, and in some databases, you will be shown broader and narrower terms associated with your topic.  Alternatively, you could do a standard keyword search and identify the subject headings or descriptors assigned to one or two relevant documents.

One of the most well-known lists of subject headings is the Medical Subject Headings or  MeSH , used to search MEDLINE/PubMed or the National Library of Medicine.

• << Previous: Doing a Systematic Literature Review
• Next: Useful Links and Resources >>
• Last Updated: Jan 10, 2024 6:50 PM
• URL: https://tudublin.libguides.com/systematicliteraturereview

• Study and research support
• Literature searching

Literature searching explained

Develop a search strategy.

A search strategy is an organised structure of key terms used to search a database. The search strategy combines the key concepts of your search question in order to retrieve accurate results.

Your search strategy will account for all:

• possible search terms
• keywords and phrases
• truncated and wildcard variations of search terms
• subject headings (where applicable)

Each database works differently so you need to adapt your search strategy for each database. You may wish to develop a number of separate search strategies if your research covers several different areas.

It is a good idea to test your strategies and refine them after you have reviewed the search results.

How a search strategy looks in practice

Take a look at this example literature search in PsycINFO (PDF) about self-esteem.

The example shows the subject heading and keyword searches that have been carried out for each concept within our research question and how they have been combined using Boolean operators. It also shows where keyword techniques like truncation, wildcards and adjacency searching have been used.

Search strategy techniques

The next sections show some techniques you can use to develop your search strategy.

Skip straight to:

• Choosing search terms
• Searching with keywords
• Searching for exact phrases
• Using truncated and wildcard searches

Searching with subject headings

• Using Boolean logic

Citation searching

Choose search terms.

Concepts can be expressed in different ways eg “self-esteem” might be referred to as “self-worth”. Your aim is to consider each of your concepts and come up with a list of the different ways they could be expressed.

To find alternative keywords or phrases for your concepts try the following:

• Use a thesaurus to identify synonyms.
• Search for your concepts on a search engine like Google Scholar, scanning the results for alternative words and phrases.
• Examine relevant abstracts or articles for alternative words, phrases and subject headings (if the database uses subject headings).

When you've done this, you should have lists of words and phrases for each concept as in this completed PICO model (PDF) or this example concept map (PDF).

As you search and scan articles and abstracts, you may discover different key terms to enhance your search strategy.

Using truncation and wildcards can save you time and effort by finding alternative keywords.

Search with keywords

Keywords are free text words and phrases. Database search strategies use a combination of free text and subject headings (where applicable).

A keyword search usually looks for your search terms in the title and abstract of a reference. You may wish to search in title fields only if you want a small number of specific results.

Some databases will find the exact word or phrase, so make sure your spelling is accurate or you will miss references.

Search for the exact phrase

If you want words to appear next to each other in an exact phrase, use quotation marks, eg “self-esteem”.

Phrase searching decreases the number of results you get and makes your results more relevant. Most databases allow you to search for phrases, but check the database guide if you are unsure.

Truncation and wildcard searches

You can use truncated and wildcard searches to find variations of your search term. Truncation is useful for finding singular and plural forms of words and variant endings.

Many databases use an asterisk (*) as their truncation symbol. Check the database help section if you are not sure which symbol to use. For example, “therap*” will find therapy, therapies, therapist or therapists. A wildcard finds variant spellings of words. Use it to search for a single character, or no character.

Check the database help section to see which symbol to use as a wildcard.

Wildcards are useful for finding British and American spellings, for example: “behavio?r” in Medline will find both behaviour and behavior.

There are sometimes different symbols to find a variable single character. For example, in the Medline database, “wom#n” will find woman and also women.

Use adjacency searching for more accurate results

You can specify how close two words appear together in your search strategy. This can make your results more relevant; generally the closer two words appear to each other, the closer the relationship is between them.

Commands for adjacency searching differ among databases, so make sure you consult database guides.

In OvidSP databases (like Medline), searching for “physician ADJ3 relationship” will find both physician and relationship within two major words of each other, in any order. This finds more papers than "physician relationship".

Using this adjacency retrieves papers with phrases like "physician patient relationship", "patient physician relationship", "relationship of the physician to the patient" and so on.

Database subject headings are controlled vocabulary terms that a database uses to describe what an article is about.

Watch our 3-minute introduction to subject headings video . You can also  View the video using Microsoft Stream (link opens in a new window, available for University members only).

Using appropriate subject headings enhances your search and will help you to find more results on your topic. This is because subject headings find articles according to their subject, even if the article does not use your chosen key words.

You should combine both subject headings and keywords in your search strategy for each of the concepts you identify. This is particularly important if you are undertaking a systematic review or an in-depth piece of work

Subject headings may vary between databases, so you need to investigate each database separately to find the subject headings they use. For example, for Medline you can use MeSH (Medical Subject Headings) and for Embase you can use the EMTREE thesaurus.

SEARCH TIP: In Ovid databases, search for a known key paper by title, select the "complete reference" button to see which subject headings the database indexers have given that article, and consider adding relevant ones to your own search strategy.

Use Boolean logic to combine search terms

Boolean operators (AND, OR and NOT) allow you to try different combinations of search terms or subject headings.

Databases often show Boolean operators as buttons or drop-down menus that you can click to combine your search terms or results.

The main Boolean operators are:

OR is used to find articles that mention either of the topics you search for.

AND is used to find articles that mention both of the searched topics.

NOT excludes a search term or concept. It should be used with caution as you may inadvertently exclude relevant references.

For example, searching for “self-esteem NOT eating disorders” finds articles that mention self-esteem but removes any articles that mention eating disorders.

Citation searching is a method to find articles that have been cited by other publications.

Use citation searching (or cited reference searching) to:

• find out whether articles have been cited by other authors
• find more recent papers on the same or similar subject
• discover how a known idea or innovation has been confirmed, applied, improved, extended, or corrected
• help make your literature review more comprehensive.

You can use cited reference searching in:

• OvidSP databases
• Google Scholar
• Web of Science

Cited reference searching can complement your literature search. However be careful not to just look at papers that have been cited in isolation. A robust literature search is also needed to limit publication bias.

• Subject guides
• Researching for your literature review
• Develop a search strategy

Researching for your literature review: Develop a search strategy

• Literature reviews
• Literature sources
• Before you start
• Keyword search activity
• Subject search activity
• Combined keyword and subject searching
• Online tutorials
• Apply search limits
• Run a search in different databases
• Supplementary searching
• Save your searches
• Manage results

Identify key terms and concepts

Start developing a search strategy by identifying the key words and concepts within your research question. 

For example:   How do s t udents view inclusive educational practices in schools ?

Treat each component as a separate concept (there are usually between 2-4 concepts).

For each concept list the key words derived from your research question, as well as any other relevant terms or synonyms that you have found in your preliminary searches. Also consider singular and plural forms of words, variant spellings, acronyms and relevant index terms (subject headings).  

As part of the process of developing a search strategy, it is recommended that you keep a master list of search terms for each key concept. This will make it easier when it comes to translating your search strategy across multiple database platforms. 

Concept map template for documenting search terms

Combine search terms and concepts

Boolean operators are used to combine the different concepts in your topic to form a search strategy. The main operators used to connect your terms are AND and OR . See an explanation below:

• Link keywords related to a single concept with OR
• Linking with OR broadens a search (increases the number of results) by searching for any of the alternative keywords

Example: perspective  OR attitude

• Link different concepts with AND
• Linking with AND narrows a search (reduces the number of results) by retrieving only those records that include all of your specified keywords

Example: inclusive education  AND student perspective

• using NOT narrows a search by excluding certain search terms
• Most searches do not require the use of the NOT operator

Example: education  NOT higher education  will retrieve all results that include the word education  but don’t contain the phrase  higher education .

See the website for venn diagrams demonstrating the function of AND/OR/NOT:

Combine the search terms using Boolean

Advanced search operators - truncation and wildcards

Use symbols to retrieve word variations:

The truncation symbol is commonly an asterisk * and is added at the end of a word.

• The asterisk applied to the root of a word captures other endings to that root word making it useful for retrieving singular, plural and other variations of a keyword.

Example:  educat *  will retrieve educat ion, educat ors, educat ional , etc

Note: If you don't want to retrieve all possible variations, an easy alternative is to utilise the OR operator instead e.g. education OR educational.

The wildcard symbols include the question mark ? and hash #. They replace zero, one or more characters in the middle of a word.

Example:  wom # n finds woman or women, behavio ? r finds behaviour or behavior.

The symbols may vary in different databases - See the Database search tips guide for details or check the Help link in any database.

Phrase searching

Use quotes to keep word order when searching for phrases.

For phrase searching, place two or more words in "inverted commas" or "quote marks".

Example: “inclusive education”

In some databases, words may be searched separately if the quote marks are not used. In other databases, word order may be maintained without the need for quote marks.

See the Database search tips for details on phrase searching in key databases, or check the Help link in any database.

Subject headings (index terms)

Identify appropriate subject headings (index terms).

Many databases use subject headings to index content. These are selected from a controlled list and describe what the article is about. 

A comprehensive search strategy is often best achieved by using a combination of keywords and subject headings where possible.

In-depth knowledge of subject headings is not required for users to benefit from improved search performance using them in their searches.

Advantages of subject searching:

• Helps locate articles that use synonyms, variant spellings, plurals
• Search terms don’t have to appear in the title or abstract

Note: Subject headings are often unique to a particular database, so you will need to look for appropriate subject headings in each database you intend to use.

Subject headings are not available for every topic, and it is best to only select them if they relate closely to your area of interest.

Create a gold set

It is useful to build a ‘sample set’ or ‘gold set’ of relevant references before you develop your search strategy.  .

Sources for a 'gold set' may include:

• key papers recommended by subject experts or supervisors
• citation searching - looking at a reference list to see who has been cited, or using a citation database (eg. Scopus, Web of Science) to see who has cited a known relevant article
• results of preliminary scoping searches.

The papers in your 'gold set' can then be used to help you identify relevant search terms

• Look up your 'sample set' articles in a database that you will use for your literature review. For the articles indexed in the database, look at the records to see what keywords and/or subject headings are listed.

The 'gold set' will also provide a means of testing your search strategy

• When an indexed article is not retrieved, your search strategy can be revised in order to include it (see what concepts or keywords can be incorporated into your search strategy so that the article is retrieved).
• If your search strategy is retrieving a lot of irrelevant results, look at the irrelevant records to determine why they are being retrieved. What keywords or subject headings are causing them to appear? Can you change these without losing any relevant articles from your results?

Example search strategy

An example of a search strategy incorporating all three concepts that could be applied to different databases is shown below:.

The above search strategy in a nested format (for use in a single search box) would look like:

(student* OR pupil* OR "young people" OR learner*) AND (perception* OR experience OR voice OR perspective*) AND (inclusi* OR "special education" OR belonging OR disabilit*)

• << Previous: Search strategies - Education/Social sciences topic example
• Next: Keyword search activity >>

Systematic Reviews & Meta-Analysis

• Identifying Your Research Question
• Developing Your Protocol
• Conducting Your Search
• Screening & Selection
• Data Extraction & Appraisal
• Meta-Analyses
• Writing the Systematic Review
• Suggested Readings

What is a Systematic Search?

The goal of systematic review searches is to identify all relevant studies on a topic. Therefore, systematic review searches are typically quite extensive. The process should be transparent and repeatable, meaning that others can use your process to repeat it for themselves.

Developing your search strategy is the key to ensuring that you find the right kind of evidence for your systematic review. Your search strategy refers to the specific keywords, subject headings, filters and connectors you will use to find relevant literature. The search terms for each one of your concepts should consist of keywords and subject headings when available.

Having already formed an answerable question before beginning your search, you now have the key topics and components necessary to build your search strategy. To help build your search strategy, it would be helpful to conduct a preliminary search and a final comprehensive search.

Preliminary Searches (non-systematic)

Conducting some preliminary, non-systematic searches on your topic will help you to:

• Identify keywords and synonyms to use in your search
• Find existing systematic reviews on any component of your topic and review search strategies included in the methods or appendix
• Locate relevant articles that will likely be included in your study

Final Comprehensive Search (systematic)

When you have completed your preliminary searches, consider the following when conducting the final search:

• Make sure all keywords for each concept as well as synonyms/related terms have been identified
• Identify any databases and resources to search through the library's website or by discussing with a librarian
• Once you are happy with your final search strategy, translate this search strategy to other database(s)
• Run your search on more than one database
• Choose a day to run all of your searches. This helps to reduce bias
• Compile ALL results from all databases searched; export these results into a citation management tool and remove duplicates
• Check the reference lists of all included studies; also check whether the included study has been cited using Scopus, Web of Science, or Google Scholar
• Determine relevant grey literature sources and search with modified strategies

Identifying Search Terms

• Subject Headings
• Search Hedges

Keywords are words or phrases that can be searched for in different database fields such as title, abstract, author keywords, journal etc. You can use the PICO, SPICE, SPIDER, etc. concepts of your research question as your preliminary key terms. It is important to consider the synonyms for each concept so that you have as few gaps as possible in your extensive search.

Below is an example of how you may keep track of your keywords:

Subject Headings are assigned to articles, ebooks, and any other resources found in a database by indexers in order to identify the main topics of an article. Different databases use their own subject heading classification systems.

Some examples of subject heading systems or controlled vocabulary include Medical Subject Headings (MeSH) and Library of Congress Subject Headings (LCSH) . It useful to use subject heading, since they are used to tag resources on similar subjects. When you search using a subject heading, you will get more relevant results in return.

Not all databases will have subject heading searching and for those that do, the subject heading categories may differ between databases. This is because databases classify articles using different criteria.

It is also helpful to keep track of your subject headings in a similar fashion as you would with your keywords:

Search Hedges are search strings created by experts to help you retrieve specific types of studies or topics; a hedge will filter your results by adding specific search terms, or specific combinations of search terms, to your search.

Hedges can be good starting points but you may need to modify the search string to fit your research. Resources for hedges:

• University of Texas, School of Public Health
• McMaster University Health Information Research Unit

Finding Evidence Through the Library's Resources

Once you have identified all of your keywords and subjects, you will be ready to develop your search strategy. You can find more information on developing a search strategy and searching for evidence through the library in the following guide: 

• How to Search the Library and Databases
• << Previous: Developing Your Protocol
• Next: Screening & Selection >>
• Last Updated: Jun 5, 2024 8:45 AM
• URL: https://libguides.chapman.edu/systematic_reviews

A systematic literature review of empirical research on ChatGPT in education

• Open access
• Published: 26 May 2024
• Volume 3 , article number  60 , ( 2024 )

Cite this article

You have full access to this open access article

• Yazid Albadarin   ORCID: orcid.org/0009-0005-8068-8902 1 ,
• Mohammed Saqr 1 ,
• Nicolas Pope 1 &
• Markku Tukiainen 1  

643 Accesses

Explore all metrics

Over the last four decades, studies have investigated the incorporation of Artificial Intelligence (AI) into education. A recent prominent AI-powered technology that has impacted the education sector is ChatGPT. This article provides a systematic review of 14 empirical studies incorporating ChatGPT into various educational settings, published in 2022 and before the 10th of April 2023—the date of conducting the search process. It carefully followed the essential steps outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines, as well as Okoli’s (Okoli in Commun Assoc Inf Syst, 2015) steps for conducting a rigorous and transparent systematic review. In this review, we aimed to explore how students and teachers have utilized ChatGPT in various educational settings, as well as the primary findings of those studies. By employing Creswell’s (Creswell in Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook], Pearson Education, London, 2015) coding techniques for data extraction and interpretation, we sought to gain insight into their initial attempts at ChatGPT incorporation into education. This approach also enabled us to extract insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of this review show that learners have utilized ChatGPT as a virtual intelligent assistant, where it offered instant feedback, on-demand answers, and explanations of complex topics. Additionally, learners have used it to enhance their writing and language skills by generating ideas, composing essays, summarizing, translating, paraphrasing texts, or checking grammar. Moreover, learners turned to it as an aiding tool to facilitate their directed and personalized learning by assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. However, the results of specific studies (n = 3, 21.4%) show that overuse of ChatGPT may negatively impact innovative capacities and collaborative learning competencies among learners. Educators, on the other hand, have utilized ChatGPT to create lesson plans, generate quizzes, and provide additional resources, which helped them enhance their productivity and efficiency and promote different teaching methodologies. Despite these benefits, the majority of the reviewed studies recommend the importance of conducting structured training, support, and clear guidelines for both learners and educators to mitigate the drawbacks. This includes developing critical evaluation skills to assess the accuracy and relevance of information provided by ChatGPT, as well as strategies for integrating human interaction and collaboration into learning activities that involve AI tools. Furthermore, they also recommend ongoing research and proactive dialogue with policymakers, stakeholders, and educational practitioners to refine and enhance the use of AI in learning environments. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Similar content being viewed by others

Empowering learners with ChatGPT: insights from a systematic literature exploration

Incorporating AI in foreign language education: An investigation into ChatGPT’s effect on foreign language learners

Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT

Avoid common mistakes on your manuscript.

1 Introduction

Educational technology, a rapidly evolving field, plays a crucial role in reshaping the landscape of teaching and learning [ 82 ]. One of the most transformative technological innovations of our era that has influenced the field of education is Artificial Intelligence (AI) [ 50 ]. Over the last four decades, AI in education (AIEd) has gained remarkable attention for its potential to make significant advancements in learning, instructional methods, and administrative tasks within educational settings [ 11 ]. In particular, a large language model (LLM), a type of AI algorithm that applies artificial neural networks (ANNs) and uses massively large data sets to understand, summarize, generate, and predict new content that is almost difficult to differentiate from human creations [ 79 ], has opened up novel possibilities for enhancing various aspects of education, from content creation to personalized instruction [ 35 ]. Chatbots that leverage the capabilities of LLMs to understand and generate human-like responses have also presented the capacity to enhance student learning and educational outcomes by engaging students, offering timely support, and fostering interactive learning experiences [ 46 ].

The ongoing and remarkable technological advancements in chatbots have made their use more convenient, increasingly natural and effortless, and have expanded their potential for deployment across various domains [ 70 ]. One prominent example of chatbot applications is the Chat Generative Pre-Trained Transformer, known as ChatGPT, which was introduced by OpenAI, a leading AI research lab, on November 30th, 2022. ChatGPT employs a variety of deep learning techniques to generate human-like text, with a particular focus on recurrent neural networks (RNNs). Long short-term memory (LSTM) allows it to grasp the context of the text being processed and retain information from previous inputs. Also, the transformer architecture, a neural network architecture based on the self-attention mechanism, allows it to analyze specific parts of the input, thereby enabling it to produce more natural-sounding and coherent output. Additionally, the unsupervised generative pre-training and the fine-tuning methods allow ChatGPT to generate more relevant and accurate text for specific tasks [ 31 , 62 ]. Furthermore, reinforcement learning from human feedback (RLHF), a machine learning approach that combines reinforcement learning techniques with human-provided feedback, has helped improve ChatGPT’s model by accelerating the learning process and making it significantly more efficient.

This cutting-edge natural language processing (NLP) tool is widely recognized as one of today's most advanced LLMs-based chatbots [ 70 ], allowing users to ask questions and receive detailed, coherent, systematic, personalized, convincing, and informative human-like responses [ 55 ], even within complex and ambiguous contexts [ 63 , 77 ]. ChatGPT is considered the fastest-growing technology in history: in just three months following its public launch, it amassed an estimated 120 million monthly active users [ 16 ] with an estimated 13 million daily queries [ 49 ], surpassing all other applications [ 64 ]. This remarkable growth can be attributed to the unique features and user-friendly interface that ChatGPT offers. Its intuitive design allows users to interact seamlessly with the technology, making it accessible to a diverse range of individuals, regardless of their technical expertise [ 78 ]. Additionally, its exceptional performance results from a combination of advanced algorithms, continuous enhancements, and extensive training on a diverse dataset that includes various text sources such as books, articles, websites, and online forums [ 63 ], have contributed to a more engaging and satisfying user experience [ 62 ]. These factors collectively explain its remarkable global growth and set it apart from predecessors like Bard, Bing Chat, ERNIE, and others.

In this context, several studies have explored the technological advancements of chatbots. One noteworthy recent research effort, conducted by Schöbel et al. [ 70 ], stands out for its comprehensive analysis of more than 5,000 studies on communication agents. This study offered a comprehensive overview of the historical progression and future prospects of communication agents, including ChatGPT. Moreover, other studies have focused on making comparisons, particularly between ChatGPT and alternative chatbots like Bard, Bing Chat, ERNIE, LaMDA, BlenderBot, and various others. For example, O’Leary [ 53 ] compared two chatbots, LaMDA and BlenderBot, with ChatGPT and revealed that ChatGPT outperformed both. This superiority arises from ChatGPT’s capacity to handle a wider range of questions and generate slightly varied perspectives within specific contexts. Similarly, ChatGPT exhibited an impressive ability to formulate interpretable responses that were easily understood when compared with Google's feature snippet [ 34 ]. Additionally, ChatGPT was compared to other LLMs-based chatbots, including Bard and BERT, as well as ERNIE. The findings indicated that ChatGPT exhibited strong performance in the given tasks, often outperforming the other models [ 59 ].

Furthermore, in the education context, a comprehensive study systematically compared a range of the most promising chatbots, including Bard, Bing Chat, ChatGPT, and Ernie across a multidisciplinary test that required higher-order thinking. The study revealed that ChatGPT achieved the highest score, surpassing Bing Chat and Bard [ 64 ]. Similarly, a comparative analysis was conducted to compare ChatGPT with Bard in answering a set of 30 mathematical questions and logic problems, grouped into two question sets. Set (A) is unavailable online, while Set (B) is available online. The results revealed ChatGPT's superiority in Set (A) over Bard. Nevertheless, Bard's advantage emerged in Set (B) due to its capacity to access the internet directly and retrieve answers, a capability that ChatGPT does not possess [ 57 ]. However, through these varied assessments, ChatGPT consistently highlights its exceptional prowess compared to various alternatives in the ever-evolving chatbot technology.

The widespread adoption of chatbots, especially ChatGPT, by millions of students and educators, has sparked extensive discussions regarding its incorporation into the education sector [ 64 ]. Accordingly, many scholars have contributed to the discourse, expressing both optimism and pessimism regarding the incorporation of ChatGPT into education. For example, ChatGPT has been highlighted for its capabilities in enriching the learning and teaching experience through its ability to support different learning approaches, including adaptive learning, personalized learning, and self-directed learning [ 58 , 60 , 91 ]), deliver summative and formative feedback to students and provide real-time responses to questions, increase the accessibility of information [ 22 , 40 , 43 ], foster students’ performance, engagement and motivation [ 14 , 44 , 58 ], and enhance teaching practices [ 17 , 18 , 64 , 74 ].

On the other hand, concerns have been also raised regarding its potential negative effects on learning and teaching. These include the dissemination of false information and references [ 12 , 23 , 61 , 85 ], biased reinforcement [ 47 , 50 ], compromised academic integrity [ 18 , 40 , 66 , 74 ], and the potential decline in students' skills [ 43 , 61 , 64 , 74 ]. As a result, ChatGPT has been banned in multiple countries, including Russia, China, Venezuela, Belarus, and Iran, as well as in various educational institutions in India, Italy, Western Australia, France, and the United States [ 52 , 90 ].

Clearly, the advent of chatbots, especially ChatGPT, has provoked significant controversy due to their potential impact on learning and teaching. This indicates the necessity for further exploration to gain a deeper understanding of this technology and carefully evaluate its potential benefits, limitations, challenges, and threats to education [ 79 ]. Therefore, conducting a systematic literature review will provide valuable insights into the potential prospects and obstacles linked to its incorporation into education. This systematic literature review will primarily focus on ChatGPT, driven by the aforementioned key factors outlined above.

However, the existing literature lacks a systematic literature review of empirical studies. Thus, this systematic literature review aims to address this gap by synthesizing the existing empirical studies conducted on chatbots, particularly ChatGPT, in the field of education, highlighting how ChatGPT has been utilized in educational settings, and identifying any existing gaps. This review may be particularly useful for researchers in the field and educators who are contemplating the integration of ChatGPT or any chatbot into education. The following research questions will guide this study:

What are students' and teachers' initial attempts at utilizing ChatGPT in education?

What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?

2 Methodology

To conduct this study, the authors followed the essential steps of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) and Okoli’s [ 54 ] steps for conducting a systematic review. These included identifying the study’s purpose, drafting a protocol, applying a practical screening process, searching the literature, extracting relevant data, evaluating the quality of the included studies, synthesizing the studies, and ultimately writing the review. The subsequent section provides an extensive explanation of how these steps were carried out in this study.

2.1 Identify the purpose

Given the widespread adoption of ChatGPT by students and teachers for various educational purposes, often without a thorough understanding of responsible and effective use or a clear recognition of its potential impact on learning and teaching, the authors recognized the need for further exploration of ChatGPT's impact on education in this early stage. Therefore, they have chosen to conduct a systematic literature review of existing empirical studies that incorporate ChatGPT into educational settings. Despite the limited number of empirical studies due to the novelty of the topic, their goal is to gain a deeper understanding of this technology and proactively evaluate its potential benefits, limitations, challenges, and threats to education. This effort could help to understand initial reactions and attempts at incorporating ChatGPT into education and bring out insights and considerations that can inform the future development of education.

2.2 Draft the protocol

The next step is formulating the protocol. This protocol serves to outline the study process in a rigorous and transparent manner, mitigating researcher bias in study selection and data extraction [ 88 ]. The protocol will include the following steps: generating the research question, predefining a literature search strategy, identifying search locations, establishing selection criteria, assessing the studies, developing a data extraction strategy, and creating a timeline.

2.3 Apply practical screen

The screening step aims to accurately filter the articles resulting from the searching step and select the empirical studies that have incorporated ChatGPT into educational contexts, which will guide us in answering the research questions and achieving the objectives of this study. To ensure the rigorous execution of this step, our inclusion and exclusion criteria were determined based on the authors' experience and informed by previous successful systematic reviews [ 21 ]. Table 1 summarizes the inclusion and exclusion criteria for study selection.

2.4 Literature search

We conducted a thorough literature search to identify articles that explored, examined, and addressed the use of ChatGPT in Educational contexts. We utilized two research databases: Dimensions.ai, which provides access to a large number of research publications, and lens.org, which offers access to over 300 million articles, patents, and other research outputs from diverse sources. Additionally, we included three databases, Scopus, Web of Knowledge, and ERIC, which contain relevant research on the topic that addresses our research questions. To browse and identify relevant articles, we used the following search formula: ("ChatGPT" AND "Education"), which included the Boolean operator "AND" to get more specific results. The subject area in the Scopus and ERIC databases were narrowed to "ChatGPT" and "Education" keywords, and in the WoS database was limited to the "Education" category. The search was conducted between the 3rd and 10th of April 2023, which resulted in 276 articles from all selected databases (111 articles from Dimensions.ai, 65 from Scopus, 28 from Web of Science, 14 from ERIC, and 58 from Lens.org). These articles were imported into the Rayyan web-based system for analysis. The duplicates were identified automatically by the system. Subsequently, the first author manually reviewed the duplicated articles ensured that they had the same content, and then removed them, leaving us with 135 unique articles. Afterward, the titles, abstracts, and keywords of the first 40 manuscripts were scanned and reviewed by the first author and were discussed with the second and third authors to resolve any disagreements. Subsequently, the first author proceeded with the filtering process for all articles and carefully applied the inclusion and exclusion criteria as presented in Table  1 . Articles that met any one of the exclusion criteria were eliminated, resulting in 26 articles. Afterward, the authors met to carefully scan and discuss them. The authors agreed to eliminate any empirical studies solely focused on checking ChatGPT capabilities, as these studies do not guide us in addressing the research questions and achieving the study's objectives. This resulted in 14 articles eligible for analysis.

2.5 Quality appraisal

The examination and evaluation of the quality of the extracted articles is a vital step [ 9 ]. Therefore, the extracted articles were carefully evaluated for quality using Fink’s [ 24 ] standards, which emphasize the necessity for detailed descriptions of methodology, results, conclusions, strengths, and limitations. The process began with a thorough assessment of each study's design, data collection, and analysis methods to ensure their appropriateness and comprehensive execution. The clarity, consistency, and logical progression from data to results and conclusions were also critically examined. Potential biases and recognized limitations within the studies were also scrutinized. Ultimately, two articles were excluded for failing to meet Fink’s criteria, particularly in providing sufficient detail on methodology, results, conclusions, strengths, or limitations. The review process is illustrated in Fig.  1 .

The study selection process

2.6 Data extraction

The next step is data extraction, the process of capturing the key information and categories from the included studies. To improve efficiency, reduce variation among authors, and minimize errors in data analysis, the coding categories were constructed using Creswell's [ 15 ] coding techniques for data extraction and interpretation. The coding process involves three sequential steps. The initial stage encompasses open coding , where the researcher examines the data, generates codes to describe and categorize it, and gains a deeper understanding without preconceived ideas. Following open coding is axial coding , where the interrelationships between codes from open coding are analyzed to establish more comprehensive categories or themes. The process concludes with selective coding , refining and integrating categories or themes to identify core concepts emerging from the data. The first coder performed the coding process, then engaged in discussions with the second and third authors to finalize the coding categories for the first five articles. The first coder then proceeded to code all studies and engaged again in discussions with the other authors to ensure the finalization of the coding process. After a comprehensive analysis and capturing of the key information from the included studies, the data extraction and interpretation process yielded several themes. These themes have been categorized and are presented in Table  2 . It is important to note that open coding results were removed from Table  2 for aesthetic reasons, as it included many generic aspects, such as words, short phrases, or sentences mentioned in the studies.

2.7 Synthesize studies

In this stage, we will gather, discuss, and analyze the key findings that emerged from the selected studies. The synthesis stage is considered a transition from an author-centric to a concept-centric focus, enabling us to map all the provided information to achieve the most effective evaluation of the data [ 87 ]. Initially, the authors extracted data that included general information about the selected studies, including the author(s)' names, study titles, years of publication, educational levels, research methodologies, sample sizes, participants, main aims or objectives, raw data sources, and analysis methods. Following that, all key information and significant results from the selected studies were compiled using Creswell’s [ 15 ] coding techniques for data extraction and interpretation to identify core concepts and themes emerging from the data, focusing on those that directly contributed to our research questions and objectives, such as the initial utilization of ChatGPT in learning and teaching, learners' and educators' familiarity with ChatGPT, and the main findings of each study. Finally, the data related to each selected study were extracted into an Excel spreadsheet for data processing. The Excel spreadsheet was reviewed by the authors, including a series of discussions to ensure the finalization of this process and prepare it for further analysis. Afterward, the final result being analyzed and presented in various types of charts and graphs. Table 4 presents the extracted data from the selected studies, with each study labeled with a capital 'S' followed by a number.

This section consists of two main parts. The first part provides a descriptive analysis of the data compiled from the reviewed studies. The second part presents the answers to the research questions and the main findings of these studies.

3.1 Part 1: descriptive analysis

This section will provide a descriptive analysis of the reviewed studies, including educational levels and fields, participants distribution, country contribution, research methodologies, study sample size, study population, publication year, list of journals, familiarity with ChatGPT, source of data, and the main aims and objectives of the studies. Table 4 presents a comprehensive overview of the extracted data from the selected studies.

3.1.1 The number of the reviewed studies and publication years

The total number of the reviewed studies was 14. All studies were empirical studies and published in different journals focusing on Education and Technology. One study was published in 2022 [S1], while the remaining were published in 2023 [S2]-[S14]. Table 3 illustrates the year of publication, the names of the journals, and the number of reviewed studies published in each journal for the studies reviewed.

3.1.2 Educational levels and fields

The majority of the reviewed studies, 11 studies, were conducted in higher education institutions [S1]-[S10] and [S13]. Two studies did not specify the educational level of the population [S12] and [S14], while one study focused on elementary education [S11]. However, the reviewed studies covered various fields of education. Three studies focused on Arts and Humanities Education [S8], [S11], and [S14], specifically English Education. Two studies focused on Engineering Education, with one in Computer Engineering [S2] and the other in Construction Education [S3]. Two studies focused on Mathematics Education [S5] and [S12]. One study focused on Social Science Education [S13]. One study focused on Early Education [S4]. One study focused on Journalism Education [S9]. Finally, three studies did not specify the field of education [S1], [S6], and [S7]. Figure  2 represents the educational levels in the reviewed studies, while Fig.  3 represents the context of the reviewed studies.

Educational levels in the reviewed studies

Context of the reviewed studies

3.1.3 Participants distribution and countries contribution

The reviewed studies have been conducted across different geographic regions, providing a diverse representation of the studies. The majority of the studies, 10 in total, [S1]-[S3], [S5]-[S9], [S11], and [S14], primarily focused on participants from single countries such as Pakistan, the United Arab Emirates, China, Indonesia, Poland, Saudi Arabia, South Korea, Spain, Tajikistan, and the United States. In contrast, four studies, [S4], [S10], [S12], and [S13], involved participants from multiple countries, including China and the United States [S4], China, the United Kingdom, and the United States [S10], the United Arab Emirates, Oman, Saudi Arabia, and Jordan [S12], Turkey, Sweden, Canada, and Australia [ 13 ]. Figures  4 and 5 illustrate the distribution of participants, whether from single or multiple countries, and the contribution of each country in the reviewed studies, respectively.

The reviewed studies conducted in single or multiple countries

The Contribution of each country in the studies

3.1.4 Study population and sample size

Four study populations were included: university students, university teachers, university teachers and students, and elementary school teachers. Six studies involved university students [S2], [S3], [S5] and [S6]-[S8]. Three studies focused on university teachers [S1], [S4], and [S6], while one study specifically targeted elementary school teachers [S11]. Additionally, four studies included both university teachers and students [S10] and [ 12 , 13 , 14 ], and among them, study [S13] specifically included postgraduate students. In terms of the sample size of the reviewed studies, nine studies included a small sample size of less than 50 participants [S1], [S3], [S6], [S8], and [S10]-[S13]. Three studies had 50–100 participants [S2], [S9], and [S14]. Only one study had more than 100 participants [S7]. It is worth mentioning that study [S4] adopted a mixed methods approach, including 10 participants for qualitative analysis and 110 participants for quantitative analysis.

3.1.5 Participants’ familiarity with using ChatGPT

The reviewed studies recruited a diverse range of participants with varying levels of familiarity with ChatGPT. Five studies [S2], [S4], [S6], [S8], and [S12] involved participants already familiar with ChatGPT, while eight studies [S1], [S3], [S5], [S7], [S9], [S10], [S13] and [S14] included individuals with differing levels of familiarity. Notably, one study [S11] had participants who were entirely unfamiliar with ChatGPT. It is important to note that four studies [S3], [S5], [S9], and [S11] provided training or guidance to their participants before conducting their studies, while ten studies [S1], [S2], [S4], [S6]-[S8], [S10], and [S12]-[S14] did not provide training due to the participants' existing familiarity with ChatGPT.

3.1.6 Research methodology approaches and source(S) of data

The reviewed studies adopted various research methodology approaches. Seven studies adopted qualitative research methodology [S1], [S4], [S6], [S8], [S10], [S11], and [S12], while three studies adopted quantitative research methodology [S3], [S7], and [S14], and four studies employed mixed-methods, which involved a combination of both the strengths of qualitative and quantitative methods [S2], [S5], [S9], and [S13].

In terms of the source(s) of data, the reviewed studies obtained their data from various sources, such as interviews, questionnaires, and pre-and post-tests. Six studies relied on interviews as their primary source of data collection [S1], [S4], [S6], [S10], [S11], and [S12], four studies relied on questionnaires [S2], [S7], [S13], and [S14], two studies combined the use of pre-and post-tests and questionnaires for data collection [S3] and [S9], while two studies combined the use of questionnaires and interviews to obtain the data [S5] and [S8]. It is important to note that six of the reviewed studies were quasi-experimental [S3], [S5], [S8], [S9], [S12], and [S14], while the remaining ones were experimental studies [S1], [S2], [S4], [S6], [S7], [S10], [S11], and [S13]. Figures  6 and 7 illustrate the research methodologies and the source (s) of data used in the reviewed studies, respectively.

Research methodologies in the reviewed studies

Source of data in the reviewed studies

3.1.7 The aim and objectives of the studies

The reviewed studies encompassed a diverse set of aims, with several of them incorporating multiple primary objectives. Six studies [S3], [S6], [S7], [S8], [S11], and [S12] examined the integration of ChatGPT in educational contexts, and four studies [S4], [S5], [S13], and [S14] investigated the various implications of its use in education, while three studies [S2], [S9], and [S10] aimed to explore both its integration and implications in education. Additionally, seven studies explicitly explored attitudes and perceptions of students [S2] and [S3], educators [S1] and [S6], or both [S10], [S12], and [S13] regarding the utilization of ChatGPT in educational settings.

3.2 Part 2: research questions and main findings of the reviewed studies

This part will present the answers to the research questions and the main findings of the reviewed studies, classified into two main categories (learning and teaching) according to AI Education classification by [ 36 ]. Figure  8 summarizes the main findings of the reviewed studies in a visually informative diagram. Table 4 provides a detailed list of the key information extracted from the selected studies that led to generating these themes.

The main findings in the reviewed studies

4 Students' initial attempts at utilizing ChatGPT in learning and main findings from students' perspective

4.1 virtual intelligent assistant.

Nine studies demonstrated that ChatGPT has been utilized by students as an intelligent assistant to enhance and support their learning. Students employed it for various purposes, such as answering on-demand questions [S2]-[S5], [S8], [S10], and [S12], providing valuable information and learning resources [S2]-[S5], [S6], and [S8], as well as receiving immediate feedback [S2], [S4], [S9], [S10], and [S12]. In this regard, students generally were confident in the accuracy of ChatGPT's responses, considering them relevant, reliable, and detailed [S3], [S4], [S5], and [S8]. However, some students indicated the need for improvement, as they found that answers are not always accurate [S2], and that misleading information may have been provided or that it may not always align with their expectations [S6] and [S10]. It was also observed by the students that the accuracy of ChatGPT is dependent on several factors, including the quality and specificity of the user's input, the complexity of the question or topic, and the scope and relevance of its training data [S12]. Many students felt that ChatGPT's answers were not always accurate and most of them believed that it requires good background knowledge to work with.

4.2 Writing and language proficiency assistant

Six of the reviewed studies highlighted that ChatGPT has been utilized by students as a valuable assistant tool to improve their academic writing skills and language proficiency. Among these studies, three mainly focused on English education, demonstrating that students showed sufficient mastery in using ChatGPT for generating ideas, summarizing, paraphrasing texts, and completing writing essays [S8], [S11], and [S14]. Furthermore, ChatGPT helped them in writing by making students active investigators rather than passive knowledge recipients and facilitated the development of their writing skills [S11] and [S14]. Similarly, ChatGPT allowed students to generate unique ideas and perspectives, leading to deeper analysis and reflection on their journalism writing [S9]. In terms of language proficiency, ChatGPT allowed participants to translate content into their home languages, making it more accessible and relevant to their context [S4]. It also enabled them to request changes in linguistic tones or flavors [S8]. Moreover, participants used it to check grammar or as a dictionary [S11].

4.3 Valuable resource for learning approaches

Five studies demonstrated that students used ChatGPT as a valuable complementary resource for self-directed learning. It provided learning resources and guidance on diverse educational topics and created a supportive home learning environment [S2] and [S4]. Moreover, it offered step-by-step guidance to grasp concepts at their own pace and enhance their understanding [S5], streamlined task and project completion carried out independently [S7], provided comprehensive and easy-to-understand explanations on various subjects [S10], and assisted in studying geometry operations, thereby empowering them to explore geometry operations at their own pace [S12]. Three studies showed that students used ChatGPT as a valuable learning resource for personalized learning. It delivered age-appropriate conversations and tailored teaching based on a child's interests [S4], acted as a personalized learning assistant, adapted to their needs and pace, which assisted them in understanding mathematical concepts [S12], and enabled personalized learning experiences in social sciences by adapting to students' needs and learning styles [S13]. On the other hand, it is important to note that, according to one study [S5], students suggested that using ChatGPT may negatively affect collaborative learning competencies between students.

4.4 Enhancing students' competencies

Six of the reviewed studies have shown that ChatGPT is a valuable tool for improving a wide range of skills among students. Two studies have provided evidence that ChatGPT led to improvements in students' critical thinking, reasoning skills, and hazard recognition competencies through engaging them in interactive conversations or activities and providing responses related to their disciplines in journalism [S5] and construction education [S9]. Furthermore, two studies focused on mathematical education have shown the positive impact of ChatGPT on students' problem-solving abilities in unraveling problem-solving questions [S12] and enhancing the students' understanding of the problem-solving process [S5]. Lastly, one study indicated that ChatGPT effectively contributed to the enhancement of conversational social skills [S4].

4.5 Supporting students' academic success

Seven of the reviewed studies highlighted that students found ChatGPT to be beneficial for learning as it enhanced learning efficiency and improved the learning experience. It has been observed to improve students' efficiency in computer engineering studies by providing well-structured responses and good explanations [S2]. Additionally, students found it extremely useful for hazard reporting [S3], and it also enhanced their efficiency in solving mathematics problems and capabilities [S5] and [S12]. Furthermore, by finding information, generating ideas, translating texts, and providing alternative questions, ChatGPT aided students in deepening their understanding of various subjects [S6]. It contributed to an increase in students' overall productivity [S7] and improved efficiency in composing written tasks [S8]. Regarding learning experiences, ChatGPT was instrumental in assisting students in identifying hazards that they might have otherwise overlooked [S3]. It also improved students' learning experiences in solving mathematics problems and developing abilities [S5] and [S12]. Moreover, it increased students' successful completion of important tasks in their studies [S7], particularly those involving average difficulty writing tasks [S8]. Additionally, ChatGPT increased the chances of educational success by providing students with baseline knowledge on various topics [S10].

5 Teachers' initial attempts at utilizing ChatGPT in teaching and main findings from teachers' perspective

5.1 valuable resource for teaching.

The reviewed studies showed that teachers have employed ChatGPT to recommend, modify, and generate diverse, creative, organized, and engaging educational contents, teaching materials, and testing resources more rapidly [S4], [S6], [S10] and [S11]. Additionally, teachers experienced increased productivity as ChatGPT facilitated quick and accurate responses to questions, fact-checking, and information searches [S1]. It also proved valuable in constructing new knowledge [S6] and providing timely answers to students' questions in classrooms [S11]. Moreover, ChatGPT enhanced teachers' efficiency by generating new ideas for activities and preplanning activities for their students [S4] and [S6], including interactive language game partners [S11].

5.2 Improving productivity and efficiency

The reviewed studies showed that participants' productivity and work efficiency have been significantly enhanced by using ChatGPT as it enabled them to allocate more time to other tasks and reduce their overall workloads [S6], [S10], [S11], [S13], and [S14]. However, three studies [S1], [S4], and [S11], indicated a negative perception and attitude among teachers toward using ChatGPT. This negativity stemmed from a lack of necessary skills to use it effectively [S1], a limited familiarity with it [S4], and occasional inaccuracies in the content provided by it [S10].

5.3 Catalyzing new teaching methodologies

Five of the reviewed studies highlighted that educators found the necessity of redefining their teaching profession with the assistance of ChatGPT [S11], developing new effective learning strategies [S4], and adapting teaching strategies and methodologies to ensure the development of essential skills for future engineers [S5]. They also emphasized the importance of adopting new educational philosophies and approaches that can evolve with the introduction of ChatGPT into the classroom [S12]. Furthermore, updating curricula to focus on improving human-specific features, such as emotional intelligence, creativity, and philosophical perspectives [S13], was found to be essential.

5.4 Effective utilization of CHATGPT in teaching

According to the reviewed studies, effective utilization of ChatGPT in education requires providing teachers with well-structured training, support, and adequate background on how to use ChatGPT responsibly [S1], [S3], [S11], and [S12]. Establishing clear rules and regulations regarding its usage is essential to ensure it positively impacts the teaching and learning processes, including students' skills [S1], [S4], [S5], [S8], [S9], and [S11]-[S14]. Moreover, conducting further research and engaging in discussions with policymakers and stakeholders is indeed crucial for the successful integration of ChatGPT in education and to maximize the benefits for both educators and students [S1], [S6]-[S10], and [S12]-[S14].

6 Discussion

The purpose of this review is to conduct a systematic review of empirical studies that have explored the utilization of ChatGPT, one of today’s most advanced LLM-based chatbots, in education. The findings of the reviewed studies showed several ways of ChatGPT utilization in different learning and teaching practices as well as it provided insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of the reviewed studies came from diverse fields of education, which helped us avoid a biased review that is limited to a specific field. Similarly, the reviewed studies have been conducted across different geographic regions. This kind of variety in geographic representation enriched the findings of this review.

In response to RQ1 , "What are students' and teachers' initial attempts at utilizing ChatGPT in education?", the findings from this review provide comprehensive insights. Chatbots, including ChatGPT, play a crucial role in supporting student learning, enhancing their learning experiences, and facilitating diverse learning approaches [ 42 , 43 ]. This review found that this tool, ChatGPT, has been instrumental in enhancing students' learning experiences by serving as a virtual intelligent assistant, providing immediate feedback, on-demand answers, and engaging in educational conversations. Additionally, students have benefited from ChatGPT’s ability to generate ideas, compose essays, and perform tasks like summarizing, translating, paraphrasing texts, or checking grammar, thereby enhancing their writing and language competencies. Furthermore, students have turned to ChatGPT for assistance in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks, which fosters a supportive home learning environment, allowing them to take responsibility for their own learning and cultivate the skills and approaches essential for supportive home learning environment [ 26 , 27 , 28 ]. This finding aligns with the study of Saqr et al. [ 68 , 69 ] who highlighted that, when students actively engage in their own learning process, it yields additional advantages, such as heightened motivation, enhanced achievement, and the cultivation of enthusiasm, turning them into advocates for their own learning.

Moreover, students have utilized ChatGPT for tailored teaching and step-by-step guidance on diverse educational topics, streamlining task and project completion, and generating and recommending educational content. This personalization enhances the learning environment, leading to increased academic success. This finding aligns with other recent studies [ 26 , 27 , 28 , 60 , 66 ] which revealed that ChatGPT has the potential to offer personalized learning experiences and support an effective learning process by providing students with customized feedback and explanations tailored to their needs and abilities. Ultimately, fostering students' performance, engagement, and motivation, leading to increase students' academic success [ 14 , 44 , 58 ]. This ultimate outcome is in line with the findings of Saqr et al. [ 68 , 69 ], which emphasized that learning strategies are important catalysts of students' learning, as students who utilize effective learning strategies are more likely to have better academic achievement.

Teachers, too, have capitalized on ChatGPT's capabilities to enhance productivity and efficiency, using it for creating lesson plans, generating quizzes, providing additional resources, generating and preplanning new ideas for activities, and aiding in answering students’ questions. This adoption of technology introduces new opportunities to support teaching and learning practices, enhancing teacher productivity. This finding aligns with those of Day [ 17 ], De Castro [ 18 ], and Su and Yang [ 74 ] as well as with those of Valtonen et al. [ 82 ], who revealed that emerging technological advancements have opened up novel opportunities and means to support teaching and learning practices, and enhance teachers’ productivity.

In response to RQ2 , "What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?", the findings from this review provide profound insights and raise significant concerns. Starting with the insights, chatbots, including ChatGPT, have demonstrated the potential to reshape and revolutionize education, creating new, novel opportunities for enhancing the learning process and outcomes [ 83 ], facilitating different learning approaches, and offering a range of pedagogical benefits [ 19 , 43 , 72 ]. In this context, this review found that ChatGPT could open avenues for educators to adopt or develop new effective learning and teaching strategies that can evolve with the introduction of ChatGPT into the classroom. Nonetheless, there is an evident lack of research understanding regarding the potential impact of generative machine learning models within diverse educational settings [ 83 ]. This necessitates teachers to attain a high level of proficiency in incorporating chatbots, such as ChatGPT, into their classrooms to create inventive, well-structured, and captivating learning strategies. In the same vein, the review also found that teachers without the requisite skills to utilize ChatGPT realized that it did not contribute positively to their work and could potentially have adverse effects [ 37 ]. This concern could lead to inequity of access to the benefits of chatbots, including ChatGPT, as individuals who lack the necessary expertise may not be able to harness their full potential, resulting in disparities in educational outcomes and opportunities. Therefore, immediate action is needed to address these potential issues. A potential solution is offering training, support, and competency development for teachers to ensure that all of them can leverage chatbots, including ChatGPT, effectively and equitably in their educational practices [ 5 , 28 , 80 ], which could enhance accessibility and inclusivity, and potentially result in innovative outcomes [ 82 , 83 ].

Additionally, chatbots, including ChatGPT, have the potential to significantly impact students' thinking abilities, including retention, reasoning, analysis skills [ 19 , 45 ], and foster innovation and creativity capabilities [ 83 ]. This review found that ChatGPT could contribute to improving a wide range of skills among students. However, it found that frequent use of ChatGPT may result in a decrease in innovative capacities, collaborative skills and cognitive capacities, and students' motivation to attend classes, as well as could lead to reduced higher-order thinking skills among students [ 22 , 29 ]. Therefore, immediate action is needed to carefully examine the long-term impact of chatbots such as ChatGPT, on learning outcomes as well as to explore its incorporation into educational settings as a supportive tool without compromising students' cognitive development and critical thinking abilities. In the same vein, the review also found that it is challenging to draw a consistent conclusion regarding the potential of ChatGPT to aid self-directed learning approach. This finding aligns with the recent study of Baskara [ 8 ]. Therefore, further research is needed to explore the potential of ChatGPT for self-directed learning. One potential solution involves utilizing learning analytics as a novel approach to examine various aspects of students' learning and support them in their individual endeavors [ 32 ]. This approach can bridge this gap by facilitating an in-depth analysis of how learners engage with ChatGPT, identifying trends in self-directed learning behavior, and assessing its influence on their outcomes.

Turning to the significant concerns, on the other hand, a fundamental challenge with LLM-based chatbots, including ChatGPT, is the accuracy and quality of the provided information and responses, as they provide false information as truth—a phenomenon often referred to as "hallucination" [ 3 , 49 ]. In this context, this review found that the provided information was not entirely satisfactory. Consequently, the utilization of chatbots presents potential concerns, such as generating and providing inaccurate or misleading information, especially for students who utilize it to support their learning. This finding aligns with other findings [ 6 , 30 , 35 , 40 ] which revealed that incorporating chatbots such as ChatGPT, into education presents challenges related to its accuracy and reliability due to its training on a large corpus of data, which may contain inaccuracies and the way users formulate or ask ChatGPT. Therefore, immediate action is needed to address these potential issues. One possible solution is to equip students with the necessary skills and competencies, which include a background understanding of how to use it effectively and the ability to assess and evaluate the information it generates, as the accuracy and the quality of the provided information depend on the input, its complexity, the topic, and the relevance of its training data [ 28 , 49 , 86 ]. However, it's also essential to examine how learners can be educated about how these models operate, the data used in their training, and how to recognize their limitations, challenges, and issues [ 79 ].

Furthermore, chatbots present a substantial challenge concerning maintaining academic integrity [ 20 , 56 ] and copyright violations [ 83 ], which are significant concerns in education. The review found that the potential misuse of ChatGPT might foster cheating, facilitate plagiarism, and threaten academic integrity. This issue is also affirmed by the research conducted by Basic et al. [ 7 ], who presented evidence that students who utilized ChatGPT in their writing assignments had more plagiarism cases than those who did not. These findings align with the conclusions drawn by Cotton et al. [ 13 ], Hisan and Amri [ 33 ] and Sullivan et al. [ 75 ], who revealed that the integration of chatbots such as ChatGPT into education poses a significant challenge to the preservation of academic integrity. Moreover, chatbots, including ChatGPT, have increased the difficulty in identifying plagiarism [ 47 , 67 , 76 ]. The findings from previous studies [ 1 , 84 ] indicate that AI-generated text often went undetected by plagiarism software, such as Turnitin. However, Turnitin and other similar plagiarism detection tools, such as ZeroGPT, GPTZero, and Copyleaks, have since evolved, incorporating enhanced techniques to detect AI-generated text, despite the possibility of false positives, as noted in different studies that have found these tools still not yet fully ready to accurately and reliably identify AI-generated text [ 10 , 51 ], and new novel detection methods may need to be created and implemented for AI-generated text detection [ 4 ]. This potential issue could lead to another concern, which is the difficulty of accurately evaluating student performance when they utilize chatbots such as ChatGPT assistance in their assignments. Consequently, the most LLM-driven chatbots present a substantial challenge to traditional assessments [ 64 ]. The findings from previous studies indicate the importance of rethinking, improving, and redesigning innovative assessment methods in the era of chatbots [ 14 , 20 , 64 , 75 ]. These methods should prioritize the process of evaluating students' ability to apply knowledge to complex cases and demonstrate comprehension, rather than solely focusing on the final product for assessment. Therefore, immediate action is needed to address these potential issues. One possible solution would be the development of clear guidelines, regulatory policies, and pedagogical guidance. These measures would help regulate the proper and ethical utilization of chatbots, such as ChatGPT, and must be established before their introduction to students [ 35 , 38 , 39 , 41 , 89 ].

In summary, our review has delved into the utilization of ChatGPT, a prominent example of chatbots, in education, addressing the question of how ChatGPT has been utilized in education. However, there remain significant gaps, which necessitate further research to shed light on this area.

7 Conclusions

This systematic review has shed light on the varied initial attempts at incorporating ChatGPT into education by both learners and educators, while also offering insights and considerations that can facilitate its effective and responsible use in future educational contexts. From the analysis of 14 selected studies, the review revealed the dual-edged impact of ChatGPT in educational settings. On the positive side, ChatGPT significantly aided the learning process in various ways. Learners have used it as a virtual intelligent assistant, benefiting from its ability to provide immediate feedback, on-demand answers, and easy access to educational resources. Additionally, it was clear that learners have used it to enhance their writing and language skills, engaging in practices such as generating ideas, composing essays, and performing tasks like summarizing, translating, paraphrasing texts, or checking grammar. Importantly, other learners have utilized it in supporting and facilitating their directed and personalized learning on a broad range of educational topics, assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. Educators, on the other hand, found ChatGPT beneficial for enhancing productivity and efficiency. They used it for creating lesson plans, generating quizzes, providing additional resources, and answers learners' questions, which saved time and allowed for more dynamic and engaging teaching strategies and methodologies.

However, the review also pointed out negative impacts. The results revealed that overuse of ChatGPT could decrease innovative capacities and collaborative learning among learners. Specifically, relying too much on ChatGPT for quick answers can inhibit learners' critical thinking and problem-solving skills. Learners might not engage deeply with the material or consider multiple solutions to a problem. This tendency was particularly evident in group projects, where learners preferred consulting ChatGPT individually for solutions over brainstorming and collaborating with peers, which negatively affected their teamwork abilities. On a broader level, integrating ChatGPT into education has also raised several concerns, including the potential for providing inaccurate or misleading information, issues of inequity in access, challenges related to academic integrity, and the possibility of misusing the technology.

Accordingly, this review emphasizes the urgency of developing clear rules, policies, and regulations to ensure ChatGPT's effective and responsible use in educational settings, alongside other chatbots, by both learners and educators. This requires providing well-structured training to educate them on responsible usage and understanding its limitations, along with offering sufficient background information. Moreover, it highlights the importance of rethinking, improving, and redesigning innovative teaching and assessment methods in the era of ChatGPT. Furthermore, conducting further research and engaging in discussions with policymakers and stakeholders are essential steps to maximize the benefits for both educators and learners and ensure academic integrity.

It is important to acknowledge that this review has certain limitations. Firstly, the limited inclusion of reviewed studies can be attributed to several reasons, including the novelty of the technology, as new technologies often face initial skepticism and cautious adoption; the lack of clear guidelines or best practices for leveraging this technology for educational purposes; and institutional or governmental policies affecting the utilization of this technology in educational contexts. These factors, in turn, have affected the number of studies available for review. Secondly, the utilization of the original version of ChatGPT, based on GPT-3 or GPT-3.5, implies that new studies utilizing the updated version, GPT-4 may lead to different findings. Therefore, conducting follow-up systematic reviews is essential once more empirical studies on ChatGPT are published. Additionally, long-term studies are necessary to thoroughly examine and assess the impact of ChatGPT on various educational practices.

Despite these limitations, this systematic review has highlighted the transformative potential of ChatGPT in education, revealing its diverse utilization by learners and educators alike and summarized the benefits of incorporating it into education, as well as the forefront critical concerns and challenges that must be addressed to facilitate its effective and responsible use in future educational contexts. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Data availability

The data supporting our findings are available upon request.

Abbreviations

• Artificial intelligence

AI in education

Large language model

Artificial neural networks

Chat Generative Pre-Trained Transformer

Recurrent neural networks

Long short-term memory

Reinforcement learning from human feedback

Natural language processing

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

AlAfnan MA, Dishari S, Jovic M, Lomidze K. ChatGPT as an educational tool: opportunities, challenges, and recommendations for communication, business writing, and composition courses. J Artif Intell Technol. 2023. https://doi.org/10.37965/jait.2023.0184 .

Article   Google Scholar  

Ali JKM, Shamsan MAA, Hezam TA, Mohammed AAQ. Impact of ChatGPT on learning motivation. J Engl Stud Arabia Felix. 2023;2(1):41–9. https://doi.org/10.56540/jesaf.v2i1.51 .

Alkaissi H, McFarlane SI. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 2023. https://doi.org/10.7759/cureus.35179 .

Anderson N, Belavý DL, Perle SM, Hendricks S, Hespanhol L, Verhagen E, Memon AR. AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation. BMJ Open Sport Exerc Med. 2023;9(1): e001568. https://doi.org/10.1136/bmjsem-2023-001568 .

Ausat AMA, Massang B, Efendi M, Nofirman N, Riady Y. Can chat GPT replace the role of the teacher in the classroom: a fundamental analysis. J Educ. 2023;5(4):16100–6.

Google Scholar  

Baidoo-Anu D, Ansah L. Education in the Era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4337484 .

Basic Z, Banovac A, Kruzic I, Jerkovic I. Better by you, better than me, chatgpt3 as writing assistance in students essays. 2023. arXiv preprint arXiv:2302.04536 .‏

Baskara FR. The promises and pitfalls of using chat GPT for self-determined learning in higher education: an argumentative review. Prosiding Seminar Nasional Fakultas Tarbiyah dan Ilmu Keguruan IAIM Sinjai. 2023;2:95–101. https://doi.org/10.47435/sentikjar.v2i0.1825 .

Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Inform. 2019;129:154–66. https://doi.org/10.1016/j.ijmedinf.2019.04.024 .

Chaka C. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: the case of five AI content detection tools. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.2.12 .

Chiu TKF, Xia Q, Zhou X, Chai CS, Cheng M. Systematic literature review on opportunities, challenges, and future research recommendations of artificial intelligence in education. Comput Educ Artif Intell. 2023;4:100118. https://doi.org/10.1016/j.caeai.2022.100118 .

Choi EPH, Lee JJ, Ho M, Kwok JYY, Lok KYW. Chatting or cheating? The impacts of ChatGPT and other artificial intelligence language models on nurse education. Nurse Educ Today. 2023;125:105796. https://doi.org/10.1016/j.nedt.2023.105796 .

Cotton D, Cotton PA, Shipway JR. Chatting and cheating: ensuring academic integrity in the era of ChatGPT. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2190148 .

Crawford J, Cowling M, Allen K. Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.3.02 .

Creswell JW. Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook]. 4th ed. London: Pearson Education; 2015.

Curry D. ChatGPT Revenue and Usage Statistics (2023)—Business of Apps. 2023. https://www.businessofapps.com/data/chatgpt-statistics/

Day T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. Prof Geogr. 2023. https://doi.org/10.1080/00330124.2023.2190373 .

De Castro CA. A Discussion about the Impact of ChatGPT in education: benefits and concerns. J Bus Theor Pract. 2023;11(2):p28. https://doi.org/10.22158/jbtp.v11n2p28 .

Deng X, Yu Z. A meta-analysis and systematic review of the effect of Chatbot technology use in sustainable education. Sustainability. 2023;15(4):2940. https://doi.org/10.3390/su15042940 .

Eke DO. ChatGPT and the rise of generative AI: threat to academic integrity? J Responsib Technol. 2023;13:100060. https://doi.org/10.1016/j.jrt.2023.100060 .

Elmoazen R, Saqr M, Tedre M, Hirsto L. A systematic literature review of empirical research on epistemic network analysis in education. IEEE Access. 2022;10:17330–48. https://doi.org/10.1109/access.2022.3149812 .

Farrokhnia M, Banihashem SK, Noroozi O, Wals AEJ. A SWOT analysis of ChatGPT: implications for educational practice and research. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2195846 .

Fergus S, Botha M, Ostovar M. Evaluating academic answers generated using ChatGPT. J Chem Educ. 2023;100(4):1672–5. https://doi.org/10.1021/acs.jchemed.3c00087 .

Fink A. Conducting research literature reviews: from the Internet to Paper. Incorporated: SAGE Publications; 2010.

Firaina R, Sulisworo D. Exploring the usage of ChatGPT in higher education: frequency and impact on productivity. Buletin Edukasi Indonesia (BEI). 2023;2(01):39–46. https://doi.org/10.56741/bei.v2i01.310 .

Firat, M. (2023). How chat GPT can transform autodidactic experiences and open education.  Department of Distance Education, Open Education Faculty, Anadolu Unive .‏ https://orcid.org/0000-0001-8707-5918

Firat M. What ChatGPT means for universities: perceptions of scholars and students. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.22 .

Fuchs K. Exploring the opportunities and challenges of NLP models in higher education: is Chat GPT a blessing or a curse? Front Educ. 2023. https://doi.org/10.3389/feduc.2023.1166682 .

García-Peñalvo FJ. La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. Educ Knowl Soc. 2023;24: e31279. https://doi.org/10.14201/eks.31279 .

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor A, Chartash D. How does ChatGPT perform on the United States medical Licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9: e45312. https://doi.org/10.2196/45312 .

Hashana AJ, Brundha P, Ayoobkhan MUA, Fazila S. Deep Learning in ChatGPT—A Survey. In   2023 7th international conference on trends in electronics and informatics (ICOEI) . 2023. (pp. 1001–1005). IEEE. https://doi.org/10.1109/icoei56765.2023.10125852

Hirsto L, Saqr M, López-Pernas S, Valtonen T. (2022). A systematic narrative review of learning analytics research in K-12 and schools.  Proceedings . https://ceur-ws.org/Vol-3383/FLAIEC22_paper_9536.pdf

Hisan UK, Amri MM. ChatGPT and medical education: a double-edged sword. J Pedag Educ Sci. 2023;2(01):71–89. https://doi.org/10.13140/RG.2.2.31280.23043/1 .

Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023. https://doi.org/10.1093/jncics/pkad010 .

Househ M, AlSaad R, Alhuwail D, Ahmed A, Healy MG, Latifi S, Sheikh J. Large Language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;9: e48291. https://doi.org/10.2196/48291 .

Ilkka T. The impact of artificial intelligence on learning, teaching, and education. Minist de Educ. 2018. https://doi.org/10.2760/12297 .

Iqbal N, Ahmed H, Azhar KA. Exploring teachers’ attitudes towards using CHATGPT. Globa J Manag Adm Sci. 2022;3(4):97–111. https://doi.org/10.46568/gjmas.v3i4.163 .

Irfan M, Murray L, Ali S. Integration of Artificial intelligence in academia: a case study of critical teaching and learning in Higher education. Globa Soc Sci Rev. 2023;8(1):352–64. https://doi.org/10.31703/gssr.2023(viii-i).32 .

Jeon JH, Lee S. Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11834-1 .

Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT—Reshaping medical education and clinical management. Pak J Med Sci. 2023. https://doi.org/10.12669/pjms.39.2.7653 .

King MR. A conversation on artificial intelligence, Chatbots, and plagiarism in higher education. Cell Mol Bioeng. 2023;16(1):1–2. https://doi.org/10.1007/s12195-022-00754-8 .

Kooli C. Chatbots in education and research: a critical examination of ethical implications and solutions. Sustainability. 2023;15(7):5614. https://doi.org/10.3390/su15075614 .

Kuhail MA, Alturki N, Alramlawi S, Alhejori K. Interacting with educational chatbots: a systematic review. Educ Inf Technol. 2022;28(1):973–1018. https://doi.org/10.1007/s10639-022-11177-3 .

Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. 2023. https://doi.org/10.1002/ase.2270 .

Li L, Subbareddy R, Raghavendra CG. AI intelligence Chatbot to improve students learning in the higher education platform. J Interconnect Netw. 2022. https://doi.org/10.1142/s0219265921430325 .

Limna P. A Review of Artificial Intelligence (AI) in Education during the Digital Era. 2022. https://ssrn.com/abstract=4160798

Lo CK. What is the impact of ChatGPT on education? A rapid review of the literature. Educ Sci. 2023;13(4):410. https://doi.org/10.3390/educsci13040410 .

Luo W, He H, Liu J, Berson IR, Berson MJ, Zhou Y, Li H. Aladdin’s genie or pandora’s box For early childhood education? Experts chat on the roles, challenges, and developments of ChatGPT. Early Educ Dev. 2023. https://doi.org/10.1080/10409289.2023.2214181 .

Meyer JG, Urbanowicz RJ, Martin P, O’Connor K, Li R, Peng P, Moore JH. ChatGPT and large language models in academia: opportunities and challenges. Biodata Min. 2023. https://doi.org/10.1186/s13040-023-00339-9 .

Mhlanga D. Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4354422 .

Neumann, M., Rauschenberger, M., & Schön, E. M. (2023). “We Need To Talk About ChatGPT”: The Future of AI and Higher Education.‏ https://doi.org/10.1109/seeng59157.2023.00010

Nolan B. Here are the schools and colleges that have banned the use of ChatGPT over plagiarism and misinformation fears. Business Insider . 2023. https://www.businessinsider.com

O’Leary DE. An analysis of three chatbots: BlenderBot, ChatGPT and LaMDA. Int J Intell Syst Account, Financ Manag. 2023;30(1):41–54. https://doi.org/10.1002/isaf.1531 .

Okoli C. A guide to conducting a standalone systematic literature review. Commun Assoc Inf Syst. 2015. https://doi.org/10.17705/1cais.03743 .

OpenAI. (2023). https://openai.com/blog/chatgpt

Perkins M. Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.02.07 .

Plevris V, Papazafeiropoulos G, Rios AJ. Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. arXiv (Cornell University) . 2023. https://doi.org/10.48550/arxiv.2305.18618

Rahman MM, Watanobe Y (2023) ChatGPT for education and research: opportunities, threats, and strategies. Appl Sci 13(9):5783. https://doi.org/10.3390/app13095783

Ram B, Verma P. Artificial intelligence AI-based Chatbot study of ChatGPT, google AI bard and baidu AI. World J Adv Eng Technol Sci. 2023;8(1):258–61. https://doi.org/10.30574/wjaets.2023.8.1.0045 .

Rasul T, Nair S, Kalendra D, Robin M, de Oliveira Santini F, Ladeira WJ, Heathcote L. The role of ChatGPT in higher education: benefits, challenges, and future research directions. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.29 .

Ratnam M, Sharm B, Tomer A. ChatGPT: educational artificial intelligence. Int J Adv Trends Comput Sci Eng. 2023;12(2):84–91. https://doi.org/10.30534/ijatcse/2023/091222023 .

Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys Syst. 2023;3:121–54. https://doi.org/10.1016/j.iotcps.2023.04.003 .

Roumeliotis KI, Tselikas ND. ChatGPT and Open-AI models: a preliminary review. Future Internet. 2023;15(6):192. https://doi.org/10.3390/fi15060192 .

Rudolph J, Tan S, Tan S. War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.23 .

Ruiz LMS, Moll-López S, Nuñez-Pérez A, Moraño J, Vega-Fleitas E. ChatGPT challenges blended learning methodologies in engineering education: a case study in mathematics. Appl Sci. 2023;13(10):6039. https://doi.org/10.3390/app13106039 .

Sallam M, Salim NA, Barakat M, Al-Tammemi AB. ChatGPT applications in medical, dental, pharmacy, and public health education: a descriptive study highlighting the advantages and limitations. Narra J. 2023;3(1): e103. https://doi.org/10.52225/narra.v3i1.103 .

Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care. 2023. https://doi.org/10.1186/s13054-023-04380-2 .

Saqr M, López-Pernas S, Helske S, Hrastinski S. The longitudinal association between engagement and achievement varies by time, students’ profiles, and achievement state: a full program study. Comput Educ. 2023;199:104787. https://doi.org/10.1016/j.compedu.2023.104787 .

Saqr M, Matcha W, Uzir N, Jovanović J, Gašević D, López-Pernas S. Transferring effective learning strategies across learning contexts matters: a study in problem-based learning. Australas J Educ Technol. 2023;39(3):9.

Schöbel S, Schmitt A, Benner D, Saqr M, Janson A, Leimeister JM. Charting the evolution and future of conversational agents: a research agenda along five waves and new frontiers. Inf Syst Front. 2023. https://doi.org/10.1007/s10796-023-10375-9 .

Shoufan A. Exploring students’ perceptions of CHATGPT: thematic analysis and follow-up survey. IEEE Access. 2023. https://doi.org/10.1109/access.2023.3268224 .

Sonderegger S, Seufert S. Chatbot-mediated learning: conceptual framework for the design of Chatbot use cases in education. Gallen: Institute for Educational Management and Technologies, University of St; 2022. https://doi.org/10.5220/0010999200003182 .

Book   Google Scholar  

Strzelecki A. To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interact Learn Environ. 2023. https://doi.org/10.1080/10494820.2023.2209881 .

Su J, Yang W. Unlocking the power of ChatGPT: a framework for applying generative AI in education. ECNU Rev Educ. 2023. https://doi.org/10.1177/20965311231168423 .

Sullivan M, Kelly A, McLaughlan P. ChatGPT in higher education: Considerations for academic integrity and student learning. J ApplLearn Teach. 2023;6(1):1–10. https://doi.org/10.37074/jalt.2023.6.1.17 .

Szabo A. ChatGPT is a breakthrough in science and education but fails a test in sports and exercise psychology. Balt J Sport Health Sci. 2023;1(128):25–40. https://doi.org/10.33607/bjshs.v127i4.1233 .

Taecharungroj V. “What can ChatGPT do?” analyzing early reactions to the innovative AI chatbot on Twitter. Big Data Cognit Comput. 2023;7(1):35. https://doi.org/10.3390/bdcc7010035 .

Tam S, Said RB. User preferences for ChatGPT-powered conversational interfaces versus traditional methods. Biomed Eng Soc. 2023. https://doi.org/10.58496/mjcsc/2023/004 .

Tedre M, Kahila J, Vartiainen H. (2023). Exploration on how co-designing with AI facilitates critical evaluation of ethics of AI in craft education. In: Langran E, Christensen P, Sanson J (Eds).  Proceedings of Society for Information Technology and Teacher Education International Conference . 2023. pp. 2289–2296.

Tlili A, Shehata B, Adarkwah MA, Bozkurt A, Hickey DT, Huang R, Agyemang B. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn Environ. 2023. https://doi.org/10.1186/s40561-023-00237-x .

Uddin SMJ, Albert A, Ovid A, Alsharef A. Leveraging CHATGPT to aid construction hazard recognition and support safety education and training. Sustainability. 2023;15(9):7121. https://doi.org/10.3390/su15097121 .

Valtonen T, López-Pernas S, Saqr M, Vartiainen H, Sointu E, Tedre M. The nature and building blocks of educational technology research. Comput Hum Behav. 2022;128:107123. https://doi.org/10.1016/j.chb.2021.107123 .

Vartiainen H, Tedre M. Using artificial intelligence in craft education: crafting with text-to-image generative models. Digit Creat. 2023;34(1):1–21. https://doi.org/10.1080/14626268.2023.2174557 .

Ventayen RJM. OpenAI ChatGPT generated results: similarity index of artificial intelligence-based contents. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4332664 .

Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2023. https://doi.org/10.1177/08465371231171125 .

Wardat Y, Tashtoush MA, AlAli R, Jarrah AM. ChatGPT: a revolutionary tool for teaching and learning mathematics. Eurasia J Math, Sci Technol Educ. 2023;19(7):em2286. https://doi.org/10.29333/ejmste/13272 .

Webster J, Watson RT. Analyzing the past to prepare for the future: writing a literature review. Manag Inf Syst Quart. 2002;26(2):3.

Xiao Y, Watson ME. Guidance on conducting a systematic literature review. J Plan Educ Res. 2017;39(1):93–112. https://doi.org/10.1177/0739456x17723971 .

Yan D. Impact of ChatGPT on learners in a L2 writing practicum: an exploratory investigation. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11742-4 .

Yu H. Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front Psychol. 2023;14:1181712. https://doi.org/10.3389/fpsyg.2023.1181712 .

Zhu C, Sun M, Luo J, Li T, Wang M. How to harness the potential of ChatGPT in education? Knowl Manag ELearn. 2023;15(2):133–52. https://doi.org/10.34105/j.kmel.2023.15.008 .

Download references

The paper is co-funded by the Academy of Finland (Suomen Akatemia) Research Council for Natural Sciences and Engineering for the project Towards precision education: Idiographic learning analytics (TOPEILA), Decision Number 350560.

Author information

Authors and affiliations.

School of Computing, University of Eastern Finland, 80100, Joensuu, Finland

Yazid Albadarin, Mohammed Saqr, Nicolas Pope & Markku Tukiainen

You can also search for this author in PubMed   Google Scholar

Contributions

YA contributed to the literature search, data analysis, discussion, and conclusion. Additionally, YA contributed to the manuscript’s writing, editing, and finalization. MS contributed to the study’s design, conceptualization, acquisition of funding, project administration, allocation of resources, supervision, validation, literature search, and analysis of results. Furthermore, MS contributed to the manuscript's writing, revising, and approving it in its finalized state. NP contributed to the results, and discussions, and provided supervision. NP also contributed to the writing process, revisions, and the final approval of the manuscript in its finalized state. MT contributed to the study's conceptualization, resource management, supervision, writing, revising the manuscript, and approving it.

Corresponding author

Correspondence to Yazid Albadarin .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table  4

The process of synthesizing the data presented in Table  4 involved identifying the relevant studies through a search process of databases (ERIC, Scopus, Web of Knowledge, Dimensions.ai, and lens.org) using specific keywords "ChatGPT" and "education". Following this, inclusion/exclusion criteria were applied, and data extraction was performed using Creswell's [ 15 ] coding techniques to capture key information and identify common themes across the included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Albadarin, Y., Saqr, M., Pope, N. et al. A systematic literature review of empirical research on ChatGPT in education. Discov Educ 3 , 60 (2024). https://doi.org/10.1007/s44217-024-00138-2

Download citation

Received : 22 October 2023

Accepted : 10 May 2024

Published : 26 May 2024

DOI : https://doi.org/10.1007/s44217-024-00138-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Large language models
• Educational technology
• Systematic review

Advertisement

• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 03 June 2024

The effectiveness of digital twins in promoting precision health across the entire population: a systematic review

• Mei-di Shen 1 ,
• Si-bing Chen 2 &
• Xiang-dong Ding   ORCID: orcid.org/0009-0001-1925-0654 2  

npj Digital Medicine volume  7 , Article number:  145 ( 2024 ) Cite this article

217 Accesses

2 Altmetric

Metrics details

• Public health
• Risk factors
• Signs and symptoms

Digital twins represent a promising technology within the domain of precision healthcare, offering significant prospects for individualized medical interventions. Existing systematic reviews, however, mainly focus on the technological dimensions of digital twins, with a limited exploration of their impact on health-related outcomes. Therefore, this systematic review aims to explore the efficacy of digital twins in improving precision healthcare at the population level. The literature search for this study encompassed PubMed, Embase, Web of Science, Cochrane Library, CINAHL, SinoMed, CNKI, and Wanfang Database to retrieve potentially relevant records. Patient health-related outcomes were synthesized employing quantitative content analysis, whereas the Joanna Briggs Institute (JBI) scales were used to evaluate the quality and potential bias inherent in each selected study. Following established inclusion and exclusion criteria, 12 studies were screened from an initial 1321 records for further analysis. These studies included patients with various conditions, including cancers, type 2 diabetes, multiple sclerosis, heart failure, qi deficiency, post-hepatectomy liver failure, and dental issues. The review coded three types of interventions: personalized health management, precision individual therapy effects, and predicting individual risk, leading to a total of 45 outcomes being measured. The collective effectiveness of these outcomes at the population level was calculated at 80% (36 out of 45). No studies exhibited unacceptable differences in quality. Overall, employing digital twins in precision health demonstrates practical advantages, warranting its expanded use to facilitate the transition from the development phase to broad application.

PROSPERO registry: CRD42024507256.

Similar content being viewed by others

Digital twins for health: a scoping review

Digital twins in medicine

The health digital twin to tackle cardiovascular disease—a review of an emerging interdisciplinary field

Introduction.

Precision health represents a paradigm shift from the conventional “one size fits all” medical approach, focusing on specific diagnosis, treatment, and health management by incorporating individualized factors such as omics data, clinical information, and health outcomes 1 , 2 . This approach significantly impacts various diseases, potentially improving overall health while reducing healthcare costs 3 , 4 . Within this context, digital twins emerged as a promising technology 5 , creating digital replicas of the human body through two key steps: building mappings and enabling dynamic evolution 6 . Unlike traditional data mining methods, digital twins consider individual variability, providing continuous, dynamic recommendations for clinical practice 7 . This approach has gained significant attention among researchers, highlighting its potential applications in advancing precision health.

Several systematic reviews have explored the advancement of digital twins within the healthcare sector. One rapid review 8 identified four core functionalities of digital twins in healthcare management: safety management, information management, health management/well-being promotion, and operational control. Another systematic review 9 , through an analysis of 22 selected publications, summarized the diverse application scenarios of digital twins in healthcare, confirming their potential in continuous monitoring, personalized therapy, and hospital management. Furthermore, a quantitative review 10 assessed 94 high-quality articles published from 2018 to 2022, revealing a primary focus on technological advancements (such as artificial intelligence and the Internet of Things) and application scenarios (including personalized, precise, and real-time healthcare solutions), thus highlighting the pivotal role of digital twins technology in the field of precision health. Another systematic review 11 , incorporating 18 framework papers or reviews, underscored the need for ongoing research into digital twins’ healthcare applications, especially during the COVID-19 pandemic. Moreover, a systematic review 12 on the application of digital twins in cardiovascular diseases presented proof-of-concept and data-driven approaches, offering valuable insights for implementing digital twins in this specific medical area.

While the existing literature offers valuable insights into the technological aspects of digital twins in healthcare, these systematic reviews failed to thoroughly examine the actual impacts on population health. Despite the increasing interest and expanding body of research on digital twins in healthcare, the direct effects on patient health-related outcomes remain unclear. This knowledge gap highlights the need to investigate how digital twins promote and restore patient health, which is vital for advancing precision health technologies. Therefore, the objective of our systematic review is to assess the effectiveness of digital twins in improving health-related outcomes at the population level, providing a clearer understanding of their practical benefits in the context of precision health.

Search results

The selection process for the systematic review is outlined in the PRISMA flow chart (Fig. 1 ). Initially, 1321 records were identified. Of these, 446 duplicates (446/1321, 33.76%) were removed, leaving 875 records (875/1321, 66.24%) for title and abstract screening. Applying the pre-defined inclusion and exclusion criteria led to the exclusion of 858 records (858/875, 98.06%), leaving 17 records (17/875, 1.94%) for full-text review. Further scrutiny resulted in the exclusion of one study (1/17, 5.88%) lacking health-related outcomes and four studies (4/17, 23.53%) with overlapping data. Ultimately, 12 (12/17, 70.59%) original studies 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 were included in the systematic review. Supplementary Table 1 provides a summary of the reasons for exclusion at the full-text reading phase.

Flow chart of included studies in the systematic review.

Study characteristics

The studies included in this systematic review were published between 2021 (2/12, 16.67%) 23 , 24 and 2023 (8/12, 66.67%) 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 . Originating from diverse regions, 4/12 studies (33.33%) were from Asia 13 , 14 , 21 , 24 , 5/12 (41.67%) from America 15 , 17 , 19 , 20 , 22 , and 3/12 (25.00%) from Europe 16 , 18 , 23 . The review encompassed various study designs, including randomized controlled trials (1/12, 8.33%) 14 , quasi-experiments (6/12, 50.00%) 13 , 15 , 16 , 18 , 19 , 21 , and cohort studies (5/12, 41.67%) 17 , 20 , 22 , 23 , 24 . The sample sizes ranged from 15 13 to 3500 patients 19 . Five studies assessed the impact of digital twins on virtual patients 15 , 16 , 18 , 19 , 20 , while seven examined their effect on real-world patients 13 , 14 , 17 , 21 , 22 , 23 , 24 . These patients included had various diseases, including cancer (4/12, 33.33%) 15 , 16 , 19 , 22 , type 2 diabetes (2/12, 16.66%) 13 , 14 , multiple sclerosis (2/12, 16.66%) 17 , 18 , qi deficiency (1/12, 8.33%) 21 , heart failure (1/12, 8.33%) 20 , post-hepatectomy liver failure (1/12, 8.33%) 23 , and dental issues (1/12, 8.33%) 24 . This review coded interventions into three types: personalized health management (3/12, 25.00%) 13 , 14 , 21 , precision individual therapy effects (3/12, 25.00%) 15 , 16 , 18 , 19 , 20 , 22 , and predicting individual risk (3/12, 25.00%) 17 , 23 , 24 , with a total of 45 measured outcomes. Characteristics of the included studies are detailed in Table 1 .

Risk of bias assessment

The risk of bias for the studies included in this review is summarized in Fig. 2 . In the single RCT 14 assessed, 10 out of 13 items received positive responses. Limitations were observed due to incomplete reporting of baseline characteristics and issues with blinding. Among the six quasi-experimental studies evaluated, five (83.33%) 13 , 15 , 16 , 18 , 21 achieved at least six positive responses, indicating an acceptable quality, while one study (16.67%) 19 fell slightly below this threshold with five positive responses. The primary challenges in these quasi-experimental studies were due to the lack of control groups, inadequate baseline comparisons, and limited follow-up reporting. Four out of five (80.00%) 17 , 20 , 22 , 23 of the cohort studies met or exceeded the criterion with at least eight positive responses, demonstrating their acceptable quality. However, one study (20.00%) 24 had a lower score due to incomplete data regarding loss to follow-up and the specifics of the interventions applied. Table 1 elaborates on the specific reasons for these assessments. Despite these concerns, the overall quality of the included studies is considered a generally acceptable risk of bias.

The summary of bias risk via the Joanna Briggs Institute assessment tools.

The impact of digital twins on health-related outcomes among patients

This review includes 12 studies that collectively assessed 45 outcomes, achieving an overall effectiveness rate of 80% (36 out of 45 outcomes), as depicted in Fig. 3a . The digital twins analyzed were coded into three functional categories: personalized health management, precision individual therapy effects, and predicting individual risks. A comprehensive analysis of the effectiveness of digital twins across these categories is provided, detailing the impact and outcomes associated with each function.

a The overall effectiveness of digital twins; b The effectiveness of personalized health management driven by digital twins; c The effectiveness of precision individualized therapy effects driven by digital twins; d The effectiveness of prediction of individual risk driven by digital twins.

The effectiveness of digital twins in personalized health management

In this review, three studies 13 , 14 , 21 employing digital twins for personalized health management reported an effectiveness of 80% (24 out of 30 outcomes), as shown in Fig. 3b . A self-control study 13 involving 15 elderly patients with diabetes, used virtual patient representations based on health information to guide individualized insulin infusion. Over 14 days, this approach improved the time in range (TIR) from 3–75% to 86–97%, decreased hypoglycemia duration from 0–22% to 0–9%, and reduced hyperglycemia time from 0–98% to 0–12%. A 1-year randomized controlled trial 14 with 319 type 2 diabetes patients, implemented personalized digital twins interventions based on nutrition, activity, and sleep. This trial demonstrated significant improvements in Hemoglobin A1c (HbA1C), Homeostatic Model Assessment 2 of Insulin Resistance (HOMA2-IR), Nonalcoholic Fatty Liver Disease Liver Fat Score (NAFLD-LFS), and Nonalcoholic Fatty Liver Disease Fibrosis Score (NAFLD-NFS), and other primary outcomes (all, P  < 0.001; Table 2 ). However, no significant changes were observed in weight, Alanine Aminotransferase (ALT), Fibrosis-4 Score (FIB4), and AST to Platelet Ratio Index (APRI) (all, P  > 0.05). A non-randomized controlled trial 21 introduced a digital twin-based Traditional Chinese Medicine (TCM) health management platform for patients with qi deficiency. It was found to significantly improve blood pressure, main and secondary TCM symptoms, total TCM symptom scores, and quality of life (all, P  < 0.05). Nonetheless, no significant improvements were observed in heart rate and BMI (all, P  > 0.05; Table 2 ).

The effectiveness of digital twins in precision individual therapy effects

Six studies 15 , 16 , 18 , 19 , 20 , 22 focused on the precision of individual therapy effects using digital twins, demonstrating a 70% effectiveness rate (7 out of 10 outcomes), as detailed in Fig. 3c . In a self-control study 15 , a data-driven approach was employed to create digital twins, generating 100 virtual patients to predict the potential tumor biology outcomes of radiotherapy regimens with varying contents and doses. This study showed that personalized radiotherapy plans derived from digital twins could extend the median tumor progression time by approximately six days and reduce radiation doses by 16.7%. Bahrami et al. 16 created 3000 virtual patients experiencing cancer pain to administer precision dosing of fentanyl transdermal patch therapy. The intervention led to a 16% decrease in average pain intensity and an additional median pain-free duration of 23 hours, extending from 72 hours in cancer patients. Another quasi-experimental study 18 created 3000 virtual patients with multiple sclerosis to assess the impact of Ocrelizumab. Findings indicated Ocrelizumab can resulted in a reduction in relapses (0.191 [0.143, 0.239]) and lymphopenic adverse events (83.73% vs . 19.9%) compared to a placebo. American researchers 19 developed a quantitative systems pharmacology model using digital twins to identify the optimal dosing for aggressive non-Hodgkin lymphoma patients. This approach resulted in at least a 50% tumor size reduction by day 42 among 3500 virtual patients. A cohort study 20 assessed the 5-year composite cardiovascular outcomes in 2173 virtual patients who were treated with spironolactone or left untreated and indicated no statistically significant inter-group differences (0.85, [0.69–1.04]). Tardini et al. 22 employed digital twins to optimize multi-step treatment for oropharyngeal squamous cell carcinoma in 134 patients. The optimized treatment selection through digital twins predicted increased survival rates by 3.73 (−0.75, 8.96) and dysphagia rates by 0.75 (−4.48, 6.72) compared to clinician decisions, with no statistical significance.

The effectiveness of digital twins in predicting individual risk

Three studies 17 , 23 , 24 employing digital twins to predict individual patient risks demonstrated a 100% effectiveness rate (5 out of 5 outcomes), as shown in Fig. 3d . A cohort study 17 used digital twins to forecast the onset age for disease-specific brain atrophy in patients with multiple sclerosis. Findings indicated that the onset of progressive brain tissue loss, on average, preceded clinical symptoms by 5-6 years among the 519 patients ( P  < 0.01). Another study 23 focused on predicting postoperative liver failure in 47 patients undergoing major hepatectomy through mathematical models of blood circulation. The study highlighted that elevated Postoperative Portal Vein pressure (PPV) and Portocaval Gradient (PCG) values above 17.5 mmHg and 13.5 mmHg, respectively, correlated with the measured values (all, P  < 0.0001; Table 2 ). These indicators were effective in predicting post-hepatectomy liver failure, accurately identifying three out of four patients who experienced this complication. Cho et al. 24 created digital twins for 50 adult female patients using facial scans and cone-beam computed tomography images to evaluate the anteroposterior position of the maxillary central incisors and forehead inclination. The analysis demonstrated significant differences in the position of the maxillary central incisors ( P  = 0.04) and forehead inclination ( P  = 0.02) between the two groups.

This systematic review outlines the effectiveness of digital twins in improving health-related outcomes across various diseases, including cancers, type 2 diabetes, multiple sclerosis, qi deficiency, heart failure, post-hepatectomy liver failure, and dental issues, at the population level. Distinct from prior reviews that focused on the technological dimensions of digital twins, our analysis shows the practical applications of digital twins in healthcare. The applications have been categorized into three main areas: personalized health management, precision individual therapy effects, and predicting individual risks, encompassing a total of 45 outcomes. An overall effectiveness of 80% was observed across these outcomes. This review offers valuable insights into the application of digital twins in precision health and supports the transition of digital twins from construction to population-wide implementation.

Digital twins play a crucial role in achieving precision health 25 . They serve as virtual models of human organs, tissues, cells, or microenvironments, dynamically updating based on real-time data to offer feedback for interventions on their real counterparts 26 , 27 . Digital twins can solve complex problems in personalized health management 28 , 29 and enable comprehensive, proactive, and precise healthcare 30 . In the studies reviewed, researchers implemented digital twins by creating virtual patients based on personal health data and using simulations to generate personalized recommendations and predictions. It is worth noting that while certain indicators have not experienced significant improvement in personalized health management for patients with type 2 diabetes and Qi deficiency, it does not undermine the effectiveness of digital twins. Firstly, these studies have demonstrated significant improvements in primary outcome measures. Secondly, improving health-related outcomes in chronic diseases is an ongoing, complex process heavily influenced by changes in health behaviors 31 , 32 . While digital twins can provide personalized health guidance based on individual health data, their impact on actual behaviors warrants further investigation.

The dual nature of medications, providing benefits yet potentially leading to severe clinical outcomes like morbidity or mortality, must be carefully considered. The impact of therapy is subject to various factors, including the drug attributes and the specific disease characteristics 33 . Achieving accurate medication administration remains a significant challenge for healthcare providers 34 , underscoring the need for innovative methodologies like computational precise drug delivery 35 , 36 , a example highlighted in our review of digital twins. Regarding the prediction of individual therapy effects for conditions such as cancer, multiple sclerosis, and heart failure, six studies within this review have reported partly significant improvements in patient health-related outcomes. These advancements facilitate the tailored selection and dosing of therapy, underscoring the ability of digital twins to optimize patient-specific treatment plans effectively.

Furthermore, digital twins can enhance clinical understanding and personalize disease risk prediction 37 . It enables a quantitative understanding and prediction of individuals by continuously predicting and evaluating patient data in a virtual environment 38 . In patients with multiple sclerosis, digital twins have facilitated predictions regarding the onset of disease-specific brain atrophy, allowing for early intervention strategies. Similarly, digital twins assessed the risk of liver failure after liver resection, aiding healthcare professionals in making timely decisions. Moreover, the application of digital twins in the three-dimensional analysis of patients with dental problems has demonstrated highly effective clinical significance, underscoring its potential across various medical specialties. In summary, the adoption of digital twins has significantly contributed to advancing precision health and restoring patient well-being by creating virtual patients based on personal health data and using simulations to generate personalized recommendations and predictions.

Recent studies have introduced various digital twin systems, covering areas such as hospital management 8 , remote monitoring 9 , and diagnosing and treating various conditions 39 , 40 . Nevertheless, these systems were not included in this review due to the lack of detailed descriptions at the population health level, which constrains the broader application of this emerging technology. Our analysis underscores the reported effectiveness of digital twins, providing unique opportunities for dynamic prevention and precise intervention across different diseases. Multiple research methodologies and outcome measures poses a challenge for quantitative publication detection. This systematic review employed a comprehensive retrieval strategy across various databases for screening articles on the effectiveness of digital twins, to reduce the omission of negative results. And four repeated publications were excluded based on authors, affiliation, population, and other criteria to mitigate the bias of overestimating the digital twins effect due to repeated publication.

However, there are still limitations. Firstly, the limited published research on digital twins’ application at the population level hinders the ability to perform a quantitative meta-analysis, possibly limiting our findings’ interpretability. We encourage reporting additional high-quality randomized controlled trials on the applicability of digital twins to facilitate quantitative analysis of their effectiveness in precision health at the population level. Secondly, this review assessed the effectiveness of digital twins primarily through statistical significance ( P -value or 95% confidence interval). However, there are four quasi-experimental studies did not report statistical significance. One of the limitations of this study is the use of significant changes in author self-reports as a criterion in these four quasi-experimental studies for identifying effectiveness. In clinical practice, the author’s self-reported clinical significance can also provide the effectiveness of digital twins. Thirdly, by focusing solely on studies published in Chinese and English, this review may have omitted relevant research available in other languages, potentially limiting the scope of the analyzed literature. Lastly, our review primarily emphasized reporting statistical differences between groups. Future work should incorporate more application feedback from real patients to expose digital twins to the nuances of actual patient populations.

The application of digital twins is currently limited and primarily focused on precision health for individual patients. Expanding digital twins’ application from individual to group precision health is recommended to signify a more extensive integration in healthcare settings. This expansion involves sharing real-time data and integrating medical information across diverse medical institutions within a region, signifying the development of group precision health. Investigating both personalized medical care and collective health management has significant implications for improving medical diagnosis and treatment approaches, predicting disease risks, optimizing health management strategies, and reducing societal healthcare costs 41 .

Digital twins intervention encompasses various aspects such as health management, decision-making, and prediction, among others 9 . It represents a technological and conceptual innovation in traditional population health intervention. However, the current content design of the digital twins intervention is insufficient and suggests that it should be improved by incorporating more effective content strategies tailored to the characteristics of the target population. Findings from this study indicate that interventions did not differ significantly in our study is from digital twins driven by personalized health management, which means that compared with the other two function-driven digital twins, personalized health management needs to receive more attention to enhance its effect in population-level. For example, within the sphere of chronic disease management, integrating effective behavioral change strategies into digital twins is advisable to positively influence health-related indicators, such as weight and BMI. The effectiveness of such digital behavior change strategies has been reported in previous studies 42 , 43 . The consensus among researchers on the importance of combining effective content strategies with digital intervention technologies underscores the potential for this approach to improve patient health-related outcomes significantly.

The applications of digital twins in precision health are mainly focused on model establishment and prediction description, with limited implementation in multi-center settings. A more robust and detailed data foundation is recommended to improve clinical decision-making and reduce the likelihood of imprecise treatments. This requires continuous updating and capturing of dynamic information by digital twins in the future, as well as the improvement of the data platform that facilitates mapping, interaction, and iterative optimization. Integrating digital twins effectively into clinical workflows can support clinical interventions, assist physicians in making informed decisions, and increase the standard of patient care 6 .

The accessibility of health data is a significant challenge for the clinical implementation of digital twins. Although the internet and information technology have significantly enhanced health data availability, health data, including information systems and electronic health records, remain heterogeneous and are difficult to share 44 . Health data often contains confidential patient information, as well as unreliable information, posing challenges for implementing digital twins in healthcare settings. The primary technology utilized in digital twins, artificial intelligence algorithms, demands high-performance hardware devices and software platforms for data analysis 45 , necessitating healthcare organizations to allocate increased investment and budget for computing infrastructure supporting digital twins’ application. Therefore, future research should be focused on the technical aspects of digital twins to resolve these challenges. The automated processing of health data using a large language model and the rapid conversion of complex natural language texts into comprehensive knowledge texts are encouraged. The development of high-performance computing technology is essential for cost-effective computing requirements, which can facilitate the application of digital twins in clinical practice 46 .

Overall, this systematic review offers a comprehensive overview of digital twins in precision health, examining their impact at the population level. The findings indicate a significant overall effectiveness rate of 80% for the measured outcomes, highlighting digital twins’ pivotal role in advancing precision health. Future research should broaden the application of digital twins across various populations, integrate proven content strategies, and implement these approaches in various healthcare settings. Such efforts will maximize the benefits of digital technologies in healthcare, promoting more precise and efficacious strategies, thereby elevating patient outcomes and improving overall healthcare experiences. While digital twins offer great promise for precision health, their broad adoption and practical implementation are still in the early stages. Development, and application are essential to unlock the full potential of digital twins in revolutionizing healthcare delivery.

This systematic review was performed following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines 47 . The protocol for this systematic review was prospectively registered on PROSPERO, which can be accessed via the following link: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42024507256 . The registered protocol underwent an update, which included polishing the title of the article, modifying the limitation of the control group and language in the inclusion/exclusion criteria, and refining the process of data synthesis and analysis to enhance that clarity and readability of this systematic review. These modifications were updated in the revision notes section of the PROSPERO.

Literature search strategy

Literature searches were conducted in PubMed, Embase, Web of Science, Cochrane Library, CINAHL, SinoMed, CNKI, and Wanfang Database, covering publications up to December 24, 2023. A comprehensive search strategy was developed using a combination of Medical Subject Headings terms and free-text terms, as detailed in Supplementary Table 2 . Furthermore, reference lists of articles and reviews meeting the inclusion criteria were reviewed for additional relevant studies.

Inclusion and exclusion criteria

The inclusion criteria for this systematic review included: 1) Population: Patients diagnosed with any diseases or symptoms; 2) Intervention: Any interventions involving digital twins; 3) Controls: Non-digital twin groups, such as standard care or conventional therapy, as well as no control group; 4) Outcomes: Health-related outcomes as the primary outcomes of interest; 5) Study design: All study designs that measured patient health-related outcomes after digital twins were included, including intervention studies and predictive cohort studies.

Initially, duplicates were removed. Exclusion criteria included: 1) Papers lacking original data, such as reviews, protocols, and conference abstracts; 2) Studies not in English or Chinese; 3) Surveys focusing on implementation and qualitative studies related to requirements. In cases of data duplication, the most comprehensive data report was included.

Study selection and Data extraction

Following the automatic removal of duplicates, two independent reviewers (MD.SHEN and SB.CHEN) conducted initial screenings of titles and abstracts against the predefined inclusion and exclusion criteria to identify potentially relevant studies. Afterward, the same reviewers examined the full texts of these shortlisted articles to confirm their suitability for inclusion. This process also involved checking the reference lists of these articles for any additional studies that might meet the criteria. Data from the included studies were systematically extracted using a pre-designed extraction form. Recorded information included the first author’s name, publication year, country of origin, type of study, sample size, study population, intervention, controls, measurements, and an appraisal of each study. Disagreements between the reviewers were resolved by consultation with a third senior reviewer (XD.DING), ensuring consensus.

Quality appraisal

The Joanna Briggs Institute (JBI) scales 48 were used to assess the quality and potential bias of each study included in the review, employing specific tools tailored to the type of study under evaluation. These tools feature response options of “yes,” “no,” “unclear,” or “not applicable” for each assessment item. For randomized controlled trials (RCTs), the JBI scale includes 13 items, with answering “yes” to at least six items indicating a high-quality study. Quasi-experimental studies were evaluated using a nine-item checklist, where five or more positive responses qualify the research as high quality. Cohort studies underwent evaluation through an 11-item checklist, with six or more affirmative responses indicating high quality. The assessment was independently carried out by two reviewers (MD.SHEN and SB.CHEN), and any disagreements were resolved through consultation with a third senior reviewer (XD.DING), ensuring the integrity and accuracy of the quality assessment.

Data synthesis and analysis

Given the heterogeneity in type of study and outcome measures, a meta-analysis was deemed unfeasible. Instead, a quantitative content analysis was employed to analyze all the selected studies 49 , 50 . Key information was extracted using a pre-designed standardized form, including the first author’s name, patient characteristics, intervention functional characteristics, measurements, results, effectiveness, and adverse events. Two reviewers (MD.SHEN and SB.CHEN) independently coded digital twin technology into three categories for descriptive analysis: personalized health management, precision individual therapy effects, and predicting individual risk, based on its functional characteristics. The Kappa statistic was applied to evaluate the inter-rater reliability during the coding process, yielding a value of 0.871, which signifies good agreement between the researchers 51 , 52 . The assessment of digital twins effectiveness was based on statistical significance ( P -value or 95% confidence interval). Outcomes with statistical significance were labeled as “resultful,” whereas those lacking statistical significance were deemed “resultless.” For quasi-experimental studies, significant changes in the authors’ self-reports were used to determine the effectiveness in the absence of reporting of statistical significance. The proportion of effectiveness was calculated as the number of “resultful” indicators divided by the total number of outcomes within each category.

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Code availability

Code sharing is not applicable to this article as no codes were generated or analyzed during the current study.

Fu, M. R. et al. Precision health: A nursing perspective. Int. J. Nurs. Sci. 7 , 5–12 (2020).

PubMed   Google Scholar  

Naithani, N., Sinha, S., Misra, P., Vasudevan, B. & Sahu, R. Precision medicine: Concept and tools. Med. J., Armed Forces India 77 , 249–257 (2021).

Article   PubMed   Google Scholar  

Payne, K. & Gavan, S. P. Economics and precision medicine. Handb. Exp. Pharmacol. 280 , 263–281 (2023).

Ielapi, N. et al. Precision medicine and precision nursing: the era of biomarkers and precision health. Int. J. Gen. Med. 13 , 1705–1711 (2020).

Article   PubMed   PubMed Central   Google Scholar  

Corral-Acero, J. et al. The ‘Digital Twin’ to enable the vision of precision cardiology. Eur. Heart J. 41 , 4556–4564 (2020).

Ferdousi, R., Laamarti, F., Hossain, M. A., Yang, C. S. & Saddik, A. E. Digital twins for well-being: an overview. Digital Twin 1 , 2022 (2022).

Article   Google Scholar  

Vallée, A. Digital twin for healthcare systems. Front. Digital health 5 , 1253050 (2023).

Elkefi, S. & Asan, O. Digital twins for managing health care systems: rapid literature review. J. Med. Internet Res. 24 , e37641 (2022).

Sun, T., He, X. & Li, Z. Digital twin in healthcare: Recent updates and challenges. Digital Health 9 , 20552076221149651 (2023).

Sheng, B. et al. Detecting latent topics and trends of digital twins in healthcare: A structural topic model-based systematic review. Digital Health 9 , 20552076231203672 (2023).

Khan, A. et al. A scoping review of digital twins in the context of the Covid-19 pandemic. Biomed. Eng. Comput. Biol. 13 , 11795972221102115 (2022).

Coorey, G. et al. The health digital twin to tackle cardiovascular disease-a review of an emerging interdisciplinary field. NPJ Digital Med. 5 , 126 (2022).

Thamotharan, P. et al. Human Digital Twin for Personalized Elderly Type 2 Diabetes Management. J. Clin. Med. 12 , https://doi.org/10.3390/jcm12062094 (2023).

Joshi, S. et al. Digital twin-enabled personalized nutrition improves metabolic dysfunction-associated fatty liver disease in type 2 diabetes: results of a 1-year randomized controlled study. Endocr. Pract. : Off. J. Am. Coll. Endocrinol. Am. Assoc. Clin. Endocrinologists 29 , 960–970 (2023).

Chaudhuri, A. et al. Predictive digital twin for optimizing patient-specific radiotherapy regimens under uncertainty in high-grade gliomas. Front. Artif. Intell. 6 , 1222612–1222612 (2023).

Bahrami, F., Rossi, R. M., De Nys, K. & Defraeye, T. An individualized digital twin of a patient for transdermal fentanyl therapy for chronic pain management. Drug Deliv. Transl. Res. 13 , 2272–2285 (2023).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cen, S., Gebregziabher, M., Moazami, S., Azevedo, C. J. & Pelletier, D. Toward precision medicine using a “digital twin” approach: modeling the onset of disease-specific brain atrophy in individuals with multiple sclerosis. Sci. Rep. 13 , 16279 (2023).

Maleki, A. et al. Moving forward through the in silico modeling of multiple sclerosis: Treatment layer implementation and validation. Comput. Struct. Biotechnol. J. 21 , 3081–3090 (2023).

Susilo, M. E. et al. Systems-based digital twins to help characterize clinical dose–response and propose predictive biomarkers in a Phase I study of bispecific antibody, mosunetuzumab, in NHL. Clin. Transl. Sci. 16 , 1134–1148 (2023).

Thangaraj, P. M., Vasisht Shankar, S., Oikonomou, E. K. & Khera, R. RCT-Twin-GAN Generates Digital Twins of Randomized Control Trials Adapted to Real-world Patients to Enhance their Inference and Application. medRxiv : the preprint server for health sciences , https://doi.org/10.1101/2023.12.06.23299464 (2023).

Jiang, J., Li, Q. & Yang, F. TCM Physical Health Management Training and Nursing Effect Evaluation Based on Digital Twin. Sci. Progr. 2022 , https://doi.org/10.1155/2022/3907481 (2022).

Tardini, E. et al. Optimal treatment selection in sequential systemic and locoregional therapy of oropharyngeal squamous carcinomas: deep Q-learning with a patient-physician digital twin dyad. J. Med. Int. Res. 24 , e29455 (2022).

Google Scholar  

Golse, N. et al. Predicting the risk of post-hepatectomy portal hypertension using a digital twin: A clinical proof of concept. J. Hepatol. 74 , 661–669 (2021).

Cho, S.-W. et al. Sagittal relationship between the maxillary central incisors and the forehead in digital twins of korean adult females. J. Personal. Med. 11 , https://doi.org/10.3390/jpm11030203 (2021).

Imoto, S., Hasegawa, T. & Yamaguchi, R. Data science and precision health care. Nutr. Rev. 78 , 53–57 (2020).

Drummond, D. & Coulet, A. Technical, ethical, legal, and societal challenges with digital twin systems for the management of chronic diseases in children and young people. J. Med. Internet Res. 24 , e39698 (2022).

Bertezene, S. The digital twin in health: Organizational contributions and epistemological limits in a context of health crisis. Med. Sci. M/S 38 , 663–668 (2022).

Johnson, K. B. et al. Precision Medicine, AI, and the Future of Personalized Health Care. Clin. Transl. Sci. 14 , 86–93 (2021).

Powell, J. & Li, X. Integrated, data-driven health management: A step closer to personalized and predictive healthcare. Cell Syst. 13 , 201–203 (2022).

Article   CAS   PubMed   Google Scholar  

Delpierre, C. & Lefèvre, T. Precision and personalized medicine: What their current definition says and silences about the model of health they promote. Implication for the development of personalized health. Front. Sociol. 8 , 1112159 (2023).

Raiff, B. R., Burrows, C. & Dwyer, M. Behavior-analytic approaches to the management of diabetes mellitus: current status and future directions. Behav. Anal. Pract. 14 , 240–252 (2021).

Ahern, D. K. et al. Behavior-based diabetes management: impact on care, hospitalizations, and costs. Am. J. Managed care 27 , 96–102 (2021).

Tyson, R. J. et al. Precision dosing priority criteria: drug, disease, and patient population variables. Front. Pharmacol. 11 , 420 (2020).

Walton, R., Dovey, S., Harvey, E. & Freemantle, N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ (Clin. Res.) 318 , 984–990 (1999).

Article   CAS   Google Scholar  

Friedrichs, M. & Shoshi, A. History and future of KALIS: Towards computer-assisted decision making in prescriptive medicine. J. Integr. Bioinform. 16 , https://doi.org/10.1515/jib-2019-0011 (2019).

Zhao, H. et al. Identifying the serious clinical outcomes of adverse reactions to drugs by a multi-task deep learning framework. Commun. Biol. 6 , 870 (2023).

Thiong’o, G. M. & Rutka, J. T. Digital twin technology: the future of predicting neurological complications of pediatric cancers and their treatment. Front. Oncol. 11 , 781499 (2021).

Sun, T., He, X., Song, X., Shu, L. & Li, Z. The digital twin in medicine: a key to the future of healthcare? Front. Med. 9 , 907066 (2022).

Sarp, S., Kuzlu, M., Zhao, Y. & Gueler, O. Digital twin in healthcare: a study for chronic wound management. IEEE J. Biomed. health Inform. 27 , 5634–5643 (2023).

Chu, Y., Li, S., Tang, J. & Wu, H. The potential of the Medical Digital Twin in diabetes management: a review. Front. Med. 10 , 1178912 (2023).

Barricelli, B. R., Casiraghi, E. & Fogli, D. A survey on digital twin: definitions, characteristics, applications, and design implications. IEEE Access 7 , 167653–167671 (2019).

Keller, R. et al. Digital behavior change interventions for the prevention and management of type 2 diabetes: systematic market analysis. J. Med. Internet Res. 24 , e33348 (2022).

Priesterroth, L., Grammes, J., Holtz, K., Reinwarth, A. & Kubiak, T. Gamification and behavior change techniques in diabetes self-management apps. J. diabetes Sci. Technol. 13 , 954–958 (2019).

Venkatesh, K. P., Raza, M. M. & Kvedar, J. C. Health digital twins as tools for precision medicine: Considerations for computation, implementation, and regulation. NPJ digital Med. 5 , 150 (2022).

Venkatesh, K. P., Brito, G. & Kamel Boulos, M. N. Health digital twins in life science and health care innovation. Annu. Rev. Pharmacol. Toxicol. 64 , 159–170 (2024).

Katsoulakis, E. et al. Digital twins for health: a scoping review. NPJ Digital Med. 7 , 77 (2024).

Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ (Clin. Res. ed.) 372 , n71 (2021).

Barker, T. H. et al. Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process. JBI Evid. Synth. 21 , 478–493 (2023).

Manganello, J. & Blake, N. A study of quantitative content analysis of health messages in U.S. media from 1985 to 2005. Health Commun. 25 , 387–396 (2010).

Giannantonio, C. M. Content Analysis: An Introduction to Its Methodology, 2nd edition. Organ. Res. Methods 13 , 392–394 (2010).

Rigby, A. S. Statistical methods in epidemiology. v. Towards an understanding of the kappa coefficient. Disabil. Rehabilitation 22 , 339–344 (2000).

Lantz, C. A. & Nebenzahl, E. Behavior and interpretation of the kappa statistic: resolution of the two paradoxes. J. Clin. Epidemiol. 49 , 431–434 (1996).

Download references

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Author information

Authors and affiliations.

School of Nursing, Peking University, Beijing, China

Mei-di Shen

Department of Plastic and Reconstructive Microsurgery, China-Japan Union Hospital, Jilin University, Changchun, Jilin, China

Si-bing Chen & Xiang-dong Ding

You can also search for this author in PubMed   Google Scholar

Contributions

MD.SHEN contributed to the data collection, analysis and the manuscript writing. SB.CHEN contributed to the data collection and analysis. XD.DING contributed to the critical revision of the manuscript as well as the initial study conception. All authors read and approved the final manuscript, and jointly take responsibility for the decision to submit this work for publication.

Corresponding author

Correspondence to Xiang-dong Ding .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary file, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shen, Md., Chen, Sb. & Ding, Xd. The effectiveness of digital twins in promoting precision health across the entire population: a systematic review. npj Digit. Med. 7 , 145 (2024). https://doi.org/10.1038/s41746-024-01146-0

Download citation

Received : 29 January 2024

Accepted : 22 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1038/s41746-024-01146-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.