literature search advantages and disadvantages

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

204k Accesses

205 Citations

118 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Become Involved |
• Give to the Library |
• Staff Directory |
• UNF Library
• Thomas G. Carpenter Library

Conducting a Literature Review

Benefits of conducting a literature review.

• Steps in Conducting a Literature Review
• Summary of the Process
• Additional Resources
• Literature Review Tutorial by American University Library
• The Literature Review: A Few Tips On Conducting It by University of Toronto
• Write a Literature Review by UC Santa Cruz University Library

While there might be many reasons for conducting a literature review, following are four key outcomes of doing the review.

Assessment of the current state of research on a topic . This is probably the most obvious value of the literature review. Once a researcher has determined an area to work with for a research project, a search of relevant information sources will help determine what is already known about the topic and how extensively the topic has already been researched.

Identification of the experts on a particular topic . One of the additional benefits derived from doing the literature review is that it will quickly reveal which researchers have written the most on a particular topic and are, therefore, probably the experts on the topic. Someone who has written twenty articles on a topic or on related topics is more than likely more knowledgeable than someone who has written a single article. This same writer will likely turn up as a reference in most of the other articles written on the same topic. From the number of articles written by the author and the number of times the writer has been cited by other authors, a researcher will be able to assume that the particular author is an expert in the area and, thus, a key resource for consultation in the current research to be undertaken.

Identification of key questions about a topic that need further research . In many cases a researcher may discover new angles that need further exploration by reviewing what has already been written on a topic. For example, research may suggest that listening to music while studying might lead to better retention of ideas, but the research might not have assessed whether a particular style of music is more beneficial than another. A researcher who is interested in pursuing this topic would then do well to follow up existing studies with a new study, based on previous research, that tries to identify which styles of music are most beneficial to retention.

Determination of methodologies used in past studies of the same or similar topics.  It is often useful to review the types of studies that previous researchers have launched as a means of determining what approaches might be of most benefit in further developing a topic. By the same token, a review of previously conducted studies might lend itself to researchers determining a new angle for approaching research.

Upon completion of the literature review, a researcher should have a solid foundation of knowledge in the area and a good feel for the direction any new research should take. Should any additional questions arise during the course of the research, the researcher will know which experts to consult in order to quickly clear up those questions.

• << Previous: Home
• Next: Steps in Conducting a Literature Review >>
• Last Updated: Aug 29, 2022 8:54 AM
• URL: https://libguides.unf.edu/litreview

• My Account |
• StudentHome |
• TutorHome |
• IntranetHome |
• Contact the OU Contact the OU Contact the OU |
• Accessibility Accessibility

Postgraduate

• International
• News & media
• Business & apprenticeships

You are here

Researcher skills, literature searching explained.

• Site Accessibility: Library Research Support

1. What is a literature search?

2. Decide the topic of your search

3. Identify the main concepts in your question

4. Choose a database

What is a literature search?

A literature search is a considered and organised search to find key literature on a topic. To complete a thorough literature search you should:

• define what you are searching for
• decide where to search
• develop a search strategy
• refine your search strategy
• save your search for future use.

For background reading or an introduction to a subject, you can do a shorter and more basic  Library search .

Use this guide to work your way through the all the stages of the literature searching process.

You should form a search question before you begin. Reframing your research project into a defined and searchable question will make your literature search more specific and your results more relevant.

Decide the topic of your search

You should start by deciding the topic of your search. This means identifying the broad topic, refining it to establish which particular aspect of the topic interests you, and reframing that topic as a question.

For example:

Broad topic:  active learning and engagement in higher education

Main focus topic:  international students and online learning

Topic stated as a question:  "What is the role of active learning in improving the engagement of international students during online learning?"

Identify the main concepts in your question

Once you have a searchable question, highlight the major concepts. For example: "What is the role of  active learning  in improving the  engagement  of  international students  during  online learning ?"

You should then find keywords and phrases to express the different concepts. For example, the concept “active learning” covers a wide range of key terms, including student-based learning, problem solving and paired discussion.

It may be useful to create a concept map. First identify the major concepts within your question and then organise your appropriate key terms.

If you are researching a medicine or health related topic then you might want to use a PICO search model. PICO helps you identify the Patient, Intervention, Comparison and Outcome concepts within your research question.

P atient: Who is the treatment being delivered to? What is happening to the patient?

I ntervention: What treatment is being delivered? What is happening to the patient?

Comparison: How much better is the procedure than another? What are the alternatives?

Outcome: How is the effect measured? What can be achieved?

List synonyms for each concept. You may wish to include variant spellings or endings (plural, singular terms). Exclude parts of the PICO that do not relate to your search question. For example, you may not be drawing any comparisons in your research.

Choose a database

Subject-specific databases are the most effective way to search for journal articles on a topic. However, you can also search the Library for common information sources, such as government documents, grey literature, patents and statistics.

Find the most appropriate databases for your subject

Databases help you to find a broad range of evidence, including peer-reviewed academic articles from all over the world, from many different publishers, and over a long time period.

Databases such as Scopus and Web of Science hold expansive records of research literature, including conference proceedings, letters and grey literature.

Many databases have links to full-text articles where the Library has a subscription.

Other information sources

Go to your  subject-specific page  to see the most appropriate information sources listed for your subject area. You may need to explore more than one subject page if your topic is multi-disciplinary.

You may find it useful to make a list of which information sources you want to search to find information for your research;  a search activity template (DOCX)  can help you do this.

• The Research Support Team
• Bibliometrics and alternative metrics
• Researcher visibility
• Advanced Literature Searching
• Reference management tools

Library Research Support team

The Open University

• Study with us
• Supported distance learning
• Funding your studies
• International students
• Global reputation
• Apprenticeships
• Develop your workforce
• Contact the OU

Undergraduate

• Arts and Humanities
• Art History
• Business and Management
• Combined Studies
• Computing and IT
• Counselling
• Creative Writing
• Criminology
• Early Years
• Electronic Engineering
• Engineering
• Environment
• Film and Media
• Health and Social Care
• Health and Wellbeing
• Health Sciences
• International Studies
• Mathematics
• Mental Health
• Nursing and Healthcare
• Religious Studies
• Social Sciences
• Social Work
• Software Engineering
• Sport and Fitness
• Postgraduate study
• Research degrees
• Masters in Art History (MA)
• Masters in Computing (MSc)
• Masters in Creative Writing (MA)
• Masters degree in Education
• Masters in Engineering (MSc)
• Masters in English Literature (MA)
• Masters in History (MA)
• Master of Laws (LLM)
• Masters in Mathematics (MSc)
• Masters in Psychology (MSc)
• A to Z of Masters degrees
• Accessibility statement
• Conditions of use
• Privacy policy
• Cookie policy
• Manage cookie preferences
• Modern slavery act (pdf 149kb)

Follow us on Social media

• Student Policies and Regulations
• Student Charter
• System Status
• Contact the OU Contact the OU
• Modern Slavery Act (pdf 149kb)

© . . .

Advantages and disadvantages of literature review

This comprehensive article explores some of the advantages and disadvantages of literature review in research. Reviewing relevant literature is a key area in research, and indeed, it is a research activity in itself. It helps researchers investigate a particular topic in detail. However, it has some limitations as well.

What is literature review?

In order to understand the advantages and disadvantages of literature review, it is important to understand what a literature review is and how it differs from other methods of research. According to Jones and Gratton (2009) a literature review essentially consists of critically reading, evaluating, and organising existing literature on a topic to assess the state of knowledge in the area. It is sometimes called critical review.

A literature review is a select analysis of existing research which is relevant to a researcher’s selected topic, showing how it relates to their investigation. It explains and justifies how their investigation may help answer some of the questions or gaps in the chosen area of study (University of Reading, 2022).

A literature review is a term used in the field of research to describe a systematic and methodical investigation of the relevant literature on a particular topic. In other words, it is an analysis of existing research on a topic in order to identify any relevant studies and draw conclusions about the topic.

A literature review is not the same as a bibliography or a database search. Rather than simply listing references to sources of information, a literature review involves critically evaluating and summarizing existing research on a topic. As such, it is a much more detailed and complex process than simply searching databases and websites, and it requires a lot of effort and skills.

Advantages of literature review

Information synthesis

A literature review is a very thorough and methodical exercise. It can be used to synthesize information and draw conclusions about a particular topic. Through a careful evaluation and critical summarization, researchers can draw a clear and comprehensive picture of the chosen topic.

Familiarity with the current knowledge

According to the University of Illinois (2022), literature reviews allow researchers to gain familiarity with the existing knowledge in their selected field, as well as the boundaries and limitations of that field.

Creation of new body of knowledge

One of the key advantages of literature review is that it creates new body of knowledge. Through careful evaluation and critical summarisation, researchers can create a new body of knowledge and enrich the field of study.

Answers to a range of questions

Literature reviews help researchers analyse the existing body of knowledge to determine the answers to a range of questions concerning a particular subject.

Disadvantages of literature review

Time consuming

As a literature review involves collecting and evaluating research and summarizing the findings, it requires a significant amount of time. To conduct a comprehensive review, researchers need to read many different articles and analyse a lot of data. This means that their review will take a long time to complete.

Lack of quality sources  

Researchers are expected to use a wide variety of sources of information to present a comprehensive review. However, it may sometimes be challenging for them to identify the quality sources because of the availability of huge numbers in their chosen field. It may also happen because of the lack of past empirical work, particularly if the selected topic is an unpopular one.

Descriptive writing

One of the major disadvantages of literature review is that instead of critical appreciation, some researchers end up developing reviews that are mostly descriptive. Their reviews are often more like summaries of the work of other writers and lack in criticality. It is worth noting that they must go beyond describing the literature.

Key features of literature review

Clear organisation

A literature review is typically a very critical and thorough process. Universities usually recommend students a particular structure to develop their reviews. Like all other academic writings, a review starts with an introduction and ends with a conclusion. Between the beginning and the end, researchers present the main body of the review containing the critical discussion of sources.

No obvious bias

A key feature of a literature review is that it should be very unbiased and objective. However, it should be mentioned that researchers may sometimes be influenced by their own opinions of the world.

Proper citation

One of the key features of literature review is that it must be properly cited. Researchers should include all the sources that they have used for information. They must do citations and provide a reference list by the end in line with a recognized referencing system such as Harvard.

To conclude this article, it can be said that a literature review is a type of research that seeks to examine and summarise existing research on a particular topic. It is an essential part of a dissertation/thesis. However, it is not an easy thing to handle by an inexperienced person. It also requires a lot of time and patience.

Hope you like this ‘Advantages and disadvantages of literature review’. Please share this with others to support our research work.

Other useful articles:

How to evaluate website content

Advantages and disadvantages of primary and secondary research

Advantages and disadvantages of simple random sampling

Last update: 08 May 2022

References:

Jones, I., & Gratton, C. (2009) Research Methods for Sports Shttps://www.howandwhat.net/new/evaluate-website-content/tudies, 2 nd edition, London: Routledge

University of Illinois (2022) Literature review, available at: https://www.uis.edu/learning-hub/writing-resources/handouts/learning-hub/literature-review (accessed 08 May 2022)

University of Reading (2022) Literature reviews, available at: https://libguides.reading.ac.uk/literaturereview/starting (accessed 07 May 2022)

Author: M Rahman

M Rahman writes extensively online and offline with an emphasis on business management, marketing, and tourism. He is a lecturer in Management and Marketing. He holds an MSc in Tourism & Hospitality from the University of Sunderland. Also, graduated from Leeds Metropolitan University with a BA in Business & Management Studies and completed a DTLLS (Diploma in Teaching in the Life-Long Learning Sector) from London South Bank University.

Related Posts

How to be a good team player, competitive advantage for tourist destinations, advantages and disadvantages of snowball sampling.

Dissertations - Skills Guide

• Where to start
• Research Proposal
• Ethics Form
• Primary Research

Literature Review

• Methodology
• Downloadable Resources
• Further Reading

What is it?

Literature reviews involve collecting information from literature that is already available, similar to a long essay. It is a written argument that builds a case from previous research (Machi and McEvoy, 2012). Every dissertation should include a literature review, but a dissertation as a whole can be a literature review. In this section we discuss literature reviews for the whole dissertation.

What are the benefits of a literature review?

There are advantages and disadvantages to any approach. The advantages of conducting a literature review include accessibility, deeper understanding of your chosen topic, identifying experts and current research within that area, and answering key questions about current research. The disadvantages might include not providing new information on the subject and, depending on the subject area, you may have to include information that is out of date.

How do I write it?

A literature review is often split into chapters, you can choose if these chapters have titles that represent the information within them, or call them chapter 1, chapter 2, ect. A regular format for a literature review is:

Introduction (including methodology)

This particular example is split into 6 sections, however it may be more or less depending on your topic.

Literature Reviews Further Reading

• << Previous: Primary Research
• Next: Methodology >>
• Last Updated: Oct 18, 2023 9:32 AM
• URL: https://libguides.derby.ac.uk/c.php?g=690330

How to undertake a literature search: enhancing your search

Affiliation.

• 1 Literature Search Specialist, Library and Archive Service, Royal College of Nursing, London.
• PMID: 32324469
• DOI: 10.12968/bjon.2020.29.8.481

This article follows on from a previous article on how to carry out a literature search (Watson, 2020) and looks at how you can enhance your search by going beyond journal databases to using search engines, websites and grey literature sources. Ways to evaluate the resources you find, the use of critical appraisal tools and factors to consider when presenting your results are also discussed.

Keywords: Critical appraisal; Evaluating resources; Grey literature; Literature search; Search engines.

• Databases, Bibliographic*
• Information Storage and Retrieval / methods*
• Periodicals as Topic
• Search Engine*

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Open access
• Published: 25 February 2011

In search of the right literature search engine(s)

• Akhilesh Bajpai 1 ,
• Sravanthi Davuluri 1 ,
• Haritha Haridas 1 ,
• Greta Kasliwal 1 ,
• Deepti H 2 ,
• Sreelakshmi KS 1 ,
• Darshan Chandrashekar 2 ,
• Pranami Bora 1 ,
• Mohammed Farouk 1 ,
• Neelima Chitturi 1 ,
• Samudyata V 2 ,
• ArunNehru KP 3 &
• Kshitish Acharya 2  

Nature Precedings ( 2011 ) Cite this article

1759 Accesses

7 Citations

3 Altmetric

Metrics details

Collecting scientific publications related to a specific topic is crucial for different phases of research, health care and ‘effective text mining’. Available bio-literature search engines vary in their ability to scan different sections of articles, for the user-provided search terms and/or phrases. Since a thorough scientific analysis of all major bibliographic tools has not been done, their selection has often remained subjective. We have considered most of the existing bio-literature search engines (http://www.shodhaka.com/startbioinfo/LitSearch.html) and performed an extensive analysis of 18 literature search engines, over a period of about 3 years. Eight different topics were taken and about 50 searches were performed using the selected search engines. The relevance of retrieved citations was carefully assessed after every search, to estimate the citation retrieval efficiency. Different other features of the search tools were also compared using a semi-quantitative method.

The study provides the first tangible comparative account of relative retrieval efficiency, input and output features, resource coverage and a few other utilities of the bio-literature search tools. The results show that using a single search tool can lead to loss of up to 75% relevant citations in some cases. Hence, use of multiple search tools is recommended. But, it would also not be practical to use all or too many search engines. The detailed observations made in the study can assist researchers and health professionals in making a more objective selection among the search engines. A corollary study revealed relative advantages and disadvantages of the full-text scanning tools.*Conclusion*While many studies have attempted to compare literature search engines, important questions remained unanswered till date. Following are some of those questions, along with answers provided by the current study:a) Which tools should be used to get the maximum number of relevant citations with a reasonable effort? ANSWER: _Using PubMed, Scopus, Google Scholar and HighWire Press individually, and then compiling the hits into a union list is the best option. Citation-Compiler (http://www.shodhaka.com/compiler) can help to compile the results from each of the recommended tool._b) What is the approximate percentage of relevant citations expected to be lost if only one search engine is used? ANSWER: _About 39% of the total relevant citations were lost in searches across 4 topics; 49% hits were lost while using PubMed or HighWire Press, while 37% and 20% loss was noticed while using Google Scholar and Scopus, respectively._c) Which full text search engines can be recommended in general? ANSWER: HighWire Press and Google Scholar. d) Among the mostly used search engines, which one can be recommended for best precision? ANSWER: EBIMed. e) Among the mostly used search engines, which one can be recommended for best recall? ANSWER: Depending on the type of query used, best recall could be obtained by HighWire Press or Scopus.

Similar content being viewed by others

Augmenting large language models with chemistry tools

Interviews in the social sciences

Principal component analysis

Article pdf, author information, authors and affiliations.

Institute of Bioinformatics and Applied Biotechnology (IBAB), Biotech Park, Electronic City, Bangalore, 560100, Karnataka state, India

Akhilesh Bajpai, Sravanthi Davuluri, Haritha Haridas, Greta Kasliwal, Sreelakshmi KS, Pranami Bora, Mohammed Farouk & Neelima Chitturi

Institute of Bioinformatics and Applied Biotechnology (IBAB), Biotech Park, Electronic City, Bangalore - 560100, Karnataka state, India, www.ibab.ac.in / Shodhaka Life Sciences Pvt. Ltd., IBAB, Biotech Park, Electronic City, Bangalore, 560100, Karnataka

Deepti H, Darshan Chandrashekar, Samudyata V & Kshitish Acharya

Shodhaka Life Sciences Pvt. Ltd., IBAB, Biotech Park, Electronic City, Bangalore, 560100, Karnataka state, India

ArunNehru KP

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kshitish Acharya .

Rights and permissions

Creative Commons Attribution 3.0 License .

Reprints and permissions

About this article

Cite this article.

Bajpai, A., Davuluri, S., Haridas, H. et al. In search of the right literature search engine(s). Nat Prec (2011). https://doi.org/10.1038/npre.2011.2101.3

Download citation

Received : 25 February 2011

Accepted : 25 February 2011

Published : 25 February 2011

DOI : https://doi.org/10.1038/npre.2011.2101.3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• literature search
• text mining
• search engines
• search tools
• search programs
• publication search
• Highwire Press
• google scholar
• citation retrieval
• literature mining
• bibliography
• publications

This article is cited by

Go2pub: querying pubmed with semantic expansion of gene ontology terms.

• Charles Bettembourg
• Christian Diot
• Olivier Dameron

Journal of Biomedical Semantics (2012)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Advanced literature searching skills: Introduction

Introduction.

• Starting your search
• Choosing keywords
• Expert searching
• Managing your results
• Referencing and software to help

• How to use Internet sources more effectively and identify high-quality information sources, such as a university library's expert databases.   
• Planning an effective search – choosing keywords, considering synonyms, alternative spellings, using search syntax, Boolean logic and truncation.  
• Working with the results – filter, sort and evaluate suitability, find new keywords and amend searches – taking a strategic approach.   
• Managing search results and exporting references e.g. creating saved searches and alerts; exporting references to bibliographic software.  
• Creating and managing citations and references – manually creating references or using 'Cite While You Write' (CWYW) EndNote with Word.

Let's begin with an introduction to the literature searching process and consider the importance of defining your research question.  

What is a literature search?

Define your research question(s)

Take, for example, this topic:   Are biofuels the answer to falling oil reserves?

You  could  type this sentence into a database search box, but that is seldom helpful, as the sentence may not contain the most appropriate keywords. Also this single sentence is unlikely to encompass everything that you want to find out. You need to break down the topic into a number of separate questions and then look for the answers. For this example here are some of the questions you could ask:

• What is a biofuel?  
• How are they made?
• How much of our fuel is already from biofuel (market share)?
• Could we make enough to replace oil and/or gas?
• What are the advantages and disadvantages of using biofuels compared with oil and gas?
• Could we use biofuels for transport?
• What is UK government policy relating to biofuels?

You  may  find the answers to all of these questions using a single search engine such as Google Scholar or a single Library database, but you are more likely to succeed if you match each question to relevant resources .

Let's move on now to 'starting your search'

• Next: Starting your search >>
• Last Updated: Sep 22, 2021 2:28 PM
• URL: https://library.bath.ac.uk/advanced-research-skills-for-literature-searching

Google Scholar

• Google Scholar Access
• Advanced Settings
• Advantages and Disadvantages

Advantages & Disadvantages

Google Scholar can be useful when searching:

• "Grey literature" such as conference proceedings
• Across multiple disciplines
• For multiple types of documents (articles, proceedings, books)

Google Scholar can also help you quickly identify who is publishing in your area of research.

Disadvantages

• not knowing what is included or not included
• not knowing when it's updated
• many older articles
• can't browse by title
• no way to limit search results

If you need advanced searching features or recent articles, try one of the databases the library subscribes to. Many are included in the other tabs in this subject guide. Or you can browse all of our databases by subject .

Ask yourself:

• When should you use Google Scholar, and when should you use a different search engine?
• When would it be more productive to use a database that the Library subscribes to for students, faculty, and staff?
• << Previous: Advanced Settings
• Last Updated: Jan 23, 2024 11:44 AM
• URL: https://pacificu.libguides.com/GoogleScholar

Pacific University Libraries | 2043 College Way | Forest Grove, OR 97116 | 503-352-1400

Copyright © Pacific University, all rights reserved

Mission & Vision | Policies | Jobs | Staff

Journal of Materials Chemistry A

Recent advances in aqueous and non-aqueous alkali metal hybrid ion capacitors.

Alkali metal hybrid ion capacitors (AHICs) combine the advantages of batteries and supercapacitors and balance the disadvantages of both devices, which allows high energy and power densities and long cycle life to be maintained simultaneously. The paper first summarizes the working principles, classification, scientific issues, and performance evaluation methods in AHICs. Then, orientated towards performance and structural optimization, various battery-type and capacitor-type materials in aqueous and non-aqueous systems are discussed, focusing on their structure-property relationships, charge storage mechanisms, and electrochemical performance, respectively. Additionally, the common prelithiation/presodiation/ prepotassiation methods used in AHICs are outlined. Finally, some challenges and future research directions in this field are presented.

• This article is part of the themed collection: Journal of Materials Chemistry A Recent Review Articles

Article information

Download citation, permissions.

Z. Jia, S. Hou, J. Peng, X. Wu, W. Tang, W. Sun, S. Lv, X. yuan, L. Liu and Y. Wu, J. Mater. Chem. A , 2024, Accepted Manuscript , DOI: 10.1039/D4TA02060J

To request permission to reproduce material from this article, please go to the Copyright Clearance Center request page .

If you are an author contributing to an RSC publication, you do not need to request permission provided correct acknowledgement is given.

If you are the author of this article, you do not need to request permission to reproduce figures and diagrams provided correct acknowledgement is given. If you want to reproduce the whole article in a third-party publication (excluding your thesis/dissertation for which permission is not required) please go to the Copyright Clearance Center request page .

Read more about how to correctly acknowledge RSC content .

Social activity

Search articles by author.

This article has not yet been cited.

Colonial Architecture: Everything You Need to Know

By Morgan Goldberg

The term colonial architecture broadly refers to the buildings and homes that were constructed during a period of colonization in a given country. American colonial architecture emerged in the United States when Northern European settlers established colonies here in the 17th and 18th centuries. “They adapt[ed] their architectural styles to local conditions, materials, and cultural influences,” says Alan Clark of Alan Clark Architects in Atlanta.

Recognizable for their symmetry, rectangular shape, and sash windows, colonial houses define many of the oldest neighborhoods in the US. These historic areas are clustered along the East Coast, where the original 13 colonies once were, but colonial-style homes can be found all over the country, thanks to the colonial revival boom of the late 1800s and early 1900s. Colonial might just be America’s most iconic architectural style—which is one of many reasons you should learn more about it.

Here’s everything you need to know about colonial architecture.

What is the history of colonial architecture?

The earliest examples of colonial architecture in the US date back to 1626 and were simple one-story buildings with gabled roofs and few rooms, says Craig Intinarelli of Kroeger Intinarelli Architects in Katonah, New York. “These first domestic structures were almost medieval due to their transient nature,” he adds. “The principal materials of these early colonial archetypes were wood due to the abundance of forests and the immediate necessity to establish a safe haven in the new world.”

As resources became more accessible, living conditions improved and thus the classic colonial architecture style was born. “The European influence on Mid-Atlantic and New England colonial architecture would continue as grander homes developed,” says Intinarelli. “Much would come from British and French interpretations of antiquity found in Roman and Greek architecture. Homes would also be influenced by traditional pattern books from Europe illustrating the use of profiled moldings, paneled walls, and millwork.”

According to Intinarelli, “the strongest influence came from the American gentry’s embrace of the works of Andrea Palladio and the French neoclassicists. Elements of the colonial style would include a strong central axis where the front door was placed, symmetrical plans with columnar, temple-like porticos, and decorative cornices and architraves.”

Starting in the late 19th century, colonial-style architecture saw a resurgence. Such homes are aptly called colonial revival. “Colonial architecture is officially on the east coast in the original colonies—hence its name, but the elements are repeated and still used today all over,” says Maren Baker of Maren Baker Design . “I live in Boise in a colonial revival house built in 1925.”

What are the defining elements and characteristics of colonial architecture?

• Rectangular shape
• Two stories
• Wood, brick, or stone facade
• Central door
• Gable roofs
• Multi-paned sash windows
• Dormer windows
• Traditional, closed-off floor plan
• Formal entryway
• Big, central staircase
• Long hallways
• Living spaces, like the kitchen and dining room, on the main level
• Bedrooms on the second level
• Evenly-divided rooms

What are the different styles of colonial architecture?

As its name suggests, British colonial architecture is a result of English colonization in the US, from Massachusetts to Virginia and beyond. “It often combines Georgian and Victorian elements with symmetrical facades, sash windows, and decorative details,” shares Clark. Classical Greek and Roman influences, like pilasters and columns, were also common.

Dutch colonial architecture “features gambrel roofs, dormer windows, flared eaves, and a central chimney,” says Clark. “It’s primarily found in former Dutch colonies like Indonesia, South Africa, and parts of the United States, like New York and New Jersey.” Dutch colonial houses are known to resemble barns.

Louisiana is home to the majority of America’s French colonial architecture, which is distinguished by “elements like steeply pitched roofs, large verandas, and French-inspired detailing,” Clark notes. The big, wraparound porches he mentions are often on both levels of the home, which may also have tall, narrow windows and French double doors.

According to Clark, Saltbox is a “style originating in New England, featuring asymmetrical rooflines with long, pitched roofs in the back and shorter roofs in the front.” Saltbox homes were built by settlers using local timber and got their name from their “resemblance to wooden lidded boxes in which salt was once kept.”

Southern colonial architecture is native to the southeastern US and often includes “large verandas, tall windows, and grand, plantation-style designs,” Clark says. Colonnades tend to extend across the facades, while ornate moldings cover the interiors. Southern colonial homes are frequently set back from the main roads and accessed by long driveways.

The rustic aesthetic of Spanish colonial architecture is “characterized by thick adobe walls, courtyards, and ornate detailing influenced by Spanish and Moorish architecture,” Clark explains. Red clay roof tiles and wooden beams are also typical. Spanish colonial houses and their derivatives are ubiquitous in Florida, the Southwest, and California.

What are the advantages and disadvantages of colonial architecture?

Clark, Intinarelli, and Baker all agree that one of the biggest advantages of colonial architecture is its classic, timeless aesthetic that’s rooted in symmetry. “It can result in some really beautiful curb appeal,” Baker considers. “[Colonial homes] are classics in that they don’t really go out of style. The elements have been used for so long that you can feel safe in knowing you won’t be staring at a house that was so clearly made in a bad architecture period.”

Historic charm, sturdy construction, and sensible layout are among the other perks of living in a colonial house. “Many of our clients take on these homes out of passion for their history, patina, richness of material, and antique scale,” says Intinarelli. Clark adds, “Many colonial homes were built with durable materials—brick, stone, timber frame—and true craftsmanship, offering longevity.” And Baker notes, “I find this style of home to make sense—you aren’t left searching around for where the bathroom is because they are laid out in an orderly fashion.”

For Baker, the biggest disadvantage of colonial architecture is its small scale. “If you live in a home from the late 1700s or 1800s that would be considered colonial, I would say the biggest con would be ceiling height and overall size,” she opines. “The homes were obviously much smaller than they are now. Ceilings were low, which can cause issues with HVAC systems and lighting options. Kitchens were also small and not very well located in the home. If you are looking for an open floor plan and large kitchen, you are not going to find that in a colonial home.”

Meanwhile, Clark and Intinarelli cite energy inefficiency, i.e. the lack of modern insulation and the presence of old windows, which can lead to higher utility costs, break modern energy codes, and be challenging to remedy, as a major downside. “Some of the pitfalls of restoring and modifying Colonial homes of a certain vintage is balancing the historic detailing and current energy code requirements, specifically, the rehabilitation versus installing of new windows and doors,” Intinarelli explains. Additionally, “older colonial homes may require more maintenance and upkeep, especially in terms of preserving historic features,” Clark says.

By Adrian Madlener

By Alia Akkam

By Rachel Davies

By Charlotte Collins

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review

Chris cooper.

1 PenTAG, University of Exeter Medical School, Exeter, England

Andrew Booth

2 HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, England

Nicky Britten

3 Institute of Health Research, University of Exeter Medical School, Exeter, England

Ruth Garside

4 European Centre for Environment and Human Health, University of Exeter Medical School, Truro, England

Associated Data

Not applicable

The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged. Numerous studies have demonstrated their potential in identifying studies or study data that would have been missed by bibliographic database searching alone.

What is less certain is how supplementary search methods actually work, how they are applied, and the consequent advantages, disadvantages and resource implications of each search method.

The aim of this study is to compare current practice in using supplementary search methods with methodological guidance.

Four methodological handbooks in informing systematic review practice in the UK were read and audited to establish current methodological guidance.

Studies evaluating the use of supplementary search methods were identified by searching five bibliographic databases. Studies were included if they (1) reported practical application of a supplementary search method (descriptive) or (2) examined the utility of a supplementary search method (analytical) or (3) identified/explored factors that impact on the utility of a supplementary method, when applied in practice.

Thirty-five studies were included in this review in addition to the four methodological handbooks. Studies were published between 1989 and 2016, and dates of publication of the handbooks ranged from 1994 to 2014.

Five supplementary search methods were reviewed: contacting study authors, citation chasing, handsearching, searching trial registers and web searching.

Conclusions

There is reasonable consistency between recommended best practice (handbooks) and current practice (methodological studies) as it relates to the application of supplementary search methods.

The methodological studies provide useful information on the effectiveness of the supplementary search methods, often seeking to evaluate aspects of the method to improve effectiveness or efficiency. In this way, the studies advance the understanding of the supplementary search methods. Further research is required, however, so that a rational choice can be made about which supplementary search strategies should be used, and when.

The purpose and contribution of supplementary search methods in systematic reviews are increasingly acknowledged. Numerous studies have demonstrated their potential in identifying studies or study data that would have been missed by bibliographic database searching alone [ 1 – 8 ].

It is commonly believed that the inclusion of supplementary search methods adds value to the process of comprehensive study identification in systematic reviews. The methodological handbooks for systematic review methodology, such as The Cochrane or CRD Handbooks, provide practical (although limited) instruction on how to undertake each supplementary search method, and empirical studies have evaluated the effectiveness and efficiencies of these search methods. What is perhaps less certain is how supplementary search methods actually work, and what the advantages, disadvantages and resource implications of each search method are.

The aim of this study is to compare empirical studies of supplementary search techniques to the recommendations in methodological handbooks.

By re-considering the best practice guidance of methodological handbooks for systematic review, and reviewing how this guidance has been interpreted and evaluated within current practice by authors, this study seeks to identify claimed advantages, claimed disadvantages and resource requirements of using supplementary search methods.

The research question for this study

The research question for this study is how do empirical studies of supplementary search techniques compare to the recommendations in review methodology handbooks?

This study aims to produce a structured methodological overview of methodological handbooks on the conduct of supplementary searches in systematic reviews. In addition, we reviewed studies that report on the utility and practice of supplementary searches. In order to identify this literature, a systematic approach to study identification, study selection and data extraction was used, which is set out below. These two types of literature—handbooks and practical explorations of applying supplementary search strategies—were then compared. The advantages, disadvantages and resource requirements of each method were evaluated.

Study identification

We selected the following methodological handbooks as the most influential handbooks in informing systematic review practice in the UK. The current editions of each handbook were read and audited to establish current methodological guidance:

• The Cochrane Handbook for Systematic Reviews of Interventions (version 5.10, March 2011) [ 9 ];
• Systematic Reviews: CRD’s guidance for undertaking review in health care (2009) [ 10 ];
• The Campbell Information Retrieval Methods Group guide to information retrieval (October 2009) [ 11 ]; and
• The NICE manual to developing NICE guidelines (October 2014) [ 12 ].

The following five search methods, supplementary to database searches, were identified from these handbooks:

• Contacting study authors or experts

Citation chasing

Handsearching.

• Trial register searching
• Web searching.

In order to compare the existing handbook guidance to current practice, we identified studies that describe and/or evaluate how these methods are applied in practice. Studies were identified by searching five bibliographic databases: MEDLINE, EMBASE, LISTA, ASSIA and Web of Science in July 2016. Forward citation chasing was applied to studies meeting inclusion at full text, and the bibliographies were appraised. Tables of included studies were examined if aggregated within systematic reviews. The search syntax for bibliographic database searching is included as a supplementary file.

Study selection

Studies were downloaded into Endnote X6 where manual de-duplication was performed. Studies were single screened by CC using the inclusion criteria below:

Inclusion criteria

For inclusion in this review, a study was required to:

• (i) Report practical application of a supplementary search method (descriptive)
• (ii) Examine the utility of a supplementary search method (analytical)
• (iii) Identify/explore factors that impact on the utility of a supplementary method when applied in practice

Exclusion criteria

The following studies were excluded:

• i) Studies reporting the use of supplementary search methods but not discussing the practical application of the method (such as listing their use to identify studies in a systematic review, i.e. ‘we handsearched the following journals’)
• ii) Studies reported as abstracts or on-going studies
• iii) Systematic reviews or reviews, in which case tables of included studies were examined to identify eligible primary studies

Data extraction

The following data were extracted: citation details, study design, claimed advantages, claimed disadvantages and resource requirements.

Thirty-five studies were included in this review in addition to the four methodological handbooks. Studies were published between 1989 and 2016, and handbooks were published between 1994 and 2014. Table  1 summarises which studies cited which handbooks as their source of methodological reference. The handbooks audited for this study cited only three studies: Eysenbach et al. (2001) was cited in The Cochrane Handbook, Hetherington et al. (1989) was cited in The Cochrane Handbook and The Campbell Handbook and Papaioannou et al. (2010) was cited in The Campbell Handbook (Table  1 ).

Studies citing handbooks: handbooks citing studies

Studies in italics are ones cited by the handbooks identified using the symbol ‘**’ for informing their guidance

NR not reported

The results were categorised by the supplementary search methods and reported in five domains: (1) what the method is used for, (2) what the evidence says, (3) claimed advantages, (4) claimed disadvantages and (5) resource requirements. A summary of these results is presented in Table  2 .

Overview of results

Contacting study authors

The handbooks focus on identifying contact details and considering how to request studies or study data [ 9 , 10 , 13 ]. The studies evaluate the effectiveness of methods to make contact and elicit a response. Six empirical studies were included [ 6 , 14 – 18 ].

What it is used for

It is used for identifying unpublished or on-going studies [ 10 ]; identifying missing, incomplete, discordant or unreported study data, or completed but unpublished studies [ 9 , 13 , 14 , 16 – 18 ]; and asking study authors (or topic experts) to review a list of studies included at full text in a review, to see whether any studies had been inadvertently overlooked [ 9 , 10 ].

What the evidence says

Two handbooks and one study provided detail on identifying contact details [ 6 , 9 , 10 ]. The Cochrane Handbook suggests that review authors should contact the original investigators, identifying contact details from study reports, recent publications, staff listings or a search of the internet [ 13 ]. Colleagues, relevant research organisations and specialist libraries can also be a valuable source of author information and contact details [ 9 , 10 ]. A study by McManus et al. used a questionnaire, primarily to request study data or references, but also to ask recipients to recommend the names of other authors to contact [ 6 ]. A study by Hetherington et al. contacted authors and experts by letter in an attempt to identify unpublished trials [ 17 ].

Two studies reported using a multi-stage protocol to contact authors and request data: Selph et al. devised and followed a protocol that used both e-mail and telephone contact with the corresponding authors at defined stages over a period of 15 days [ 16 ]. Gibson et al. devised a similar protocol, although focused on e-mail contact, targeting first the corresponding authors and finally the last author and statisticians by e-mail and then telephone (statisticians were contacted due to the specific focus of the case study) [ 14 ]. Selph et al. contacted 45 authors and 28 (62%) provided study data [ 16 ], and Gibson et al. contacted 146 authors and 46 (31.5%) provided study data [ 14 ].

Two studies claimed that e-mail was considered an effective method of contact [ 14 , 15 ]. O’Leary reported a response rate of 73% using e-mail contact, finding that more responses were obtained from an institutional address compared to a hotmail address (86 vs 57%, p  = 0.02) [ 15 ]. Conversely, Reveiz et al. achieved a 7.5% response rate from contacting 525 study authors to identify RCTs but identified 10 unpublished RCTs and links to 21 unregistered and on-going RCTs [ 18 ]. Gibson et al. found that e-mail was most likely to receive a reply when compared to letter (hazard ratio [HR] = 2.5; 95% confidence interval [CI] = 1.3–4.0) but that a combined approach of letter and e-mail, whilst generating a higher response rate, was not statistically different from e-mail alone (73 vs 47%, p  = 0.36. One hundred forty-six authors were contacted overall and 46 responded) [ 14 ].

Hetherington et al. sent letters to 42,000 obstetricians and paediatricians in 18 countries in an attempt to identify unpublished controlled trials in perinatal medicine [ 17 ]. Responses were received from 481 individuals indicating they would provide details concerning unpublished studies, and 453 questionnaires were completed and returned which identified 481 unpublished trials [ 17 ].

Chapter Seven of The Cochrane Handbook offers guidance on how to set out requests for studies or study data when contacting study authors [ 13 ]. The guidance suggests considering if the request is open-ended, or seeking specific information, and whether (therefore) to include a (uncompleted or partially completed) data collection form or request specific data (i.e. individual patient data) [ 13 ]. McManus et al. evaluated the use of a questionnaire to identify studies, study data and the names of relevant authors to contact for a systematic review [ 6 ]. The questionnaire resulted in the identification of 1057 references unique to the review, but no unpublished data were offered [ 6 ].

Two handbooks recommend submitting a list of included studies to authors [ 9 ] or topic experts [ 10 ] to identify any potentially missing studies. The Cochrane Handbook suggests including the review’s inclusion criteria as a guide to authors [ 9 ].

Claimed advantages

Five studies claimed that identifying additional published or unpublished studies, study data or references is possible by contacting study authors [ 6 , 14 , 16 – 18 ]. McManus et al. identified 23 references (out of 75 included in the review overall) by contacting study authors [ 6 ]; Reveiz et al. identified 10 unpublished RCTs and 21 unregistered or on-going RCTs [ 18 ]; two studies stated that they identified additional study data but did not separate their findings from contacting study authors from other methods of study identification [ 14 , 16 ]; and Hetherington et al. identified 481 unpublished trials by contacting 42,000 obstetricians and paediatricians in 17 countries [ 17 ].

O’Leary found that more detailed study information was provided as a result of contacting study authors [ 14 ].

Claimed disadvantages

The CRD handbook claims that contacting authors/experts offers no guarantee of obtaining relevant information [ 10 ]. Selph et al. found that, whilst identifying additional studies or study data is possible, contacting study authors is challenging and, despite extensive effort, missing data remains likely [ 16 ].

Hetherington et al. claimed that methodologically sound trials were not reported through author contact, even by the investigators responsible for them. This was attributed, anecdotally, to the possibility that the trials yielded results that the investigators found disappointing [ 17 ].

Reveiz et al. reported low response rates. Of 525 study authors contacted, only 40 (7.5%) replied [ 18 ].

Two studies and one handbook claimed that contacting authors/experts is time consuming for researchers [ 10 , 14 , 16 ]. Selph et al. noted that this method is time consuming for the study authors too, who must identify the data requested [ 16 ].

Gibson et al. claimed that contacting authors/experts may be less successful for older studies, given the increased possibility that authors’ contact details are out of date [ 14 ]. Gibson et al. reported a 78% (CI = 0.107–0.479) reduction in the odds of response if the article was 10 years old or older [ 14 ].

Resource requirements

Gibson et al. claimed that additional resources were required to undertake author contact [ 14 ]. No specific details of the costs or time implications were recorded.

Gibson et al. recorded the duration between the information request and response [ 14 ]. This averaged 14 ± 22 days (median = 6 days) and was shortest for e-mail (3 ± 3 days; median = 1 day) compared to e-mail plus letter (13 ± 12 days; median = 9 days) and letter only (27 ± 30 days; median = 10 days) [ 14 ].

Selph et al. reported that all authors who provided data did so by the third attempt, suggesting that repeated attempts to elicit studies or study data may be ineffective [ 16 ].

The handbooks provide a brief overview of the method and list some of the tools commonly used [ 9 , 10 ]. The studies typically evaluate the effectiveness of the tools used to undertake the search methods. Nine studies assessing the use of citation chasing were included [ 1 – 3 , 19 – 24 ].

It is used for identifying further studies, and clusters or networks of studies, that cite or are cited by a primary study [ 10 ].

Two studies provided detail on the application of the search method [ 1 , 3 ]. The studies noted that backward citation searching is undertaken by reviewing bibliographies of relevant or included studies and forward citation chasing is undertaken by checking if a study, already known to be relevant, has since been cited by another study [ 1 , 3 ].

Three tools for electronic citation searching dominate the studies: Web of Science, Scopus and Google Scholar. The first two are subscription databases, and Google Scholar is presently free [ 19 ].

Four studies claimed that an advantage of citation chasing is that it is not limited by keywords or indexing as is bibliographic database searching [ 2 , 3 , 20 , 21 ]. Accordingly, four studies claimed the following advantages: Robinson et al. claimed that a small initial number of studies can create a network [ 21 ]; Hinde et al. claimed that citation searching can help inform researchers of parallel topics that may be missed by the focus of bibliographic database searches [ 2 ]; Janssens and Gwinn claimed that citation searching may be valuable in topic areas where there is no consistent terminology, so searches focus on links between studies rather than keywords [ 20 ]; and Papaioannou et al. reported that citation searching facilitated ‘serendipitous study identification’ due to the unstructured nature of citations [ 3 ].

One study appraised the quality of the studies identified through citation searching (and by other search methods) [ 3 ]. Papaioannou et al. reported that citation searching identified high-quality studies in their case study, although they do not define which quality appraisal tool was used to appraise study quality, so it is not clear if this observation is empirically derived [ 3 ].

Three studies stated that citation searching is reliant on the currency, accuracy and completeness of the underlying citation network [ 1 , 21 , 22 ]. Levay et al. identified ‘linking lag’, namely the delay between a study being cited and the citation being recorded in a citation database, which impacts on the currency of results [ 1 ]; Janssens and Gwinn stated that the accuracy and efficiency of citation searching depends on study authors citing studies, which means that selective citation of studies could cause relevant studies to be missed in citation searching [ 20 ]; Robinson et al. reported limited returns from citation searching where ‘broken citation links’ created ‘island’ studies which makes for incomplete citation networks and study identification [ 21 ].

Two studies questioned the efficiency of citation searching [ 2 , 22 ]. Wright et al. screened 4161 studies to identify one study (yield rate of 0.0002) [ 22 ], and Hinde et al. screened 4529 citations to identify 76 relevant studies (yield rate of 0.0168) [ 2 ]. Wright et al. specifically recorded the time to undertake citation chasing in their study (discussed below in resource use), [ 22 ] whereas Hinde et al. did not report the time taken to search but state that the search was ‘very time consuming’ [ 2 ].

Two studies claimed that replicability of citation searching strategies could be affected by the choice of the tools used [ 1 , 24 ]. Levay et al. questioned the replicability of Google Scholar, since search returns are controlled by Google’s algorithm, meaning that the results returned will change over time and cannot be replicated [ 1 ]. Bramer et al. found reproducibility of citation searching to be low, due to inaccurate or incomplete reporting of citation search strategies by study authors [ 24 ].

Two studies recorded the time taken to citation search, and one study commented on the time needed [ 1 , 3 , 22 ]. Levay et al. reported that citation searching the same 46 studies in Web of Science and Google Scholar took 79 h (Web of Science = 4 h and Google Scholar 75 h) to identify and de-duplicate 783 studies (Web of Science = 46 studies and Google Scholar = 737 studies) [ 1 ]. Wright et al. reported that citation chasing the same 40 studies in Web of Science, Medline, Google Scholar and Scopus took 5 days in total (2 days to download 1680 results from Google Scholar; 1 day to download 2481 results from Web of Science, Scopus and Medline; and 2 days to screen all the studies) [ 22 ]. Both studies commented on the administrative burden of exporting studies from Google Scholar which accounted for the majority of time searching in both cases [ 1 , 22 ]. Conversely, Papaioannou et al. claimed reference tracking and citation searching to be minimally time intensive, yielding unique and high-quality studies. The number of studies citation chased, the time taken to search and the tool used to appraise study quality were not reported [ 3 ].

One study provided data on the costs involved in citation chasing [ 1 ]. Levay et al. reported that the staff time to search Web of Science for 4 h cost between £88 and £136 and the 75 h to search Google Scholar cost between £1650 and £2550, based on staff grades ranging from £22–£34 per hour (all UK Sterling: 2012) [ 1 ].

The handbooks focus on where to handsearch [ 9 , 10 ], and they provide guidance on who should do this [ 9 ]. The studies have a similar focus but they have sought to evaluate effectiveness compared with other search methods [ 25 – 28 ] as well as to evaluate the effectiveness and/or the efficiency of handsearchers in identifying studies [ 29 , 30 ]. Twelve studies were included [ 25 – 36 ].

It is used for ensuring the complete identification of studies or publication types that are not routinely indexed in, or identified by, searches of bibliographic databases, including recently published studies [ 10 ].

Handsearching involves a manual, page-by-page, examination of the entire contents of relevant journals, conference proceedings and abstracts [ 9 , 10 , 27 , 31 ].

Two handbooks and six studies provide detail on selecting journals to handsearch [ 9 , 10 , 25 , 27 , 30 – 33 ]. Three strategies were identified, as set out below.

Using databases (or database search results) to identify journals to handsearch

The handbooks suggest that bibliographic databases can be used to identify which journals to handsearch [ 9 , 10 ]. The Cochrane Handbook, with its focus on identifying studies reporting randomised controlled trails (RCTs), suggests that searches of The Cochrane CENTRAL database, MEDLINE and EMBASE can be used to identify journals that return the greatest number of studies by study design in the relevant topic area of research [ 9 ]. Variations of this approach to selecting journals to handsearch were utilised in three studies [ 25 , 30 , 31 ]. The CRD Handbook suggests analysing the relevant results of the review’s bibliographic database searches in order to identify journals that contain the largest number of relevant studies [ 10 ].

Handsearching journals not indexed in bibliographic databases

The Cochrane Handbook suggests that journals not indexed in MEDLINE or EMBASE should be considered for handsearching [ 9 ]. A study by Blümle et al. considered this strategy necessary to obtain a complete search [ 32 ].

Contacting experts to identify journals to handsearch

Two studies contacted experts to develop a list of journals to handsearch [ 30 , 31 ]. Armstrong et al. contacted organisations to develop a list of non-indexed journals to handsearch (in addition to database searching), and Langham et al. used a combination of database searches, contacting organisations and searches of library shelves to identity relevant journals (in addition to database searching) [ 30 , 31 ]. A list of possible journals to handsearch was provided to professional contacts to appraise and identify any missing journals [ 30 ]. Neither study specifically reports the number of journals identified by experts to handsearch, when compared to the number of journals to handsearch identified by database searching, and there is no discussion of the effectiveness of either method in identifying journals to handsearch.

Five studies explored specifically where or which sections of a journal to handsearch [ 25 , 27 , 28 , 31 , 33 ]. A study by Hopewell et al. handsearched full reports, short reports, editorials, correspondence sections, meeting abstracts and supplements [ 27 ]. Hopewell et al. found that, of the 369 reports uniquely identified by handsearching, 92% were abstracts and/or published in the supplement of journals [ 27 ]; two studies reported the greatest value in searching supplement editions of journals [ 28 , 31 ], since these are not routinely indexed in databases [ 28 ]. Armstrong et al. identified three studies (out of 131) through searching supplement editions of journals [ 31 ], and Jadad et al. identified 162 eligible RCTs from a total of 2889 abstracts reported in four journals [ 28 ]; Croft et al. claimed value in searching the correspondence section of journals but they did not record the effect of handsearching this section in terms of identification of studies [ 33 ]; and Adams et al. reported handsearching book reviews and identifying one study [ 25 ].

Table  3 summarises a claimed advantage of handsearching, since the studies demonstrate that handsearching identifies studies missed through database searching. Where the studies reported the reason that the studies were missed by database searching (the advantage of handsearching), these are summarised in Table  3 .

Handsearching results

Table  3 also summarises a claimed disadvantage of handsearching since, even though this method is often defined as a ‘gold standard’, the studies demonstrate that database searching can identify studies missed by handsearching. Where the studies reported the reason that the studies were missed by handsearching (the disadvantage over database searching), these are summarised in Table  3 .

Two studies claimed that the precision of handsearching was low when compared to the precision found in database searching [ 25 , 28 ]. Table  3 records the relative precision between handsearching and MEDLINE searching. Two studies claimed that the time needed to handsearch, and access to resources (including handsearchers), was a disadvantage of handsearching [ 31 , 36 ].

Seven studies reported detail on the time taken to handsearch [ 25 , 28 , 29 , 31 , 33 , 34 , 36 ]. There was no agreement between the studies on how long handsearching takes. The range was between 6 min [ 36 ] and 1 h [ 29 ] per journal handsearched. It is not possible to calculate an average, since not all studies reported their handsearching as time per journal handsearched. One study reported handsearching in ‘two hour bursts’ across 3 months in order to focus concentration, but the detail of how often these ‘bursts’ occurred and the effectiveness relative to ‘non-burst’ handsearching is not reported [ 33 ].

Jadad et al. reported the time taken specifically to handsearch the supplement editions [ 28 ]. Two thousand, eight hundred and eighty-nine abstracts were handsearched in 172 min with an average of 1.1 min per eligible study identified [ 28 ].

The use of volunteers [ 29 , 30 ] or experienced handsearchers [ 27 , 31 ] varied in studies. Due to the varied outcome measures used between the studies, it is not possible to aggregate the effectiveness of experienced handsearchers against volunteers. Moher et al., however, specifically sought to test the effectiveness of volunteers in identifying RCTs, finding that volunteers with minimal training can contribute to handsearching [ 29 ]. Conversely, a study by Langham et al. discussed a possible explanation of their volunteer handsearcher missing studies was a lack of specific knowledge to identify RCTs [ 30 ], which suggests experience or training is necessary. Milne and Thorogood suggested that handsearching may need to be undertaken by more than one person [ 36 ].

Five studies provided data on training given to handsearchers [ 25 , 27 , 29 , 30 , 34 ]. This included specific training on RCTs [ 27 , 29 ], a 2-h training session [ 29 , 34 ] and an information pack including guidelines to handsearching, developed by experienced handsearchers, and a thesaurus of terms to identify RCTs [ 30 ].This data was reported narratively, and supporting information, such as the information pack reported in the study by Langham et al., was not provided in the studies. [ 30 ].

Two studies provided guidance on approaches to handsearching if resources were limited [ 27 , 28 ]. Hopewell et al. claimed that, where resources are limited (and it was accepted that studies would be missed), and the aim of searching is the comprehensive identification of studies reporting RCTs, handsearching is best targeted on journals not indexed in MEDLINE and journals published before 1991 (the year the publication type indexing term for RCTs was introduced into MEDLINE [ 37 ]) [ 27 ]. Jadad et al., in a study focused on identifying RCTs, claimed that a combination of MEDLINE searches with selective handsearching of abstracts of letters may be a good alternative to comprehensive handsearching [ 28 ].

Armstrong et al. claimed that researchers handsearching for non-randomised study designs may need more time to handsearch. No guidance on speculative timing was given [ 31 ].

Moher et al. provided data on costs. Moher et al. recorded costs for photocopying (10–15 Cents Canadian per page) and car parking (10 Dollars Canadian) in their 1995 study assessing the use of volunteers to handsearch [ 29 ].

Searching trial registers

The handbooks focus on the benefit of searching registers [ 10 ], with The Cochrane Handbook providing specific guidance on where to search [ 9 ]. The studies focused on the searching of the registers [ 38 ] and the advantages and disadvantages of doing so [ 39 , 40 ]. Three studies were included [ 38 – 40 ].

It is used for identifying unpublished, recently completed or on-going trials [ 9 , 10 , 39 , 40 ] and keeping a track of any adaptations to trial protocols and reported study outcomes [ 39 , 40 ]. Trials that have been stopped, or were unable to reach optimal recruitment, can also be identified.

The Cochrane Handbook includes a comprehensive list of trial registers to search [ 9 ]. Distinctions are made between national and international trial registers (which hold trials of any population or intervention), subject (i.e. population)-specific registers and pharmaceutical/industry trial registers [ 9 ]. There is a further distinction between on-going, completed trial registers and result registers. Glanville et al. also drew a distinction between trial registers (e.g. ClinicalTrials.gov) and portals to trial registers (e.g. WHO) [ 38 ].

Glanville et al. explored the need to search trial registers as a complementary search method to comprehensive searches of bibliographic databases [ 38 ]. Glanville et al. reported that, in both ClinicalTrials.gov and WHO International Clinical Trials Registry Platform (ICTRP), their ‘highly sensitive single concept search’ of the basic interface offered the greatest reliability in identifying known records. The methods of searching are explored in greater detail in this study [ 38 ].

Two studies claimed that searching trial registers will identify unique studies or study data [ 39 , 40 ]. Van est. et al. reported that, in four out of 80 Cochrane reviews included in their study, primary studies were identified and included from a prospective search of a trial register search [ 39 ]. Jones et al. reported that, of 29 studies to record registry search results in their study, 15 found at least one relevant study through searching a register [ 40 ].

Two studies claimed that searching of trial registers facilitates checking of a priori outcome measures against reported final outcome measures [ 39 , 40 ]. Jones et al. suggested that the comparison of registered trials (and trial data) against published trials (and data) will aid the understanding of any potential bias in the trials [ 40 ].

Jones et al. noted that an advantage of trial registers is that they often include contact details for trial investigators, thereby facilitating author contact [ 40 ].

Two studies concluded that trial registers must be searched in combination with other bibliographic resources [ 38 , 39 ]. Glanville et al. concluded that trial registers lag behind major bibliographic databases in terms of their search interfaces [ 38 ].

None were reported.

Web searching

The handbooks report limited guidance for web searching. The CRD Handbook suggests that web searching may be a useful means of identifying grey literature [ 10 ], and The Campbell Handbook provides some guidance on how to undertake web searches, including a list of grey literature websites [ 11 ]. The studies explored the role of web searching in systematic reviews. Five studies were included [ 41 – 45 ].

What is it used for

It is used for identifying published or unpublished studies not indexed or included in bibliographic databases, or studies missed by database (or other) search methods, identifying and retrieving grey literature and identifying study protocols and on-going studies [ 10 , 11 , 42 , 45 ].

The CRD Handbook makes a separation between a search of the internet through a ‘search engine’ and searches of specific and relevant websites [ 10 ]. It considers the latter to be more practical than a general search of the World Wide Web in systematic reviews [ 10 ].

The Campbell Handbook provides guidance on searching using a search engine [ 11 ], and Eysenbach et al. reported the results of a pilot study to assess the search features of 11 search engines for use in searching for systematic reviews [ 45 ]. 1 The Campbell Handbook suggests that, when using search engines, researchers should use the advanced search function. In some cases, this allows searchers to use Boolean logic and employ strategies to limit searches, such as precise phrases like “control group” [ 11 ].

Godin et al. reported the development and use of a web-searching protocol to identify grey literature as part of study identification in a systematic review [ 43 ]. Godin et al. broke their web searching into three parts: first, searches using Google for documents published on the internet; secondly, searches using custom Google search engines; and thirdly, browsing targeted websites of relevant organisations and agencies [ 43 ].

Two studies identified studies uniquely by web searching [ 43 , 45 ]. Eysenbach et al. identified 14 unpublished, on-going or recently finished trials, and at least nine were considered relevant for four systematic reviews [ 45 ]. Godin et al. identified 302 potentially relevant reports of which 15 were included in their systematic review [ 43 ].

Three studies commented on the types of study or study data identified [ 42 , 43 , 45 ]. Eysenbach et al. claimed that internet searches may identify ‘hints’ to on-going or recently completed studies via grey literature [ 45 ]; Godin et al. uniquely identified report literature [ 43 ]; and Stansfield et al. suggested that web searching may identify studies not identified from ‘traditional’ database searches [ 42 ].

Five studies discussed the disadvantages of web searching [ 41 – 45 ]. The studies drew illustrative comparisons between database searching and web searching in order to highlight the disadvantages of web searching:

Three studies commented on searching using a web search engine: Eysenbach et al. reported that current search engines are limited by functionality and that they cover only a fraction of the visible web [ 45 ]; Mahood et al. claimed that their chosen search engines could not accommodate either full or modified search strategies, nor did they support controlled indexing [ 44 ]; and Godin et al. claimed that, in contrast to systematic searches of bibliographic databases, where one search strategy combining all search terms would be used, Google searches may require several search enquiries containing multiple combinations of search terms [ 43 ].

Three studies commented on the number of studies returned through web searching [ 43 – 45 ]. Godin et al. claimed that searching Google can be overwhelming due to the amount of information and lack of consistent organisation of websites [ 43 ]; Mahood et al. had to limit their web searches to title only in order to control search returns [ 44 ], and Eysenbach et al. recorded recall of between 0 and 43.6%, finding references to published studies and precision for hints to published or unpublished studies ranged between 0 and 20.2% [ 45 ].

Three studies commented on the search returns [ 42 , 43 , 45 ]. Eysenbach et al. and Stansfield et al. commented on the lack of abstracts when web searching, which impacts on the precision of web searching and volume of studies identified [ 42 , 45 ], and Godin et al. claimed that it was impossible to screen all results from a Google search, so researchers were reliant on page ranking [ 43 ].

Three studies claimed potential issues with the reliability of items identified through web searching [ 41 , 43 , 45 ]. Godin et al. discussed the possibility of bias created in web searching, where search results are presented depending on geographic location or previous search history [ 43 ]; Briscoe reported that algorithms used by search engines change over time and according to the user, which will influence the identification of studies and impact the transparency and replicability of search reporting [ 41 ]; and Eysenbach et al. reported identifying a study published on-line that differed in reporting to the copy published in the peer-reviewed journal, where adverse event data was omitted in the on-line version [ 45 ].

Stansfield et al. claimed that the lack of functionality to export search results presented a challenge to web searchers [ 42 ]. Three studies claimed that web searching presented difficulties in transparent search reporting [ 41 , 43 , 44 ].

Two studies discussed time taken to web-search [ 43 , 45 ]. Eysenbach et al. reported searching 429 returned search result pages in 21 h [ 45 ], and Godin et al. reports custom Google searching taking 7.9 h and targeted web searches taking 9–11 h, both timings being specific to the case studies in question [ 43 ].

Stansfield et al. discussed planning when to undertake web searching [ 42 ]. Stansfield et al. linked planning a web search to the time-frame and resources available in order to inform where to search [ 42 ].

Mahood et al. claimed that large yields of studies can be difficult and time consuming to explore, sort, manage and process for inclusion [ 44 ]. Mahood et al. initially had to limit their web searching to title only (as a method to control volume) before eventually rejecting their web searching due to concerns about reproducibility and ability to manage search returns [ 44 ].

No studies reported any data relating to the costs involved in web searching.

The discussion will focus on two elements inherent in the research question of this study: how does current supplementary search practice compare with recommended best practice and what are the implications of the evidence for searching using these supplementary methods.

The advent of e-mail (and more specifically the standardised reporting of e-mail addresses for corresponding study authors) would appear to have improved the efficiency of contacting study authors [ 10 , 11 ], although it is possible that it has not altered the effectiveness [ 46 ]. Identifying additional studies or data (the effectiveness) is conditional upon a reply, whatever the method of contact. The guidance of the handbooks, to consider how best to set out requests for studies or study data, is well made but seldom explored in the studies themselves. Whilst making contact is important, which the studies evaluate exploring techniques to improve the rate of reply would be a valuable contribution to improve the efficiency and effectiveness of identifying studies or study data through author contact.

When to contact study authors is worthy of consideration, since the studies included in this review reported a delay between asking for studies or study data and a response. Sufficient time should be allowed between identifying the need for author contact, making contact, a response being provided and the study or data being integrated into the review (with all the methodological implications considered). A recognition for the need of this method, combined with the realisation that this method takes time to yield results, is important. It is perhaps for this reason that, whilst contacting authors is common in systematic reviews, it is not a method of study identification that is undertaken as a matter of course [ 47 ].

The concept of contacting authors could also be understood more broadly than simply contacting with a view to requesting known studies or data. Whilst in contact with authors, requests for unpublished, linked or forthcoming studies are not unreasonable requests, and authors can assist with the interpretation of specific elements of studies or topics, in order to aid the process of critical appraisal. Furthermore, Ogilvie et al. found the value in contacting experts was the link to better reports of studies already identified [ 48 ]. This highlights the potential flexibility of the search method: it is not only the chance to identify known studies or study data but also it offers the opportunity to speak with experts.

The advantages and disadvantages (and resource requirements) were most clearly stated for this supplementary search method. The handbooks, and some studies, suggested and found advantages and disadvantages in the methods and tools.

The Cochrane Handbook suggested that there is little evidence to support the methodology of citation searching, since the citation of studies ‘is far from objective’ [ 49 ]. The studies included in this review suggested that the reasons for ‘non-citation’ are unclear and could range from selective citation (i.e. selective reporting) to pragmatic reasons, such as a review of trials being cited instead of each individual trial reviewed [ 21 ]. Furthermore, a high number of citations for a study should not necessarily be confused as an indicator of study quality [ 50 , 51 ] or a complete citation network. Non-citation of studies, or ‘linking lag’ [ 1 ], forces a break in citational networks [ 1 , 2 , 21 ], meaning it becomes unclear when (or if all) studies have cited a primary study [ 20 ]. There is presently no method to assess the completeness of citational networks and no certainty as to the comprehension of any citation chasing.

There is little common agreement between the studies as to which tool (or combination of tools) is superior in citation chasing, since the relative merits of each resource depend greatly upon the topic of review, the data range of the resource and the currency of the results (c.f. [ 1 , 23 , 24 , 52 – 54 ]). A study that evaluated the tools (Web of Science, SCOPUS and Google Scholar), how the tools are best searched, how the platform hosts select data for inclusion and the advantages and disadvantages of use would make clearer statements on when (or if) to use which tools.

There are, undoubtedly, advantages to citation searching. The citational link is neutral, in the sense that it only links the studies but it does not explain the nature of the link. This is important, since a citation search will identify any study linked to the primary study, including erratum studies and studies that dispute or disagree with the primary study, and it should also link different publication types, such as editorial content, reviews or grey literature. This could not only aid interpretation of studies but also it could help researchers explore the idea of study impact. Furthermore, as reported in the ‘ Results ’ section, a citation search links by citation and it is not beholden to the use of ‘the correct’ search terms or database indexing. It may, therefore, as Papaioannou et al. reported, facilitate serendipitous study identification [ 3 ], suggesting that citation chasing is valuable in scoping review topics, to aid development of searches, and review searches, in order to ensure all studies have been identified.

The nature of bi-directional citation chasing suggests that, given the relative specificity, this method could possibly be used to efficiently update systematic reviews using known includes as the citations to chase [ 20 ]. Researchers have had positive, although incomplete, success trialling this method, and studies suggest that citation chasing alone is not a substitute for standard update searches [ 55 , 56 ].

The evidence on handsearching can be summarised as (1) selecting where to handsearch, (2) what to handsearch and (3) who does the handsearching. In relation to 1, the handbooks advocate selecting journals to handsearch on the basis of the number of relevant studies included from journals identified in database searching. This approach means handsearching is a supplementary method to database searching, since to undertake handsearching—following this method—database searches define the list of journals to handsearch.

Studies included in this review provided empirical evidence that handsearching journals identified by database searching was effective in identifying studies missed by poor indexing, lack of study design or omission of key search terms, or where sections of journals are not indexed on databases. In this way, this approach to selecting journals to handsearch could be categorised as a ‘safety net search’, since it aims to identify studies missed by deficiencies in literature searching and database indexing. This approach to selecting journals to handsearch, even though it is effective, could be argued to be a duplication of effort, since the journals being handsearched have already been ‘searched’ through the bibliographic databases. This is likely why the studies recorded low precision (compared to database searches) and why handsearching takes longer [ 28 ].

The Cochrane Handbook and three studies suggested alternative ways to identify journals to handsearch: namely, selecting journals not indexed on MEDLINE or EMBASE [ 9 , 32 ]—a suggestion that is easily changed to read ‘primary databases’ relevant to the field of study (i.e. ERIC for reviews of educational topics)—and contacting experts, contacting organisations and searches of library shelves [ 30 , 31 ]. Neither the study by Armstrong et al. nor the study by Langham et al. listed the journals identified by method of identification, so it is not clear if there were differences between the list of journals provided by experts when compared to those provided by databases [ 30 , 31 ]. This review did not identify any studies that compared the use of databases to identify journals to handsearch as against these alternative methods but such a study may be of value if efficiencies could be found in practice.

It may be that, in reviews in which a comprehensive identification of studies is required, identifying journals to handsearch should be done both by using databases and contacting experts or organisations. The former being to cover any deficiencies in the database searching and the latter to capture any unique journals or conferences known to experts but not indexed in databases.

Selecting what to handsearch and who should handsearch was another notable difference between the handbooks and studies. The studies included in this review identified studies uniquely from handsearching various sections of journals (from abstracts through to book reviews), and the studies used volunteers, provided training to handsearchers, and used experienced handsearchers to handsearch, with varying degrees of success and failure since handsearching relates to effectively identifying studies when compared to database searching. The Cochrane Collaboration arguably has one of the longest track-records of handsearching projects (c/f [ 37 ]), and it is their recommendation that handsearching is the page-by-page examination of the entire contents of a journal [ 9 , 10 ] by a well-trained handsearcher [ 9 ]. Handsearching is commonly referred to and used as a ‘gold standard’ comparator to establish effectiveness of other search methods. Given that every study included in this review uniquely identified studies by handsearching but also missed studies by handsearching too, a reminder of what constitutes handsearching is likely warranted.

Trial registers

The handbooks provide guidance on where to search and the studies focused on the effectiveness of study identification in selected registers and/or the practicalities of searching registers. In this way, the studies advance the guidance of the handbooks, since they provide empirically derived case-studies of searching the registers. The implications for searching, however, are clear: searching trial registers should still be undertaken in combination with bibliographic database searching [ 38 , 57 ]. Even despite the aims of the International Committee of Medical Journal Editors [ 58 ], comprehensive and prospective registration of trials—and keeping the trial data up to date—is still not common place. It is unclear what pressure (if any) is put upon trial managers who do not prospectively register their trials and, in fact, if there is any active penalty if trial managers do not do so. Until this issue is resolved, the comprehension of registers will remain uncertain and a combination of bibliographic database searching (to identify published trials) and searches of trial registers (to identify recruiting, on-going or completed trials) is required.

The advantages of searching trial registers are worthy of discussion. Registered trials include an e-mail address for trial managers, which can facilitate author contact, and the studies concluded that more consistent searching of trial registers may improve identification of publication and outcome reporting bias [ 40 , 59 ]. If trial managers were using the portals correctly, it would also be a practical method of reporting results and sharing study data, perhaps akin to a ‘project website’, as recommend in the Cochrane Handbook [ 9 ]. The variability of the search interfaces is notably a disadvantage and something upon which could be improved. Glanville et al. observed that the search interfaces lag behind major bibliographic databases [ 38 ]. If the registers themselves are hard to search (and in some cases impossible to export data from), they are less likely to be searched. Trial managers and information specialists/researchers could usefully work together with the registers to develop the interfaces in order to meet the needs of all who use them. The use of trial registers may be broader than only researchers [ 60 ].

In their 2001 study, Eysenbach et al. stated that the role of the internet for identifying studies for systematic reviews is less clear when compared to other methods of study identification [ 45 ]. The handbooks do not update this view, and very few studies were identified in this review which improve upon Eysenbach et al.’s claim. The studies have attempted to take on Eysenbach et al.’s suggestion that a systematic investigation to evaluate the usefulness of the internet for locating evidence is needed. Mahood et al., however, had to abandon their attempts to web-search [ 44 ], but Godin et al. took this work a little further in their case study with reference to identifying grey literature [ 43 ].

The comparative lack of guidance in the handbooks could stem either from a lack of certain knowledge of how to web-search or perhaps a lack of certainty of how to do this systematically, such that web searching could be replicable, and therefore, be included as a method to identify studies without introducing bias. Researchers are exploring the idea of how far web searching can meet the need to be replicable and transparent but still functional [ 41 ]. Further guidance is undoubtedly needed on this supplementary search method.

Limitations

The date range and age of the handbooks and studies included in this review could be considered a limitation of this study.

Comparative and non-comparative case studies form the evidence base for this study. The studies included in this review have been taken at face-value, and no formal quality appraisal has been undertaken since no suitable tool exists. Furthermore, supplementary search methods are typically evaluated in the context of effectiveness, which is potentially a limited test of the contribution they may offer in the process of study identification. Different thresholds of effectiveness and efficiency may apply in the use of supplementary search methods in systematic reviews of qualitative studies when compared to reviews of RCTs, for example.

The studies themselves do not necessarily correlate to the concepts of claimed advantages and disadvantages. In most cases, proposed advantages and disadvantages have not been tested in practice.

Whilst we have aimed to comprehensively identify and review studies for inclusion, the use of supplementary search methods is a broad field of study and it is possible that some completed studies may have been inadvertently missed or overlooked. It is possible that standard systematic review techniques, such as double-screening, would have minimised this risk, but we are confident that, whilst a more systematic approach may have improved the rigour of the study, it is unlikely to alter the conclusions below.

Current supplementary search practice aligns methodologically with recommended best practice. The search methods as recommended in the handbooks are perceptibly the same methods as used in the studies identified in this review. The difference between the handbooks and the studies is of purpose: the studies sought to test the search methods or tools used to undertake the search methods.

The causal inference between methods (as presented in the handbooks) and results (as found in the studies) could be usefully tested to develop our understanding of these supplementary search methods. Further research is needed to better understand these search methods. Specifically, consistency in measuring outcomes, so the results can be generalised and trends identified, which would provide a link not only to better effectiveness data but also to efficiency data, offers researchers a better understanding of the value of using these search methods, or not.

All of the studies discussed in this review claimed to identify additional includable material for their reviews using supplementary search methods that would have been missed using database searches alone. Few of the studies, however, reported the resources required to identify these unique studies. Further, none of the studies used a common framework or provided information that allows a common metric to be calculated. It is not, therefore, possible to compare the resources required to identify any extra study with each search method. This, alongside the use of comparative and non-comparative case studies as the primary study design to test effectiveness, limits our ability to generalise the results of the studies and so reliably interpret the broader efficiency of these search methods. Researchers could usefully consider reporting the amount of time taken to undertake each search method in their search reporting [ 28 , 61 ].

Value versus impact?

Identifying unique studies is commonly interpreted as adding value to the review and the process of searching in and of itself. Only three studies sought to extend this, appraising either the quality of the studies identified or the contribution of the studies to the synthesis as a way of considering the value of the additional studies [ 3 , 16 , 45 ]. In reviews of effectiveness, where all studies should be identified so as to generate a reliable estimate of effect, study value might be a moot point but, in resource-limited situations, or for reviews where a comprehensive identification of studies is less important, study value is an important metric in understanding the contribution of supplementary search methods and the extent to which researchers invest time in undertaking them.

Time + value

Comparing the time taken to search, with a summary estimate of the contribution or value of the studies identified uniquely, against the total number of studies identified, could alter how researchers value supplementary searches. It would permit some basic form of retrospective cost-effectiveness analysis, which would ultimately move literature searching beyond simply claiming that more studies were identified to explaining what studies were identified, at what cost and to what value.

Acknowledgements

CC is grateful to Danica Cooper for her proof-reading and comments. CC is grateful for feedback from Jo Varley-Campbell and the EST and IS team at Exeter Medical School: Jo Thompson-Coon, Rebecca Abbott, Rebecca Whear, Morwenna Rogers, Alison Bethel, Simon Briscoe and Sophie Robinson. CC is grateful to Juan Talens-Bou and Jenny Lowe for their assistance in full-text retrieval.

CC is grateful to Chris Hyde for his help in stimulating the development of this study and his on-going guidance.

This work was funded as part of a PenTAG NIHR Health Technology Assessment Grant.

Availability of data and materials

Abbreviations, authors’ contributions.

CC conceived, designed and undertook the study as a part of his PhD. AB, NB and RG provided comments on, and discussed, the study in draft as part of CC’s PhD supervision. All authors have approved this manuscript prior to submission.

Authors’ information

CC is a p/t PhD student exploring the use of tailored literature searches in complex systematic reviews. This publication forms a part of his PhD thesis.

Ethics approval and consent to participate

Consent for publication, competing interests.

AB and RG are associate editors of systematic reviews.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

1 Eysenbach et al. recommend Alta Vista but this search engine no longer exists.

Contributor Information

Chris Cooper, Email: [email protected] .

Andrew Booth, Email: [email protected] .

Nicky Britten, Email: [email protected] .

Ruth Garside, Email: [email protected] .

The Truth About WhatsApp Business: Advantages and Disadvantages

• by Bima Sandria
• Published on May 16, 2024
• Updated on May 16, 2024

You’re probably familiar with using WhatsApp Business for communicating with customers. We believe WhatsApp is an extremely user-friendly messaging application for people of all backgrounds. After all, WhatsApp can be used on all types of operating systems currently in use. However, have you thoroughly understood the advantages and disadvantages of WhatsApp Business?

Understanding these aspects is crucial for maximizing its potential. Especially where customer communication plays a pivotal role in advancing your business. Are you aware of the strengths and weaknesses of WhatsApp Business? If not, let’s dissect them one by one.

Introducing WhatsApp Business

WhatsApp Business is a WhatsApp application that can be utilized by small to medium-sized businesses to fulfill their communication needs, promotional activities, and maintaining relationships with their customers. As time progresses, WhatsApp Business has also been adopted by professionals as a primary channel for communicating with their clients.

Equipped with features tailored for businesses, this platform becomes an ideal solution for enhancing customer communication on a messaging platform that has become a part of billions of people’s daily lives. However, similar to other applications, WhatsApp Business has its own strengths and weaknesses. In this article, we will together dissect the advantages and disadvantages of WhatsApp Business.

READ MORE: How to Offer Products to Customers: Becoming More Targeted

Advantages of WhatsApp Business

After learning about the drawbacks of WhatsApp Business, it’s important not to immediately dismiss it. WhatsApp Business also boasts a range of advantages beneficial for your business. Here are some of its strengths.

1. Ensured Message Security

WhatsApp employs end-to-end encryption, ensuring that messages can only be read by the sender and the receiver. This reduces the risk of important conversations being leaked or fraudulent activities. Thus, sending crucial messages, such as transaction receipts, can be done with greater confidence.

2. Automated Messaging Features

You can send automated welcome messages or greetings to customers when they initiate a conversation. These welcome messages are automatically sent when customers send their first message, creating a friendly impression.

3. Multi-Device Usage

WhatsApp Business allows usage on two devices simultaneously. By switching your WhatsApp Business mode to beta mode within the settings, you can achieve this. This enables message synchronization across both devices, enhancing accessibility and convenience.

4. Catalog Feature

For businesses in sectors like fashion, automotive, or those requiring an online catalog, this feature is immensely helpful. You no longer need to repeatedly send product photos to customers. Instead, you can simply prompt them to view your WhatsApp Business profile for product images.

It’s essential to understand that the drawbacks of WhatsApp Business can be minimized, while its advantages can be further enhanced. You can consider transitioning to WhatsApp Business API , especially if you’re operating on a larger scale where customer service is a current concern.

Disadvantages of WhatsApp Business

WhatsApp stands as the most widely used messaging application or social media platform with the highest number of active users . In Indonesia alone, as of January 2024, 90% of the population aged between 16 and 60 actively uses WhatsApp. With this data, it can be said that WhatsApp is the primary communication application for the target market of your products. Therefore, it’s crucial for you to truly understand WhatsApp Business, especially its drawbacks, outlined below:

1. Limited Message Sending Quantity

You can only send 5000 messages per month. This amount is clearly insufficient to accommodate your messaging needs, especially if your business operates on a large scale, such as online healthcare consultation services or tutoring. You need to verify your phone number to obtain a higher message sending quota.

2. Inability to Integrate with Other Applications

Without an API, WhatsApp Business cannot be integrated with other applications. Consequently, you won’t be able to monitor messages from multiple applications on one device. It requires multiple people to monitor messages coming from various social media platforms. This makes the work less effective and less efficient.

3. Affected by Office Hours

You won’t be able to respond to customer messages quickly on holidays. WhatsApp Business only allows sending one automated message. Subsequent messages will be responded to by customer service agents. However, customer inquiries can arise at any time, which ultimately disadvantages consumers.

4. Lack of WhatsApp Green Checkmark

WhatsApp Business accounts that are not verified do not receive the automatic green checkmark like WhatsApp Business API accounts. The absence of this verification symbol can affect the credibility perception in the eyes of some customers, especially during the current prevalence of WhatsApp scams, making customers more cautious about receiving messages from unfamiliar numbers.

5. Limited WhatsApp Catalog Feature

One of WhatsApp Business’s shortcomings is its limited catalog feature. WhatsApp Business can only display a few products in the catalog. If customers want to view other product catalogs, they have to open other applications such as the official website and e-commerce accounts. Consequently, there’s a high bounce rate and a decreased likelihood of converting sales.

6. Limited Agent Numbers

WhatsApp Business can only be used on two devices, namely smartphones and computers or laptops. This means that one number can only be handled by two agents at a time. If you want to increase the number of agents, you’ll need to create a new WhatsApp Business account.

READ MORE: How to Secure Your WhatsApp Business Account: 7 Vital Steps!

7. Limited Automation Features

Although it provides features for automated messages and quick responses, the automation function on WhatsApp Business is still basic. Compared to WhatsApp Business API, its automation capabilities are lower, especially if you desire chatbots or more complex message logic.

Transitioning to WhatsApp Business API

Through the discussion of the advantages and disadvantages of WhatsApp Business earlier, hopefully, you’ve reached the conclusion you need. Upon closer inspection, WhatsApp Business indeed has many shortcomings. If you feel that you’re not sufficiently aided by WhatsApp Business features, you can start transitioning to WhatsApp Business API.

WhatsApp Business API is generally an enhancement of the WhatsApp Business you’re already using. By utilizing WhatsApp Business API, you can reach more customers. Here are the benefits you’ll receive when using WhatsApp Business API:

1. Integration with Multiple Applications

API enables integration with various applications, not limited to social media for monitoring incoming messages, but also integration with CRM applications. This allows you to monitor communication movements on your sales lead prospects.

2. No Message Pile-up

WhatsApp Business API provides an Omnichannel dashboard useful for distributing messages ideally. Incoming messages will be automatically distributed to customer service agents promptly. This ensures that customer messages receive responses promptly and assists in responding to incoming sales leads quickly.

3. Unlimited Message Sending

WhatsApp Business API allows you to send messages without limits. When using WhatsApp Business API, your account is automatically verified and gains access. Additionally, you can create various customer message templates, making them more personalized. Message sending can also be automated and scheduled.

4. 24/7 Response

Through WhatsApp Business API, you can activate the auto-reply feature for customer messages. You can set up responses to commonly asked questions, referring to your product FAQ. This ensures your presence for customers without being affected by office hours.

5. WhatsApp Green Tick

The green tick WhatsApp or verification will be provided after you verify your WhatsApp Business API. You can start the verification process through Facebook Business Manager by entering necessary data. Required data includes official email address, address, and your company’s official website. Afterward, you just need to wait for confirmation from Facebook.

WhatsApp Business API Service by Qiscus

It’s time for you to consider switching to WhatsApp Business API service. With all the conveniences it offers, your customer service will run optimally. To enjoy WhatsApp Business API service, you can partner with Qiscus as the official WhatsApp Business API partner in Indonesia.

Qiscus is ready to provide consultation regarding your needs for implementing WhatsApp Business API. In addition to consultation services, Qiscus also offers after-sales services to address all your questions and concerns after the implementation is ready. Don’t hesitate, contact Qiscus here.

• WhatsApp Business
• WhatsApp Business API

Bima Sandria

You may also like.

Providing the Best Customer Service & Support via WhatsApp Business API

• Published on March 22, 2022

5 Reasons Companies Choose to Have Their Own Messaging App

• Published on July 12, 2019
• Updated on March 26, 2020

How a Chat App Revolutionized the Banking Industry

• Published on May 16, 2019

AI and Chat Trends in 2018

• by [email protected]
• Published on January 10, 2018
• Updated on March 27, 2020

Top 5 Insights From Us! Special Gift for You

• Published on January 31, 2018

Chatbot Failed to Fulfill Customer’s Request: How Do We Respond?

• Published on March 17, 2018

Not all data are created equal; some are structured, but most of them are unstructured. Structured and unstructured data are sourced, collected and scaled in different ways and each one resides in a different type of database.

In this article, we will take a deep dive into both types so that you can get the most out of your data.

Structured data—typically categorized as quantitative data—is highly organized and easily decipherable by  machine learning algorithms .  Developed by IBM® in 1974 , structured query language (SQL) is the programming language used to manage structured data. By using a  relational (SQL) database , business users can quickly input, search and manipulate structured data.

Examples of structured data include dates, names, addresses, credit card numbers, among others. Their benefits are tied to ease of use and access, while liabilities revolve around data inflexibility:

• Easily used by machine learning (ML) algorithms:  The specific and organized architecture of structured data eases the manipulation and querying of ML data.
• Easily used by business users:  Structured data do not require an in-depth understanding of different types of data and how they function. With a basic understanding of the topic relative to the data, users can easily access and interpret the data.
• Accessible by more tools:  Since structured data predates unstructured data, there are more tools available for using and analyzing structured data.
• Limited usage:  Data with a predefined structure can only be used for its intended purpose, which limits its flexibility and usability.
• Limited storage options:  Structured data are usually stored in data storage systems with rigid schemas (for example, “ data warehouses ”). Therefore, changes in data requirements necessitate an update of all structured data, which leads to a massive expenditure of time and resources.
• OLAP :  Performs high-speed, multidimensional data analysis from unified, centralized data stores.
• SQLite : (link resides outside ibm.com)  Implements a self-contained,  serverless , zero-configuration, transactional relational database engine.
• MySQL :  Embeds data into mass-deployed software, particularly mission-critical, heavy-load production system.
• PostgreSQL :  Supports SQL and JSON querying as well as high-tier programming languages (C/C+, Java,  Python , among others.).
• Customer relationship management (CRM):  CRM software runs structured data through analytical tools to create datasets that reveal customer behavior patterns and trends.
• Online booking:  Hotel and ticket reservation data (for example, dates, prices, destinations, among others.) fits the “rows and columns” format indicative of the pre-defined data model.
• Accounting:  Accounting firms or departments use structured data to process and record financial transactions.

Unstructured data, typically categorized as qualitative data, cannot be processed and analyzed through conventional data tools and methods. Since unstructured data does not have a predefined data model, it is best managed in  non-relational (NoSQL) databases . Another way to manage unstructured data is to use  data lakes  to preserve it in raw form.

The importance of unstructured data is rapidly increasing.  Recent projections  (link resides outside ibm.com) indicate that unstructured data is over 80% of all enterprise data, while 95% of businesses prioritize unstructured data management.

Examples of unstructured data include text, mobile activity, social media posts, Internet of Things (IoT) sensor data, among others. Their benefits involve advantages in format, speed and storage, while liabilities revolve around expertise and available resources:

• Native format:  Unstructured data, stored in its native format, remains undefined until needed. Its adaptability increases file formats in the database, which widens the data pool and enables data scientists to prepare and analyze only the data they need.
• Fast accumulation rates:  Since there is no need to predefine the data, it can be collected quickly and easily.
• Data lake storage:  Allows for massive storage and pay-as-you-use pricing, which cuts costs and eases scalability.
• Requires expertise:  Due to its undefined or non-formatted nature, data science expertise is required to prepare and analyze unstructured data. This is beneficial to data analysts but alienates unspecialized business users who might not fully understand specialized data topics or how to utilize their data.
• Specialized tools:  Specialized tools are required to manipulate unstructured data, which limits product choices for data managers.
• MongoDB :  Uses flexible documents to process data for cross-platform applications and services.
• DynamoDB :  (link resides outside ibm.com) Delivers single-digit millisecond performance at any scale through built-in security, in-memory caching and backup and restore.
• Hadoop :  Provides distributed processing of large data sets using simple programming models and no formatting requirements.
• Azure :  Enables agile cloud computing for creating and managing apps through Microsoft’s data centers.
• Data mining :  Enables businesses to use unstructured data to identify consumer behavior, product sentiment and purchasing patterns to better accommodate their customer base.
• Predictive data analytics :  Alert businesses of important activity ahead of time so they can properly plan and accordingly adjust to significant market shifts.
• Chatbots :  Perform text analysis to route customer questions to the appropriate answer sources.

While structured (quantitative) data gives a “birds-eye view” of customers, unstructured (qualitative) data provides a deeper understanding of customer behavior and intent. Let’s explore some of the key areas of difference and their implications:

• Sources:  Structured data is sourced from GPS sensors, online forms, network logs, web server logs,  OLTP systems , among others; whereas unstructured data sources include email messages, word-processing documents, PDF files, and others.
• Forms:  Structured data consists of numbers and values, whereas unstructured data consists of sensors, text files, audio and video files, among others.
• Models:  Structured data has a predefined data model and is formatted to a set data structure before being placed in data storage (for example, schema-on-write), whereas unstructured data is stored in its native format and not processed until it is used (for example, schema-on-read).
• Storage:  Structured data is stored in tabular formats (for example, excel sheets or SQL databases) that require less storage space. It can be stored in data warehouses, which makes it highly scalable. Unstructured data, on the other hand, is stored as media files or NoSQL databases, which require more space. It can be stored in data lakes, which makes it difficult to scale.
• Uses:  Structured data is used in machine learning (ML) and drives its algorithms, whereas unstructured data is used in  natural language processing  (NLP) and text mining.

Semi-structured data (for example, JSON, CSV, XML) is the “bridge” between structured and unstructured data. It does not have a predefined data model and is more complex than structured data, yet easier to store than unstructured data.

Semi-structured data uses “metadata” (for example, tags and semantic markers) to identify specific data characteristics and scale data into records and preset fields. Metadata ultimately enables semi-structured data to be better cataloged, searched and analyzed than unstructured data.

• Example of metadata usage:  An online article displays a headline, a snippet, a featured image, image alt-text, slug, among others, which helps differentiate one piece of web content from similar pieces.
• Example of semi-structured data vs. structured data:  A tab-delimited file containing customer data versus a database containing CRM tables.
• Example of semi-structured data vs. unstructured data:  A tab-delimited file versus a list of comments from a customer’s Instagram.

Recent developments in  artificial intelligence  (AI) and machine learning (ML) are driving the future wave of data, which is enhancing business intelligence and advancing industrial innovation. In particular, the data formats and models that are covered in this article are helping business users to do the following:

• Analyze digital communications for compliance:  Pattern recognition and email threading analysis software that can search email and chat data for potential noncompliance.
• Track high-volume customer conversations in social media:  Text analytics and sentiment analysis that enables monitoring of marketing campaign results and identifying online threats.
• Gain new marketing intelligence:  ML analytics tools that can quickly cover massive amounts of data to help businesses analyze customer behavior.

Furthermore, smart and efficient usage of data formats and models can help you with the following:

• Understand customer needs at a deeper level to better serve them
• Create more focused and targeted marketing campaigns
• Track current metrics and create new ones
• Create better product opportunities and offerings
• Reduce operational costs

Whether you are a seasoned data expert or a novice business owner, being able to handle all forms of data is conducive to your success. By using structured, semi-structured and unstructured data options, you can perform optimal data management that will ultimately benefit your mission.

Get the latest tech insights and expert thought leadership in your inbox.

To better understand data storage options for whatever kind of data best serves you, check out IBM Cloud Databases

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.