Literature review and systematic literature review
Usually we differentiate between a literature review conducted to get an overview of a field of research (also called a scoping review) and the systematic literature review that is performed as a methodology. The latter is more extensive and follows more stringent rules, however the basic principles of the systematic literature review can also be applied to a less formalised literature review. Both of these research types stem from medical research, and you will be able to find many guides within these fields. The guide on this page is not to be used as an exhaustive guide, but mainly as an introdcution to the topic.
A systematic literature review aims at answering a clearly formulated research question and identifies, selects and critically appraises research. The systematic review should follow a clearly defined protocol or plan where the criteria is clearly stated before the review is conducted. It is a comprehensive, transparent search conducted over multiple databases and search systems. It needs to be replicable and reproducable by other researchers. It involves planning a well thought out search strategy which has a specific focus or answers a defined question. The review identifies the type of information searched, critiqued and reported within known timeframes. The search terms, search strategies (including database names, platforms, dates of search) and limits all need to be included in the review.
The steps you should follow are roughly:
- Your rationale and objectives, including a breakdown of the relevant framework you are using.
- Your eligibility criteria (inclusions and exclusions). “One of the features that distinguishes a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria)” Cochrane Handbook, section 3.1 If using PICO these should match those elements and might include demography, types of studies or geography.
- Where you will search (databases and grey literature: SCOPUS; Google, OneSearch, WoS) and your search strategy
- How will you screen your records? What will extraction, management and analysis of the data look like?
- Decide if you will publish your protocol, this is sometimes recommended practice in order to avoid duplications.
- Develop a search strategy
- When doing a full review: consider text mining and search filters
- Where will you search?
- Document your searches and the search strategy you are using
- Continusly review and update your search as needed
In order to count as fully systematic you must ensure your searches are comprehensive and unbiased, meaning you will need to search as many systems as possible. Consider searching both electronic and print material. You must also ensure transparency in your search, documenting the strategy in such a way that it can be reproduced.
After having collected as many texts as possible you will need to go through these in order to eliminate irrelvant material. This can of course be done manually, but it. is recommended you use software. The software will make the process both easier and quicker.
- EndNote: usually. used as a citation managemen software it can also be utilized to screen agains criteria. See this article for guidance.
- Rayyan - free software for screening and literature reviews.
- Harzing's Publish or Perish: free software mainly intended to be used to gather citation data, can be helpful in eliminating duplicates.
Screening is usally done in two satges. The first consists of title and abstract screening where obviously irrelvant texts are eliminated. After this a full text screen is conducted, so all texts at this stage must be available in full. Do these articles meet your criteria? The reason for exluding articles should be documented at the full text screening. It is also recommended that screening is conducted by more than one reviewer in orde to ward against bias.
At this stage you will need to determine if the articles and texts chosen follow scientific principles. You need to examine the research in each text to judge its trustworthiness, value and relevance.
Has the research been conducted in a way that minimises bias? (Is it trustworthy?) If so, what does the study show? (What is its value?) What do the results mean for a particular context in which a decision is being made? (Is it relevant to your research question?)
For a systematic review each study will have to be analysed using a specific set of tools. Freely available critical appraisal tools include:
- Assessment of Multiple Systematic Reviews (AMSTAR) checklist
- Centre for Evidence-based Veterinary Medicine, University of Nottingham. Appraisal Tools
- COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN)
- Critical Appraisal Skills Programme (CASP) - Critical appraisal tools for Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, and more.
- Eco-Evidence - Includes analyser software you can download which helps in the evaluation of evidence from environmental science literature. You will need to register to use the tools.
- GATE (Graphic Appraisal Tool for Epidemiology)
- International Centre for Allied Health Evidence: Critical Appraisal Tools (University of South Australia)
- Joanna Briggs Institute - critical appraisal tools - A range of appraisal tools for all different study types.
- Standard quality assessment criteria for evaluating primary research papers from a variety of fields - Canadian paper by Kmet, Lee & Cook.
- Center for Evidence-Based Management (CEBMa) - some checklists
After finalizing your article selection relevant data from each of the studies needs to be extracted. Do this in a structured, systematic way, keeping these things in mind:
- Only extract the data that is relevant to your research question
- Create a table or data collection form so that you can summarise the data consistently for each study
- Convert all the data to the same units of measurement, where possible
- Ensure your table headings are easily interpreted by others
- Ensure any abbreviations or acronyms used in the table are explained in the footnotes to that table
- Pilot your data extraction method before you start – especially if more than one person is extracting the data
You must then synthesise the data aquired using a chosen methodology.
Develop a synthesis, or critical overview, which integrates the key findings from all the studies you included, while considering the methodological quality and other pertinent features of each of those studies (such as the sample size, population or context in which each one was conducted).
- Narrative, or descriptive: If you are not completing a full systematic review, or if you are comparing different study designs, this could be the best way for you to go. There are no strict rules, but ensure you minimise bias and maximise credibility by carefully exploring the relationships between studies, assessing the robustness of the synthesis and theorising about the answer to your research question.
- Qualitative synthesis methods: Meta-synthesis, Meta-study or Meta-aggregation. These methods synthesise the findings from very different types of studies, clustering common characteristics into categories or using interpretative tools.
- Meta-analysis: To thoroughly synthesise the results of quantitative studies, this statistical technique results in an estimate of the “average” intervention effect. Data from different studies are weighted depending on the sample size and relevant criteria, and evaluated to work out a cumulative outcome. The results of a meta-analysis are often summarised in a forest plot.
Scoping review/ semi-systematic literature reviews
Differences from the systematic review
The steps for the scoping review are roughly the same as for the full systematic luterature review, however the development of the protocol does not have to be as detailed and is usually not published. The critical appraisal and synethesis of data can also be left out or at least does not have to be as thorough.
Aromataris E, Munn Z. Chapter 1: JBI Systematic Reviews. In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global .. https://doi.org/10.46658/JBIMES-20-02
Grant, M.J. & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), 91-108. DOI: 10.1111/j.1471-1842.2009.00848.x
Hart, C. (2018). Doing a literature review : releasing the research imagination (2nd edition.).
Machi, L. A., & McEvoy, B. T. (2009). The literature review : six steps to success. Corwin Press.
Graph of types of literature review: https://libguides.csu.edu.au/systematicreviews