Skip to Main Content Home Ask a Librarian

Systematic Reviews

This guide includes content about systematic reviews, including general information and information about librarian assistance.

Search Process

What is a protocol?

A systematic review protocol describes the rationale, hypothesis, and planned methods of the review. It should be prepared before a review is started and used as a guide to carry out the review.

Why register a systematic review protocol?

A systematic review protocol is important for several reasons:

  1. It allows systematic reviewers to plan carefully and thereby anticipate potential problems
  2. It allows reviewers to explicitly document what is planned before they start their review, enabling others to compare the protocol and the completed review (that is, to identify selective reporting), to replicate review methods if desired, and to judge the validity of planned methods;
  3. It prevents arbitrary decision making with respect to inclusion criteria and extraction of data
  4. It may reduce duplication of efforts and enhance collaboration, when available

Before moving forward with any systematic review, we recommend filling out a Systematic Review Protocol in PROSPERO or Open Science Framework. Systematic reviews are a time commitment, taking an average of 67.5 weeks or 15 months (see our Systematic Review Timeline page for an in-depth breakdown of the timeline/phases of a systematic review). Successfully filling out a protocol gives an insight into the amount of work that goes into a systematic review and may help increase chances of completing a systematic review. A registered systematic review protocol can also count as a citation.

Registering a protocol:

Systematic Reviews

Scoping Reviews

Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ, 349, g7647. https://doi.org/10.1136/bmj.g7647

What is the PICO format?

PICO is a format for developing a good clinical research question prior to starting one’s research. The PICO process starts with a case scenario from which a question is constructed that is relevant to the case and is phrased in such a way as to facilitate finding an answer. Once a well-structured question is formulated, researchers will be in a better position to search the literature for evidence that will support their original PICO question.

Part of the Acronym Explanation Example
P or Population Who are the relevant patients or the target audience for the problem being addressed? Adults (people over the age of 18)
I or Intervention What intervention is being considered? Interactive social media, such as blogs and microblogs (e.g. Twitter), content communities (e.g. Youtube), virtual social networks (e.g. Facebook), and webpages and wiki
C or Comparator What is the main comparator to the intervention that you want to assess? Non-interactive social media programs
O or Outcome What are the consequences of the interventions for the patient? Or what are the main outcomes of interest to the patient or decision maker? Health behaviors, health, mental health, well-being, and whether people reported unwanted effects

Other frameworks:

Question type Framework
Etiology/ Risk/ Benefit PEO: Population, Exposure(s), Outcomes
Prevalence/ Incidence PCS: Population, Condition, Setting or context
Diagnostic Test Accuracy PIRD: Population, Index Test, Reference Test, Diagnosis of
Interest
Psychometric PTM: Population, Type of measurement instrument,
Measurement properties

Read more:

Jensen, K. (2018, January 9). Seven Steps to the Perfect PICO Search. Health Noteshttps://www.ebsco.com/blogs/health-notes/seven-steps-perfect-pico-search 

Petkovic, J., Duench, S., Trawin, J., Dewidar, O., Pardo Pardo, J., Simeon, R., DesMeules, M., Gagnon, D., Hatcher Roberts, J., Hossain, A., Pottie, K., Rader, T., Tugwell, P., Yoganathan, M., Presseau, J., & Welch, V. (2021). Behavioural interventions delivered through interactive social media for health behaviour change, health outcomes, and health equity in the adult population. The Cochrane database of systematic reviews5(5), CD012932. https://doi.org/10.1002/14651858.CD012932.pub2

What is inclusion/exclusion criteria?

Inclusion and exclusion criteria should be established during protocol development or before proceeding with the searches. It is used to determine what studies from the comprehensive, reproducible searches will be used in your article to answer your research questions or objectives. 

"The population, intervention and comparison components of the question, with the additional specification of types of study that will be included, form the basis of the pre-specified eligibility criteria for the review. It is rare to use outcomes as eligibility criteria: studies should be included irrespective of whether they report outcome data, but may legitimately be excluded if they do not measure outcomes of interest, or if they explicitly aim to prevent a particular outcome." (McKenzie et al., 2022)

Read more in Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis

McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

Petkovic, J., Duench, S., Trawin, J., Dewidar, O., Pardo Pardo, J., Simeon, R., DesMeules, M., Gagnon, D., Hatcher Roberts, J., Hossain, A., Pottie, K., Rader, T., Tugwell, P., Yoganathan, M., Presseau, J., & Welch, V. (2021). Behavioural interventions delivered through interactive social media for health behaviour change, health outcomes, and health equity in the adult population. The Cochrane database of systematic reviews, 5(5), CD012932. https://doi.org/10.1002/14651858.CD012932.pub2

What is Grey Literature?

"Gray literature, or evidence not published in commercial publications, can make important contributions to a systematic review. Gray literature can include academic papers, including theses and dissertations, research and committee reports, government reports, conference papers, and ongoing research, among others. It may provide data not found within commercially published literature, providing an important forum for disseminating studies with null or negative results that might not otherwise be disseminated. Gray literature may thusly reduce publication bias, increase reviews’ comprehensiveness and timeliness, and foster a balanced picture of available evidence." (Paez, 2017)

 

Read more:

  • Cooper, C., Lovell, R., Husk, K., Booth, A., & Garside, R. (2018). Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. Research Synthesis Methods, 9(2), 195–223. https://doi.org/10.1002/jrsm.1286
  • Delaney, A., & Tamás, P. A. (2018). Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Research Synthesis Methods, 9(1), 124–131. https://doi.org/10.1002/jrsm.1282
  • Haddaway, N. R. (n.d.). The Use of Web-scraping Software in Searching for Grey Literature. Retrieved May 26, 2021, from https://www.academia.edu/16570122/The_Use_of_Web_scraping_Software_in_Searching_for_Grey_Literature
  • Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015). The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PloS One, 10(9), e0138237. https://doi.org/10.1371/journal.pone.0138237
  • Lee, M. S., Hughes, A., Lockmiller, C., Day, A., Brown, M., & Jenson, R. (2022). Working Together: How Academic Librarians Can Help Researchers Prepare for a Grey Literature Search for Systematic Reviews Involving Minoritized Populations. The Journal of Academic Librarianship, 102626. https://doi.org/10.1016/j.acalib.2022.102626
  • Saleh, A. A., Ratajeski, M. A., & Bertolet, M. (2014). Grey Literature Searching for Health Sciences Systematic Reviews: A Prospective Study of Time Spent and Resources Utilized. Evidence Based Library and Information Practice, 9(3), 28–50. https://doi.org/10.18438/b8dw3k

Paez, A. (2017). Gray literature: An important resource in systematic reviews. Journal of Evidence Based Medicine. 10(3), 233-240. https://doi.org/10.1111/jebm.12266 

Critical Appraisal Tools:

Critical appraisal is an integral part of evidence-based medicine. It can also be referred to as "quality assessment" or "risk of bias". The organizations listed below offer a variety of checklists to help with the process of critically appraising evidence


Risk of Bias Tools:

Read more:

  • Downes, M. J., Brennan, M. L., Williams, H. C., & Dean, R. S. (2016). Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open, 6(12), e011458. https://doi.org/10.1136/bmjopen-2016-011458

  • Murad, M. H., Sultan, S., Haffar, S., & Bazerbachi, F. (2018). Methodological quality and synthesis of case series and case reports. BMJ Evidence-Based Medicine, 23(2), 60–63. https://doi.org/10.1136/bmjebm-2017-110853
  • Risk of bias tools—RoB 2 tool. (n.d.). Retrieved July 6, 2023, from https://www.riskofbias.info/welcome/rob-2-0-tool

  • Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017). Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Agency for Healthcare Research and Quality. https://doi.org/10.23970/AHRQEPCMETHGUIDE2  

  • Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018). Recommendations for assessing the risk of bias in systematic reviews of health-care interventions. Journal of Clinical Epidemiology, 97, 26–34. https://doi.org/10.1016/j.jclinepi.2017.12.004