A Guide To Critical Appraisal

An overview of the fundamental elements that one must evaluate when analyzing a study.

JUMP TO

  1. General Overview
  2. Specific Study Designs
    1. Randomized Controlled Trials
    2. Cohort Studies
    3. Systematic Reviews

General Overview

Research Question & Hypothesis

  • Is the research question addressing a focused issue?

Study Design

  • Systematic Review
  • Randomized Controlled Trials
  • Cohort Studies
  • Is the study design appropriate to be used for the research question(s)?
  • Where does it lie in the hierarchy of evidence in research?


Note: Please refer to Specific Study Designs for a breakdown regarding each study design.


PICO elements

  1. Population
  2. Intervention
  3. Comparison
  4. Outcome
  • What are the PICO elements based on your study/ clinical scenario?

Study population  

  • Are the groups defined precisely? 
  • What are the inclusion/ exclusion criteria from the study?
  • Were the participants representative of a defined population (geographically and/or temporally)?
  • Was there an established reliable system for selecting the participants? (e.g random/stratified sampling)
  • Was the sample size adequate for significant results to be obtained? 
  • Were there any differences between the control & intervention groups apart from the exposure?
  • Were there any differences in the baseline characteristics of the study participants?
  • Was selection bias present?

Outcomes/Endpoints

  • Primary & Secondary
  • Outcome Measures 
  • What were the primary and secondary outcomes of the study?
  • How were the outcomes measured?
  • Is the time frame of the study relevant to disease/exposure?

Statistical Analysis 

  • What were the analysis methods used?
  • Were all participants analyzed? (including those lost to followup)
  • Were confounding factors outlined and accounted for? If so, what were the methods?

Results

  • What are the results of the study?
  • Were the results statistically significant? (consider the confidence interval)
  • Were the results precise? (consider the p-value)
  • Were all variables assessed? 
  • Consider the Number Needed to Treat/Harm (NNT/NNH)

Evaluation of Study

Strengths & Limitations

  • Were there any limitations in the study? If so, what were the limitations?
  • Was followup of patients sufficiently long and complete?

Applied Clinical Significance

  • Reproducibility of trial (if beneficial)
  • Significance in local population 
  • Can the results obtained from this study be subsequently reproduced in a separate study? (ie. results are not due to chance and are consistent with evidence from other studies)
  • Is the treatment/intervention beneficial to the local population of patients?

Conclusions 

  • Is the treatment/intervention feasible in the local setting?
  • Were the results of the study reliable?
  • Were all clinical outcomes considered?
  • Do the benefits of treatment/intervention outweigh the potential risks of harm and costs?

Specific Study Designs

Randomized Controlled Trials (RCTs)

A. Are the results of the trial valid?

  • Was the issue addressed by the trial clearly focused?
  • Was the assignment of patients to treatments/ interventions randomized? 
  • Were the groups similar at the start of the trial?
  • Were patients/ participants, healthcare workers and study personnel ‘blinded’ to the treatment? ( e.g single/double blinded)
  • Were all patients analyzed in the groups to which they were randomized at the study conclusion? 
  • Apart from the intervention, were the groups treated equally?

B. What were the results?

  • How large was the treatment effect? 
  • How precise was the estimate?

C. Are the results applicable and helpful in practice?

  • Can the results be applied in context (or to a local population)?
  • Are the benefits from the treatment/ intervention worth the potential harms and costs?

Cohort Studies

A. Are the results of the trial valid?

  • Did the study address a ‘focused’ issue? 
  • Was the cohort recruited in an acceptable way?
  • Were there clearly defined groups of participants, similar in other features, apart from the exposure of interest?
  • Was the detection of outcomes accurate and similar in both groups to minimize bias?
  • Have the authors identified possible confounding factors? (If so, were statistical adjustments made?)
  • Was the follow-up of subjects complete? (Were any drop-outs accounted for?)

B. What were the results?

  • What is the relative risk/ risk ratio? 
  • How strong is the association between the exposure and the outcome?
  • How precise is the estimate of the risk? (i.e, confidence interval of the relative risk) 
  • Reliability of results 
    • Any bias, chance or confounding present?
    • Are the design or method flawed, making the results unreliable? 

C. Are the results applicable and helpful locally?

  • Can the results be applied to the local population?
  • Do the results of this study fit with other available evidence?
  • What are the implications of this study to clinical practice?

Systematic Reviews

A. Are the results of the trial valid?

  • Did the review address a focused question?
    • Yes/no and why?
    • Did the studies possess an appropriate study design to the review question?
  • Selection criteria of papers included
  • Relevancy and adequacy of papers reviewed
    • Bibliographic databases used
    • Follow up from reference lists 
    • Personal contact with experts
    • Search for unpublished as well as published studies 
    • Search for non-English language studies
  • Methods of assessing paper’s quality 
  • Were the results of studies combined?
    •  If so, was it appropriate to do so?
    • Were the results similar from study to study?
    • Were results of included studies clearly stated? 
    •  Were variations in results justified/explained?

B. What were the results?

  • Overall results
    • How were they reported? ((NNT/NNH, RR, OR etc.)
    • Interpretation of results 
  • Precision of results
    • Confidence Interval & Interpretation

C. Are the results applicable and helpful locally?

  • Can the results be applied in context (or to a local population)?
    • How similar are the review’s population to the local population
  • Were all important outcomes considered?
    • Are there any pertinent outcomes that were overlooked in this study ?
  • Are the benefits worth the harm and cost?

Authors: Alaa Hamdi, Mikhirish Kumaran (SIGMUM 2021/2022)

References

  1. Individual Summative Assessment Marking Grid – Critical Appraisal of a Study of Therapy 2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
  2. The University of British Columbia – Department of Emergency Medicine [Internet].CASP COHORT STUDY CHECKLIST; 2021 [cited 20 September 2021]; [2,4-6] Available from: https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Cohort-Study-Checklist.pdf
  3. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada : Department of Emergency Medicine. CASP RCT  STUDY CHECKLIST; 2021 [cited 20 September 2021]; [2-5]. Available from: https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-RCT-Checklist.pdf
  4. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada: Department of Emergency Medicine;  2013[cited 2021Sep20]. Available from:https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Cohort-Study-Checklist.pdf
  5. Summative Assessment Task. CAT Harm Worksheet (RCT COHORT)2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
  6. Individual Summative Assessment Marking Grid – Critical Appraisal of a Study of Harm 2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
  7. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada: Department of Emergency Medicine;  2013[cited 2021Sep20]. Available from:https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Systematic-Review-Checklist.pdf

Other guides