A Guide To Critical Appraisal

JUMP TO

1. General Overview
2. Specific Study Designs
   1. Randomized Controlled Trials
   2. Cohort Studies
   3. Systematic Reviews

Research Question & Hypothesis

• Is the research question addressing a focused issue?

Study Design

• Is the study design appropriate to be used for the research question(s)?
• Where does it lie in the hierarchy of evidence in research?

PICO elements

• What are the PICO elements based on your study/ clinical scenario?

Study population  

• Are the groups defined precisely? 
• What are the inclusion/ exclusion criteria from the study?
• Were the participants representative of a defined population (geographically and/or temporally)?
• Was there an established reliable system for selecting the participants? (e.g random/stratified sampling)
• Was the sample size adequate for significant results to be obtained? 
• Were there any differences between the control & intervention groups apart from the exposure?
• Were there any differences in the baseline characteristics of the study participants?
• Was selection bias present?

Outcomes/Endpoints

• What were the primary and secondary outcomes of the study?
• How were the outcomes measured?
• Is the time frame of the study relevant to disease/exposure?

Statistical Analysis 

• What were the analysis methods used?
• Were all participants analyzed? (including those lost to followup)
• Were confounding factors outlined and accounted for? If so, what were the methods?

Results

• What are the results of the study?
• Were the results statistically significant? (consider the confidence interval)
• Were the results precise? (consider the p-value)
• Were all variables assessed? 
• Consider the Number Needed to Treat/Harm (NNT/NNH)

Evaluation of Study

• Were there any limitations in the study? If so, what were the limitations?
• Was followup of patients sufficiently long and complete?

Applied Clinical Significance

• Can the results obtained from this study be subsequently reproduced in a separate study? (ie. results are not due to chance and are consistent with evidence from other studies)
• Is the treatment/intervention beneficial to the local population of patients?

Conclusions 

• Is the treatment/intervention feasible in the local setting?
• Were the results of the study reliable?
• Were all clinical outcomes considered?
• Do the benefits of treatment/intervention outweigh the potential risks of harm and costs?

Randomized Controlled Trials (RCTs)

A. Are the results of the trial valid?

• Was the issue addressed by the trial clearly focused?
• Was the assignment of patients to treatments/ interventions randomized? 
• Were the groups similar at the start of the trial?
• Were patients/ participants, healthcare workers and study personnel ‘blinded’ to the treatment? ( e.g single/double blinded)
• Were all patients analyzed in the groups to which they were randomized at the study conclusion? 
• Apart from the intervention, were the groups treated equally?

B. What were the results?

• How large was the treatment effect? 
• How precise was the estimate?

C. Are the results applicable and helpful in practice?

• Can the results be applied in context (or to a local population)?
• Are the benefits from the treatment/ intervention worth the potential harms and costs?

Cohort Studies

A. Are the results of the trial valid?

• Did the study address a ‘focused’ issue? 
• Was the cohort recruited in an acceptable way?
• Were there clearly defined groups of participants, similar in other features, apart from the exposure of interest?
• Was the detection of outcomes accurate and similar in both groups to minimize bias?
• Have the authors identified possible confounding factors? (If so, were statistical adjustments made?)
• Was the follow-up of subjects complete? (Were any drop-outs accounted for?)

B. What were the results?

• What is the relative risk/ risk ratio? 
• How strong is the association between the exposure and the outcome?
• How precise is the estimate of the risk? (i.e, confidence interval of the relative risk) 
• Reliability of results 
  • Any bias, chance or confounding present?
  • Are the design or method flawed, making the results unreliable? 

C. Are the results applicable and helpful locally?

• Can the results be applied to the local population?
• Do the results of this study fit with other available evidence?
• What are the implications of this study to clinical practice?

Systematic Reviews

A. Are the results of the trial valid?

• Did the review address a focused question?
  • Yes/no and why?
  • Did the studies possess an appropriate study design to the review question?
• Selection criteria of papers included
• Relevancy and adequacy of papers reviewed
  • Bibliographic databases used
  • Follow up from reference lists 
  • Personal contact with experts
  • Search for unpublished as well as published studies 
  • Search for non-English language studies
• Methods of assessing paper’s quality 
• Were the results of studies combined?
  •  If so, was it appropriate to do so?
  • Were the results similar from study to study?
  • Were results of included studies clearly stated? 
  •  Were variations in results justified/explained?

B. What were the results?

• Overall results
  • How were they reported? ((NNT/NNH, RR, OR etc.)
  • Interpretation of results 
• Precision of results
  • Confidence Interval & Interpretation

C. Are the results applicable and helpful locally?

• Can the results be applied in context (or to a local population)?
  • How similar are the review’s population to the local population
• Were all important outcomes considered?
  • Are there any pertinent outcomes that were overlooked in this study ?
• Are the benefits worth the harm and cost?

Authors: Alaa Hamdi, Mikhirish Kumaran (SIGMUM 2021/2022)

References

1. Individual Summative Assessment Marking Grid – Critical Appraisal of a Study of Therapy 2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
2. The University of British Columbia – Department of Emergency Medicine [Internet].CASP COHORT STUDY CHECKLIST; 2021 [cited 20 September 2021]; [2,4-6] Available from: https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Cohort-Study-Checklist.pdf
3. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada : Department of Emergency Medicine. CASP RCT  STUDY CHECKLIST; 2021 [cited 20 September 2021]; [2-5]. Available from: https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-RCT-Checklist.pdf
4. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada: Department of Emergency Medicine;  2013[cited 2021Sep20]. Available from:https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Cohort-Study-Checklist.pdf
5. Summative Assessment Task. CAT Harm Worksheet (RCT COHORT)2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
6. Individual Summative Assessment Marking Grid – Critical Appraisal of a Study of Harm 2021 [assignment notes on Internet]. Melbourne: Monash University, Faculty of Medicine, Nursing and Health Sciences; 2021 [cited 20 September 2021].
7. The University of British Columbia. Critical Appraisal Checklist [Internet]. Vancouver, BC Canada: Department of Emergency Medicine;  2013[cited 2021Sep20]. Available from:https://med-fom-emerg.sites.olt.ubc.ca/files/2014/02/CASP-Systematic-Review-Checklist.pdf

---

Other guides