ICU · Statistics & evidence-based medicine
Guidelines, Evidence-Based Medicine & Quality-Improvement Methodology
Also known as Evidence-based medicine · GRADE · GRADE Evidence to Decision · Clinical guidelines · Guideline development · Quality improvement · PDSA cycle · AGREE II · PICO · Care bundles · Evidence hierarchy · Systematic review · Critical appraisal · CASP · CONSORT · PRISMA · STROBE · Bias · Implementation science · Knowledge translation · CFIR · PARIHS
Guidelines, EBM and quality improvement for the ICU First Part: evidence-based medicine as the integration of best evidence, clinical expertise and patient values; the PICO question and the evidence hierarchy; the GRADE approach to evidence quality (high/moderate/low/very low), recommendation strength (strong/weak), downgrading and upgrading domains, and the Evidence to Decision framework; the guideline-development pipeline from PICO to systematic review to evidence profile to graded recommendation; guideline appraisal (AGREE II); critical appraisal tools (CASP checklists) and reporting guidelines (CONSORT, PRISMA, STROBE); implementation science (knowledge-to-action, CFIR, PARIHS); and QI methodology - PDSA cycles, care bundles, audit and feedback, and the structure-process-outcome framework.
On this page & tools
Your progress
Saved locally on this device.
8 MCQs with explanations
Target exams
Overview
Evidence-based medicine (EBM) is the integration of the best research evidence with clinical expertise and patient values. It provides the tools to ask a clinical question, find and appraise the evidence, turn it into trustworthy guidance, and then close the gap between guidance and practice through quality improvement.[1]
Asking and grading the question
- A clinical question is framed with PICO: the Population, the Intervention, the Comparator, and the Outcome.[1]
- The hierarchy of evidence runs from systematic reviews and meta-analyses of randomised trials, through individual randomised trials, to cohort and case-control studies, case series, and expert opinion at the bottom.[1]
GRADE: turning evidence into recommendations
- The GRADE system (Grading of Recommendations, Assessment, Development and Evaluation) rates each outcome's evidence quality as high, moderate, low, or very low, and each recommendation as strong or weak.[1]
- Quality is downgraded for risk of bias, inconsistency, indirectness, imprecision, and publication bias, and may be upgraded for a large effect or a dose-response gradient. The strength of a recommendation also weighs the balance of benefits and harms, patient values, and resource use.[1]
- Major ICU guidelines - such as the Surviving Sepsis Campaign - use GRADE, so a strong, high-quality recommendation reflects both trustworthy evidence and a clear benefit-harm balance.[1]
Appraising a guideline: AGREE II
- The AGREE II instrument appraises the quality of a guideline across six domains: scope and purpose; stakeholder involvement; rigour of development; clarity of presentation; applicability; and editorial independence (freedom from industry bias).[1]
Quality-improvement methodology
- PDSA cycles (Plan-Do-Study-Act) are the iterative engine of improvement: plan a change, test it, study the result, act on what is learned, and repeat.[1]
- Care bundles group a small number of evidence-based practices that, performed together and reliably, improve outcomes (for example the central-line and ventilator bundles).[1]
- Other tools: audit and feedback, checklists, process mapping, root-cause analysis, and statistical process control charts that distinguish real change from random variation.[1]
Measuring outcomes: structure, process, outcome
- The Donabedian framework structures quality measurement into structure (the setting and resources), process (what is done to the patient), and outcome (the result). All three matter - a good outcome requires the right structures supporting the right processes.[1]
SAQ — Appraising a clinical practice guideline with AGREE II and GRADE
10 minutes · 10 marks
You are the ICU representative on the hospital antimicrobial stewardship committee reviewing a new national guideline on the use of procalcitonin-guided antibiotic discontinuation in critically ill patients with sepsis. The guideline recommends procalcitonin-guided discontinuation as a 'weak recommendation, moderate-quality evidence'. The committee asks how to assess the trustworthiness of the guideline and what the GRADE rating means.
SAQ — Critical appraisal of an RCT using the CASP checklist (worked example: PRORATA)
10 minutes · 10 marks
You are shown the PRORATA trial (Bouadma, NEJM 2010): a multicentre randomised trial of procalcitonin-guided vs standard antibiotic discontinuation in 621 critically ill patients with suspected bacterial infection. The primary outcome was mortality at 28 days; secondary outcomes included days of antibiotic exposure. The procalcitonin group had fewer days of antibiotics with no difference in mortality. Using the CASP RCT checklist, critically appraise the trial.
Red flags
The evidence hierarchy in depth
The classical pyramid of evidence ranks study designs by the degree to which they minimise bias and confounding. At the apex sits a well-conducted systematic review with meta-analysis of randomised trials; below it the individual randomised controlled trial (RCT); then cohort studies (prospective follow-up of exposed vs unexposed groups), case-control studies (retrospective comparison of those with and without the outcome), case series and case reports, and — at the base — mechanistic reasoning and expert opinion. The logic is that control of bias increases as you ascend: randomisation balances known and unknown confounders between groups, blinding minimises performance and detection bias, and a meta-analysis pools trials to narrow the estimate of effect and increase precision.[18][2]
Two cautions frame the modern reading of the pyramid. First, design is necessary but not sufficient: a small, single-centre, poorly blinded RCT may be less trustworthy than a large, well-conducted prospective cohort. GRADE therefore starts from the design (RCTs begin at high quality; observational studies at low) but then moves the rating up or down according to how the study was actually conducted — execution can override the starting position. Second, the pyramid has been redrawn: many groups now place a "living" systematic review and the systematic review itself at the apex, and recognise that some questions (prognosis, diagnostic accuracy, rare harms, long-term outcomes) cannot ethically or practically be answered by an RCT and are best answered by high-quality observational designs.[4][5]
Study designs and where they sit in the evidence hierarchy
| Level | Study design | What it does | Strengths | Key weaknesses / bias |
|---|---|---|---|---|
| I | Systematic review + meta-analysis of RCTs | Explicitly searches, appraises, and pools all relevant RCTs to a defined question | Most precise, least biased estimate of effect; reduces uncertainty | Garbage-in/garbage-out: a meta-analysis of flawed trials is still flawed; sensitive to publication bias |
| II | Randomised controlled trial (RCT) | Allocates participants to intervention vs comparator by chance, ideally with blinding | Controls for known and unknown confounders; establishes causation | Expensive; external validity (applicability to your patient); ethical/practical limits; small trials underpowered |
| III | Cohort study (prospective or retrospective) | Follows groups defined by exposure over time; compares incidence of outcome | Suitable for harm, prognosis, rare exposures; temporal sequence; real-world population | Residual confounding; selection of unexposed group; loss to follow-up (attrition bias) |
| III | Case-control study | Compares people with the outcome (cases) to those without (controls), looking back at exposure | Efficient for rare outcomes/diseases; quick, cheap | Recall bias; difficult control selection; cannot estimate incidence; vulnerable to confounding |
| IV | Case series / case report | Describes a consecutive series (or single patient) with a condition/intervention | Generates hypotheses; describes novel presentations/treatments | No comparator; no control of confounding; no causal inference possible |
| V | Expert opinion / mechanistic reasoning | Inference from physiology, bench science, or consensus | Fills gaps when no evidence exists; useful for device design, physiology | Subject to bias; frequently overturned by trials; lowest certainty |
The levels of evidence — design alone (OCEBM 2011 simplified, as adapted by GRADE)
| Rating | Study type | GRADE starting certainty | Plain meaning |
|---|---|---|---|
| Level 1 | Systematic review of homogeneous RCTs; individual high-quality RCT | High | Further research is very unlikely to change our confidence in the estimate of effect |
| Level 2 | Lower-quality RCT (e.g. <80% follow-up, unblinded subjective outcome) | Moderate (downgraded from High) | Further research is likely to have an important impact and may change the estimate |
| Level 3 | High-quality cohort study with consistent results | Low (observational starting point) | Further research is very likely to have an important impact; estimate is uncertain |
| Level 4 | Case-control, poor-quality cohort, case series | Low → Very low | Any estimate of effect is very uncertain |
| Level 5 | Expert opinion, mechanistic reasoning | Very low | Unsupported by direct evidence |
- The RCT is the gold standard for questions of therapy or harm because randomisation is the only design that balances unknown confounders. A cohort study, however large, balances only the confounders the investigators measured and remembered to record; an unmeasured confounder can entirely explain an observed association. This is why an observational finding that contradicts a well-conducted RCT is almost always resolved in favour of the RCT (the HERS and WHI hormone-replacement trials overturning the observational benefit is the classic example).[19]
- Some questions cannot be answered by an RCT, and the hierarchy is not dogma. Prognosis, diagnostic test accuracy, the natural history of a disease, the causative role of a rare exposure (e.g. clear-cell vaginal cancer and diethylstilboestrol), and very rare harms (e.g. Guillain–Barré after vaccination) are best answered by observational designs. A guideline panel therefore picks the best available design for the question, not the highest design on the pyramid.[5]
PICO — framing the question
The well-built clinical question is the entry point to evidence-based practice: without a focused question you cannot search efficiently, and the question determines the study design you seek. PICO (Population, Intervention, Comparator, Outcome) is the standard structure; it can extend to PICOT (adding Timing/duration of follow-up) or PICOS (adding Study design).[3]
The PICO components — with a worked ICU example
| Element | What it specifies | Worked example: early enteral nutrition in severe pancreatitis |
|---|---|---|
| P — Population | The patients: who, with what condition, in what setting | Adult ICU patients with severe acute pancreatitis |
| I — Intervention | The exposure or treatment of interest | Early (within 48 h) enteral nutrition |
| C — Comparator | The alternative — another treatment, placebo, or no treatment | Delayed (>72 h) enteral nutrition, or nil-by-mouth with total parenteral nutrition |
| O — Outcome | The patient-important result you care about (prioritise clinically important, not surrogate) | Infected pancreatic necrosis, mortality, ICU length of stay, need for surgery |
| (T) — Timing | Duration of follow-up | In-hospital and 90-day outcome |
| (S) — Study design | The design that best answers the question | Randomised controlled trial (therapy question) |
- Specify patient-important outcomes, not surrogates. A PICO built on a surrogate (fall in C-reactive protein, change in a biomarker) answers a different — and usually less useful — question than one built on mortality, infection, ventilator-free days, or quality of life. GRADE asks panels to rate each outcome critical, important, or not important, and to make recommendations on the critical/important outcomes only.[6]
GRADE in depth — certainty of evidence and strength of recommendation
GRADE is the dominant system for grading the certainty (quality) of a body of evidence and the strength of a recommendation. It begins by grading each outcome separately (different outcomes in the same study can carry different certainty), starting RCTs at high and observational studies at low, then moving the rating up or down. The four certainty ratings are high, moderate, low, and very low.[4][7]
GRADE — the four levels of certainty (quality) of a body of evidence
| Certainty | Definition (Guyatt/Balshem) | Practical meaning at the bedside |
|---|---|---|
| High | Further research is very unlikely to change our confidence in the estimate of effect | The true effect lies close to the estimate; act on it |
| Moderate | Further research is likely to have an important impact on confidence and may change the estimate | The effect probably lies close to the estimate, but residual uncertainty remains |
| Low | Further research is very likely to have an important impact and is likely to change the estimate | The estimate is uncertain; any estimate is provisional |
| Very low | Any estimate of effect is very uncertain | The true effect is essentially unknown; treat the estimate with extreme caution |
GRADE — the five factors that LOWER certainty, and the three that can RAISE it
| Direction | Domain | What it means | How it is judged |
|---|---|---|---|
| Lower | Risk of bias (study limitations) | Most of the evidence comes from studies with methodological flaws | Cochrane RoB 2 / ROBINS-I; overall judgment of serious / very serious limitations |
| Lower | Inconsistency | Results across studies disagree (point estimates in opposite directions, wide CIs that don't overlap, high I²) | Heterogeneity of effect size, direction, and statistical tests (I², χ²) |
| Lower | Indirectness | The evidence does not directly match the question: different P, I, or O; surrogate outcomes; indirect comparison | Compare the evidence PICO to the question PICO; surrogate vs patient-important outcome |
| Lower | Imprecision | Few events, small sample, wide confidence intervals that cross the threshold for clinical action | 95% CI crosses decision threshold; fails to meet the optimal information size (OIS) |
| Lower | Publication bias | Unpublished negative studies are systematically missing, inflating the apparent effect | Funnel-plot asymmetry, small-study effects, suspicion from an underpowered positive finding |
| Raise | Large magnitude of effect | The observed effect is so large (e.g. relative risk > 2 or < 0.5) that bias is unlikely to explain it all | Look at the pooled point estimate vs the threshold for unaccounted confounding |
| Raise | Dose-response gradient | A biologically plausible increase in effect with increasing dose/exposure | Demonstrated trend across exposure strata |
| Raise | Plausible confounding | All plausible residual confounding would have reduced the demonstrated effect (or produced a spurious null when an effect is present) | Direction of likely confounding works against the observed finding |
GRADE — strong vs weak recommendation, and what each means at the bedside
| Strong recommendation | Weak / conditional recommendation | |
|---|---|---|
| Wording | "We recommend…" | "We suggest…" / "consider…" |
| For the clinician | Most patients should receive the intervention; the benefit-harm balance is clearly favourable | Different patients will reasonably choose different options; weigh the individual patient's circumstances and values |
| For the patient | Most well-informed people would want this; you can adopt it as the default | Many would choose this, but a substantial minority would not; shared decision-making is essential |
| For policy | Suitable as a standard of care / quality metric / care-bundle element | Generally not suitable as a rigid standard; document the choice |
| Determinants | Clear net benefit (or clear net harm → strong against); high certainty; consistent values; acceptable cost | Uncertain or finely balanced benefit-harm; low certainty; variable values; high or uncertain cost |
- The direction and strength of a recommendation weigh five determinants: (1) the magnitude and certainty of the balance between desirable and undesirable outcomes; (2) the certainty of the evidence; (3) the values and preferences of patients (and their variability); (4) resource use (cost and cost-effectiveness); and, in the full Evidence to Decision framework, (5) equity, acceptability, and feasibility. This is why a panel can deliver a strong recommendation on low-certainty evidence (when the balance is one-sided) or a weak recommendation on high-certainty evidence (when the balance is close).[10][12]
The Evidence to Decision (EtD) framework
The GRADE Evidence to Decision (EtD) framework is the structured, transparent template a panel uses to move from evidence to a recommendation. It forces an explicit judgment across the criteria above, records the rationale, and publishes it alongside the recommendation so that a reader (or an adapting local guideline) can see exactly why the panel decided what it did. The EtD makes the deliberation auditable, which is what separates a trustworthy guideline from a consensus opinion.[12]
The GRADE Evidence to Decision (EtD) framework — the criteria a panel judges
| Criterion | The question the panel answers |
|---|---|
| Priority of the problem | Is this question important enough to address? |
| Certainty of the evidence | How high is the certainty across the critical outcomes? |
| Balance of benefits and harms | Do the desirable effects outweigh the undesirable (harms, burden)? |
| Values and preferences | How do patients value the outcomes, and how variable are those values? |
| Resource use / cost-effectiveness | Is the intervention a fair use of resources? |
| Equity | Will adopting it widen or narrow health inequities? |
| Acceptability | Will patients, clinicians, and the system accept it? |
| Feasibility | Can it actually be implemented in the target setting? |
Guideline development — the pipeline from PICO to graded recommendation
A trustworthy clinical-practice guideline is not a literature summary: it is a structured, reproducible process that converts a clinical question into a graded recommendation, with the evidence and the reasoning made explicit at every step. The pipeline — broadly consistent across GRADE, the WHO handbook for guideline development, and the IOM (now National Academy) standards for trustworthy guidelines — runs from priority-setting through PICO, to systematic review, to evidence profile, to recommendation, to grading, to external review and implementation.[6][13]
Developing a trustworthy clinical-practice guideline — from PICO to graded recommendation
- SET UP THE PANEL AND THE PROCESS. Convene a multidisciplinary, international panel that includes content experts, frontline clinicians, methodologists (GRADE-trained), a patient/consumer representative, and a librarian/epidemiologist. Declare and manage all conflicts of interest (the single biggest threat to credibility). Agree the scope, the timeline, the budget, and the GRADE methods up front, and publish the protocol.[13]
- PRIORITY-SET AND FRAME THE PICO QUESTIONS. Agree which questions matter most (often by surveying clinicians or by a Delphi process). Frame each as a PICO, and define the outcomes of interest, rating each critical, important, or not important. Recommendations are made only on the critical and important outcomes — discard the unimportant ones to avoid noise.[3][6]
- CONDUCT (OR COMMISSION) A SYSTEMATIC REVIEW FOR EACH PICO. Search multiple databases (PubMed/MEDLINE, Embase, Cochrane CENTRAL, CINAHL) with a reproducible strategy; screen titles/abstracts and then full texts in duplicate; extract data; assess the risk of bias of each included study (Cochrane RoB 2 for RCTs, ROBINS-I for non-randomised). Where it is sensible, pool the data in a meta-analysis. Report the review to the PRISMA 2020 standard.[16]
- GRADE THE EVIDENCE AND BUILD THE EVIDENCE PROFILE / SUMMARY OF FINDINGS (SoF) TABLE. For each critical outcome, enter the pooled effect, the number of studies and patients, and the GRADE judgments on the five lowering and three raising domains into a GRADE evidence profile and a reader-facing summary of findings table. The outcome now carries a certainty rating: high, moderate, low, or very low.[7][8]
- MOVE TO THE EVIDENCE TO DECISION (EtD) FRAMEWORK. The panel judges, for each PICO, the balance of benefits and harms, the certainty, the values and preferences, the resource use, the equity, the acceptability, and the feasibility. The deliberation is recorded in a published EtD table so the reasoning is transparent and auditable.[12]
- DECIDE THE DIRECTION AND STRENGTH OF THE RECOMMENDATION. Decide recommend for or against (direction), and strong or weak/conditional (strength), based on the EtD judgments. A strong recommendation follows a clearly favourable (or clearly unfavourable) balance; a weak recommendation follows a close balance, variable values, or important uncertainty.[10][11]
- WORD THE RECOMMENDATION UNAMBIGUOUSLY. Use the GRADE convention: "we recommend" for strong, "we suggest" for weak. State the certainty alongside the strength, link the recommendation to the supporting evidence, and make clear to whom and in what setting it applies. Vague recommendations ("consider…") are a hallmark of a weak guideline.[11]
- EXTERNAL REVIEW, DISSEMINATE, IMPLEMENT, AND UPDATE. Expose the draft to external peer review (including the societies that will endorse it) and to public consultation. Publish with the evidence profiles and EtDs. Plan implementation (see Implementation science below), set review/updating triggers (time- or evidence-based — modern guidelines are increasingly "living"), and audit adherence and outcomes.[19]
AGREE II — the six domains of guideline quality
The AGREE II instrument is the appraisal tool you use to judge whether a guideline you are thinking of following is actually trustworthy. Two or more appraisers score 23 items across six domains on a 1–7 scale; each domain is reported as a standardised percentage. The most heavily weighted domain — and the one that most often separates a good guideline from a poor one — is rigour of development (was the evidence systematically gathered and graded?).[13]
AGREE II — the six domains of guideline quality (and what each asks)
| Domain | # items | What it appraises | Common failure mode |
|---|---|---|---|
| 1. Scope and purpose | 3 | Are the health question(s), objectives, and target population clearly described? | Vague scope; the guideline answers a question nobody asked |
| 2. Stakeholder involvement | 3 | Are the relevant professional groups and the target population (patients) represented in the panel? | Panel of like-minded subspecialists with no frontline or patient voice |
| 3. Rigour of development | 8 | Systematic methods to search and grade the evidence; explicit links from evidence to recommendations; process for updating | The dominant domain — narrative ("expert") review with no systematic search, no GRADE, opaque reasoning |
| 4. Clarity of presentation | 3 | Are recommendations specific, unambiguous, and easily identifiable? | Recommendations buried in prose; key actions unclear |
| 5. Applicability | 4 | Are facilitators and barriers, advice/tools, resource implications, and monitoring criteria described? | No mention of how to implement, at what cost, or how to audit it |
| 6. Editorial independence | 2 | Are the views of the funding body absent, and are competing interests of panel members recorded and managed? | Undeclared industry funding; panel with unmanaged conflicts |
- A guideline that scores poorly on Rigour of Development is not trustworthy, however eminent its authors. The old "expert consensus" or "narrative review" guideline is the failure mode AGREE II was built to expose: without a systematic search and explicit grading, the reader cannot tell whether the recommendation rests on a meta-analysis or on the senior author's opinion. When asked in the exam "would you follow this guideline?", the answer is to apply AGREE II and look first at rigour, then at editorial independence.[13]
Critical appraisal — the checklists and reporting guidelines
Critical appraisal is the structured judgment of whether a study's methods are sound enough to trust its results. Two families of tools serve this purpose: critical appraisal checklists (used by the reader to interrogate a paper — the CASP checklists are the standard) and reporting guidelines (used by the author to ensure a paper reports what the reader needs to judge it — CONSORT, PRISMA, STROBE and the EQUATOR Network family).[1][18]
Critical appraisal tools and reporting guidelines — which to use for which study
| Tool | Type | Applies to | What it ensures |
|---|---|---|---|
| CASP (Critical Appraisal Skills Programme) | Appraisal checklist (reader) | Systematic review, RCT, cohort, case-control, qualitative, diagnostic study | A short question set (the RCT checklist is 11 questions across validity, results, applicability) that teaches you to interrogate a paper; the UK standard for journal clubs and exams |
| CONSORT 2010 (Schulz, BMJ 2010) | Reporting guideline (author) | Randomised controlled trials (parallel-group) | A 25-item checklist + a flow diagram of participant flow (enrolment → allocation → follow-up → analysis); the minimum reporting standard for an RCT |
| PRISMA 2020 (Page, BMJ 2021) | Reporting guideline (author) | Systematic reviews (with or without meta-analysis) | A 27-item checklist + an updated flow diagram (identification → screening → eligibility → inclusion across databases and registers); supersedes PRISMA 2009 |
| STROBE (von Elm, PLoS Med 2007) | Reporting guideline (author) | Observational studies — cohort, case-control, cross-sectional | A 22-item checklist ensuring observational studies report the methods (exposure, outcome, confounders, statistical methods) needed to appraise them |
| Cochrane RoB 2 / ROBINS-I | Risk-of-bias tool (reviewer) | RCTs (RoB 2) / non-randomised studies of interventions (ROBINS-I) | Domain-based judgment (selection, performance, detection, attrition, reporting, other) feeding directly into the GRADE "risk of bias" domain |
| (Other EQUATOR family) | Reporting guideline | STARD (diagnostic accuracy), TRIPOD (prediction models), SPIRIT (trial protocols), ARRIVE (animal studies), CARE (case reports), CHEERS (economic evaluation) | The growing EQUATOR library — there is a reporting guideline for almost every study type |
The domains of bias
Critical appraisal, at its core, asks how plausible it is that bias (systematic error), rather than the intervention, produced the observed result. The five classical domains, now codified in Cochrane RoB 2 and ROBINS-I and feeding directly into GRADE's "risk of bias" domain, are: [1]
The five domains of bias assessed in critical appraisal
| Bias domain | What it is | How it arises | How good design controls it |
|---|---|---|---|
| Selection bias | Systematic differences between the groups being compared | Non-random allocation; differential recruitment; confounding in observational studies | Random allocation with concealment; matched/adjusted observational designs |
| Performance bias | Systematic differences in the care provided, apart from the intervention | Lack of blinding of participants/staff → co-interventions, placebo effects | Blinding of participants and personnel (placebo/sham where feasible) |
| Detection bias | Systematic differences in how outcomes are assessed | Unblinded outcome assessors, especially for subjective outcomes | Blinded outcome assessment; objective, pre-specified outcomes; independent adjudication |
| Attrition bias | Systematic loss of participants to follow-up | Differential drop-out; exclusions after randomisation | Intention-to-treat analysis; high (>80–90%) follow-up; transparent handling of missing data |
| Reporting bias | Selective reporting of outcomes | Pre-specified outcomes with favourable results reported; unfavourable ones omitted | Trial registration (ClinicalTrials.gov) and publication of the protocol; check results vs pre-specified outcomes |
Critical appraisal of an RCT — the CASP approach (the 11 questions, grouped)
- IS THE TRIAL VALID? (Section A — the screen). (Q1) Did the study address a clearly focused question (a clear PICO)? (Q2) Was the assignment of patients to treatments randomised — and was the allocation concealed until the participant was enrolled? (Q3) Were patients, health workers, and study personnel blinded to treatment group? The answer to these three screens out the trials whose results cannot be trusted before you look at the numbers.
- IS THE TRIAL VALID? (Section B — the detail). (Q4) Were the groups similar at baseline (randomisation worked)? (Q5) Apart from the experimental intervention, were the groups treated equally (no co-intervention)? (Q6) Were all the patients who entered the trial properly accounted for at its conclusion — and were they analysed in the groups to which they were randomised (intention to treat)? (Q7) How large was the treatment effect, and how precise was the estimate of effect (the 95% confidence interval)?
- ARE THE RESULTS APPLICABLE? (Section C). (Q8) Can you apply these results to your patient (does the population, the intervention, the comparator, and the outcome match your context — external validity)? (Q9) Were all clinically important outcomes considered (not just a surrogate — were harms, quality of life, and costs reported)? (Q10) Are the likely treatment benefits worth the potential harms and costs (number needed to treat vs number needed to harm)? (Q11) The bottom line — would you offer this to your patient, and if so, to whom and under what conditions?[1]
Systematic reviews and meta-analysis
A systematic review is a structured, reproducible synthesis of all the evidence on a question; a meta-analysis is the statistical pooling of the results of the studies in that review. A systematic review need not contain a meta-analysis (the studies may be too heterogeneous to pool), but every meta-analysis should sit inside a systematic review — pooling the results of a biased literature search produces a precise but wrong answer. The Cochrane Collaboration is the flagship producer of systematic reviews of healthcare interventions; its reviews are the most-cited input to GRADE evidence profiles.[15][16]
- The PRISMA flow diagram tracks the fate of every record: identification → screening → eligibility → inclusion. It exposes the search yield, the number excluded at each stage (with reasons), and the final included set — and is the single figure that proves the review was systematic.
- Heterogeneity is judged, then explained. Statistical heterogeneity (I², the proportion of variation due to between-study differences rather than chance) flags disagreement; the panel then asks why (different populations, doses, outcomes, or methodological quality) and may perform subgroup analysis or sensitivity analysis, or downgrade for inconsistency in GRADE. A fixed-effect model assumes the studies share one true effect; a random-effects model allows the true effect to vary across studies and is the default when heterogeneity is present.
- A forest plot displays each study's effect estimate (square sized by weight) with its 95% confidence interval (horizontal line) and the pooled estimate (diamond). The diamond's width is the precision of the pooled estimate; if it crosses the line of no effect, the result is not statistically significant. [1]
Implementation science — closing the know-do gap
Producing a guideline is not the same as changing practice. On average it takes 17 years for research evidence to reach routine clinical use, and even then only a fraction is reliably delivered — the know-do gap. Implementation science (closely allied to knowledge translation) is the study of methods to promote the systematic uptake of research findings into practice, and it is now treated as the bridge between a GRADE recommendation and an actual change at the bedside.[19]
The discipline rests on three linked questions: what is the evidence-based practice we want adopted (the innovation); why is it not being adopted (the barriers and determinants); and how do we close the gap (the tailored implementation strategies). The landmark error in quality improvement is to assume that publishing a guideline or running an educational session will change practice — they rarely do on their own. Effective implementation requires a diagnosis of the local barriers (knowledge, attitudes, skills, workflow, resources, culture, system incentives) and a selection of strategies matched to those barriers (audit and feedback, reminders, clinical decision support, champions, multidisciplinary bundles, financial or regulatory levers).[19][20]
The major implementation science frameworks
| Framework | Origin | Core idea | Structure | Best used for |
|---|---|---|---|---|
| Knowledge-to-Action (KTA) (Graham, 2006) | Canadian Institutes of Health Research | A knowledge-creation cycle (inquiry → synthesis → products) feeding an action cycle of application | Action cycle: identify problem → adapt knowledge to local context → assess barriers → select/tailor interventions → monitor use → evaluate outcomes → sustain use | Planning an end-to-end implementation project |
| CFIR — Consolidated Framework for Implementation Research (Damschroder, 2009) | US health-services research | Implementation success is determined by five interacting domains | Intervention characteristics, Outer setting, Inner setting, Characteristics of individuals, Process (each with sub-constructs) | Diagnosing why adoption is or is not happening; structuring a barrier assessment |
| PARIHS — Promoting Action on Research Implementation in Health Services (Kitson, 1998) | UK nursing/health-services research | Successful implementation = f(Evidence × Context × Facilitation) | Evidence (research + clinical + patient experience); Context (culture, leadership, measurement); Facilitation (the role that makes it happen) | Deciding the intensity of facilitation and readiness of the setting |
| RE-AIM (Glasgow, 1999) | Public health | Plans and evaluates across five dimensions | Reach, Effectiveness, Adoption, Implementation, Maintenance | Designing and evaluating whether an intervention scales and lasts |
- The active ingredient of implementation is the tailored strategy, not the guideline itself. The Cochrane Effective Practice and Organisation of Care (EPOC) group has shown that no single intervention reliably changes practice; what works is a bundle of implementation strategies matched to the measured barriers. Common high-yield strategies: audit and feedback (small but reliable effect — see the QI file), clinical decision support embedded in the electronic record (e.g. a sepsis-band order set), reminders and checklists, local opinion leaders/champions, multidisciplinary care bundles, educational outreach (academic detailing), and organisational/financial incentives.[19]
- Facilitation is the under-recognised determinant. PARIHS emphasises that even excellent evidence in a receptive context will not implement itself — someone (the facilitator, often a senior nurse or improvement lead) has to make it happen: build the team, run the PDSA cycles, remove the workflow barriers, and sustain the change. The corollary for the exam: when asked "why did this guideline fail to change practice?", reach for the barrier domains (CFIR) and the missing facilitation (PARIHS), not just "education was inadequate".[21]
Clinical pearls
Red flags — must-not-miss methodology pitfalls
Prognosis and evidence
Guidelines, EBM, GRADE & QI methodology — landmark methodological evidence
Sackett et al., BMJ 1996 (PMID 8555924) — "Evidence based medicine: what it is and what it isn't." The editorial that fixed the modern definition: EBM is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients — integrating individual clinical expertise with the best available external clinical evidence from systematic research, and with the patient's values. The conceptual foundation for the entire syllabus.[2]
Evidence-Based Medicine Working Group (Guyatt et al.), JAMA 1992 (PMID 1404801) — "A new approach to teaching the practice of medicine." The paper that introduced EBM by name and reframed medical education around asking answerable questions, appraising the literature, and applying it to the individual patient.[1]
Richardson et al., ACP J Club 1995 (PMID 7582737) — "The well-built clinical question." Introduced PICO — the Population, Intervention, Comparator, Outcome structure that turns an unstructured clinical doubt into a searchable, answerable question and determines the study design to seek.[3]
Atkins et al. (GRADE Working Group), BMJ 2004 (PMID 15205295) — "Grading quality of evidence and strength of recommendations." The paper that established GRADE as an international standard: a transparent, reproducible system for rating the certainty of evidence and the strength of recommendations across outcomes, superseding the competing systems (SORT, CEBM levels) that preceded it.[4]
Guyatt et al., BMJ 2008 (PMID 18436948) — "GRADE: an emerging consensus…" Declared the emerging consensus on GRADE and triggered its adoption by >100 organisations worldwide (WHO, NICE, Cochrane, the Surviving Sepsis Campaign, ACCP, ATS, ESC).[5]
Balshem et al., J Clin Epidemiol 2011 (PMID 21208779) — GRADE guidelines 3: Rating the quality of evidence. Defined the four certainty categories (high, moderate, low, very low) and the operational meaning of each ("further research is very unlikely / likely / very likely to change…").[7]
Guyatt et al., J Clin Epidemiol 2011 (PMIDs 21247734 risk-of-bias, 21839614 imprecision, 21195583 evidence profiles/summary of findings) — the GRADE guidelines series. The methodological engine room: how to assess risk of bias (RoB), how to judge imprecision (optimal information size and decision-threshold CI), and how to present the evidence in a GRADE evidence profile and summary of findings table.[8][9][6]
Andrews et al., J Clin Epidemiol 2013 (PMIDs 23312392 guidelines 14, 23570745 guidelines 15) — GRADE going from evidence to recommendations. Defined the determinants of the direction and strength of a recommendation (balance, certainty, values, resources) and how to word recommendations ("we recommend" vs "we suggest") — the bridge from evidence to a usable recommendation.[10][11]
Alonso-Coello et al., BMJ 2016 (PMID 27353417) — the GRADE Evidence to Decision (EtD) frameworks. The structured, transparent template a panel uses to move from evidence to a recommendation, judging priority, certainty, balance, values, resources, equity, acceptability, and feasibility — and publishing the rationale so the recommendation is auditable.[12]
Brouwers et al. (AGREE Next Steps Consortium), CMAJ 2010 (PMID 20603348) — AGREE II. The 23-item, six-domain instrument for appraising the quality of a clinical-practice guideline, now the international standard (endorsed by WHO, NICE, and the Guidelines International Network).[13]
Schulz et al. (CONSORT Group), BMJ 2010 (PMID 20332509) — CONSORT 2010. The updated 25-item checklist and flow diagram for reporting parallel-group randomised trials — the minimum reporting standard for an RCT.[14]
Moher et al. (PRISMA Group), PLoS Med 2009 (PMID 19621072) and Page et al., BMJ 2021 (PMID 33782057) — PRISMA 2009 and PRISMA 2020. The reporting guidelines for systematic reviews (27-item checklist + flow diagram); PRISMA 2020 is the current standard, updating and expanding the 2009 version to cover registrations, risk of bias, and certainty assessment.[15][16]
von Elm et al. (STROBE Initiative), PLoS Med 2007 (PMID 17941714) — STROBE. The 22-item reporting guideline for observational studies (cohort, case-control, cross-sectional), ensuring observational evidence reports the methods needed to appraise it.[17]
Greenhalgh, BMJ 1997 (PMID 9253275) — "How to read a paper: getting your bearings." The opening paper of the classic BMJ series that taught a generation of clinicians to identify the study design, assess validity, and apply the results — the bedside companion to the formal frameworks.[18]
Grol & Grimshaw, Lancet 2003 (PMID 14568747) — "From best evidence to best practice." The seminal implementation-science review showing that passive dissemination (publishing, lecturing) rarely changes practice and that tailored, barrier-matched, multifaceted strategies are required to close the know-do gap.[19]
Damschroder et al., Implement Sci 2009 (PMID 19664226) — CFIR. The Consolidated Framework for Implementation Research — five interacting domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, process) for diagnosing why adoption does or does not occur.[20]
Kitson et al., Qual Health Care 1998 (PMID 10185141) — PARIHS. Successful implementation = f(Evidence × Context × Facilitation) — the framework that foregrounds facilitation as the active, modifiable ingredient.[21]
Evans et al., Crit Care Med / Intensive Care Med 2021 (PMIDs 34605781 / 34599691) — the Surviving Sepsis Campaign 2021 guidelines. The worked example of GRADE in ICU practice: an international multidisciplinary panel, systematic reviews, GRADE evidence profiles and EtDs, and recommendations each carrying a strength and a certainty, disseminated as a bundle and audited for adherence and outcome.[22][23]
Outcomes: EBM, GRADE, and implementation science are not three separate subjects but one pipeline: ASK (PICO) → ACQUIRE (search) → APPRAISE (CASP, risk of bias) → GRADE (certainty + strength, with an EtD) → APPRAISE THE GUIDELINE (AGREE II) → IMPLEMENT (tailored strategies, PDSA, bundles, audit/feedback, SPC) → MEASURE (Donabedian, leading process metrics) → SUSTAIN AND UPDATE. The Surviving Sepsis Campaign demonstrates every link of this chain in the real ICU; the failure modes are predictable and now well-described — narrative "expert" guidelines (no rigour), reading recommendation strength without its certainty (the GRADE misconception), and assuming publication equals implementation (the know-do gap). Mastering this pipeline is what distinguishes a clinician who follows guidelines from one who can build, appraise, and improve them.[1][22]
References
- [1]Evidence-Based Medicine Working Group (Guyatt G, Cairns J, Churchill D, et al.) Evidence-based medicine. A new approach to teaching the practice of medicine JAMA, 1992.PMID 1404801
- [2]Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS Evidence based medicine: what it is and what it isn't BMJ, 1996.PMID 8555924
- [3]Richardson WS, Wilson MC, Nishikawa J, Hayward RSA The well-built clinical question: a key to evidence-based decisions ACP J Club, 1995.PMID 7582737
- [4]Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW Jr, Zaza S; GRADE Working Group Grading quality of evidence and strength of recommendations BMJ, 2004.PMID 15205295
- [5]Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ; GRADE Working Group GRADE: an emerging consensus on rating quality of evidence and strength of recommendations BMJ, 2008.PMID 18436948
- [6]Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables J Clin Epidemiol, 2011.PMID 21195583
- [7]Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH GRADE guidelines: 3. Rating the quality of evidence J Clin Epidemiol, 2011.PMID 21208779
- [8]Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, Alonso-Coello P, Djulbegovic B, Atkins D, Falck-Ytter Y, Williams JW Jr, Meerpohl J, Norris SL, Akl EA, Schünemann HJ GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias) J Clin Epidemiol, 2011.PMID 21247734
- [9]Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, Devereaux PJ, Montori VM, Freyschuss B, Vist G, Jaeschke R, Williams JW Jr, Murad MH, Sinclair D, Prost Y, Pelletier T, Bhatt M, Schünemann HJ GRADE guidelines 6. Rating the quality of evidence--imprecision J Clin Epidemiol, 2011.PMID 21839614
- [10]Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl J, Coello PA, Rind D, Montori VM, Brito JP, Norris S, Elbarbary M, Post P, Nasser M, Shukla V, Jaeschke R, Brozek J, Djulbegovic B, Guyatt G GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations J Clin Epidemiol, 2013.PMID 23312392
- [11]Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, Nasser M, Meerpohl J, Post PN, Kunz R, Brozek J, Schünemann HJ GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength J Clin Epidemiol, 2013.PMID 23570745
- [12]Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Rada G, Rosenbaum S, Morelli A, Guyatt GH, Molina AA; GRADE Working Group GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction BMJ, 2016.PMID 27353417
- [13]Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, Littlejohns P, Makarski J, Zitzelsberger L; AGREE Next Steps Consortium AGREE II: advancing guideline development, reporting and evaluation in health care CMAJ, 2010.PMID 20603348
- [14]Schulz KF, Altman DG, Moher D; CONSORT Group CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials BMJ, 2010.PMID 20332509
- [15]Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement PLoS Med, 2009.PMID 19621072
- [16]Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D The PRISMA 2020 statement: an updated guideline for reporting systematic reviews BMJ, 2021.PMID 33782057
- [17]von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies PLoS Med, 2007.PMID 17941714
- [18]Greenhalgh T How to read a paper. Getting your bearings (deciding what the paper is about) BMJ, 1997.PMID 9253275
- [19]Grol R, Grimshaw J From best evidence to best practice: effective implementation of change in patients' care Lancet, 2003.PMID 14568747
- [20]Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science Implement Sci, 2009.PMID 19664226
- [21]Kitson A, Harvey G, McCormack B Enabling the implementation of evidence based practice: a conceptual framework Qual Health Care, 1998.PMID 10185141
- [22]Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith CM, French C, Machado FR, McIntyre L, O'Sullivan BE, Prescott HC, Schorr CA, Simpson SQ, Welte T, Angus DC, Beale R, Bihari S, Cepinskas G, De Backer D, Dombrovskiy V, Doble A, Dyer J, Eaton J, Ferrer R, Giraudo M, Hall RT, Jaehne AK, Kaufman DA, Heitmann S, Klompas M, Kotiso B, Kurz M, Llitjos JF, Luyt CE, Maskin B, Mehta Y, Meleiro A, Mer M, Mourtzoukou EG, Oczkowski S, Palleschi L, Perner A, Poulsen L, Preiser JC, Renton M, Ranzani OT, Rojek A, Salluh J, Schuetz P, Senzolo M, Sharif S, Siddiqui S, Singer M, Solé-Violán J, Soriano C, Szakmany T, Talmor D, Torres A, Vieira S, Waeber A, Wei N, Zimmerman J, Levy M Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2021 Crit Care Med, 2021.PMID 34605781
- [23]Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith CM, French C, Machado FR, McIntyre L, O'Sullivan BE, Schorr CA, Simpson SQ, Welte T, Angus DC, Coopersmith CM, French C, Machado FR, Levy MM Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021 Intensive Care Med, 2021.PMID 34599691