Research methods and study design

This topic is the architecture of evidence, not the arithmetic of appraisal. Critical appraisal teaches you how to stress-test a finished paper; research methods teach you which study structure can answer which clinical question and what structural threats that structure invites.[1][17] Fellowship examiners (FRANZCP written/viva, MRCPsych Paper A/B methods items, ABPN blueprint research design, MD/DNB theory) repeatedly test design identification, design–question matching, and the vocabulary of bias and confounding.

Definition and classification

A study design is the structural plan that links participants, exposures or interventions, timing, comparison, and outcomes so that a claim about association or causation can be defended or rejected.[1] Designs sit on two major trunks: experimental (the investigator assigns the intervention) and observational (the investigator observes naturally occurring exposures). Qualitative inquiry answers different questions about meaning, process, and experience; it is not a low rung of the quantitative pyramid.[15]

Hierarchy of evidence (teaching scaffold, not dogma)

Traditional teaching ranks average protection against confounding: systematic review/meta-analysis of RCTs higher than single RCTs higher than cohort higher than case-control higher than cross-sectional higher than case series and expert opinion.[13] GRADE then rates certainty of a body of evidence (risk of bias, inconsistency, indirectness, imprecision, publication bias) and allows upgrades or downgrades that a rigid pyramid cannot express — a large, carefully analysed observational harm signal can outweigh a tiny open-label RCT of a soft endpoint.[13]

Build the well-built clinical question before searching or designing; PICO is the hinge between clinical uncertainty and method.[16][17]

Epidemiology of design choice — why psychiatry needs the full toolkit

Psychiatry questions span therapy (antidepressant vs placebo), long-term harm (lithium and renal outcomes), rare catastrophic events (clozapine agranulocytosis, completed suicide), population frequency (12-month prevalence of depression), and lived experience (what recovery means after first-episode psychosis). No single design can serve all of these honestly.[1][19]

Psychiatry literature is also structurally hostile: subjective rating-scale outcomes, large placebo responses, high attrition, comorbidity exclusion that narrows external validity, and industry influence that rewards flexible analyses.[19] That is why design literacy is a clinical safety skill, not optional academic polish.

[5] [6] [7] [16]

Mechanisms — bias, confounding, and time direction

Bias is systematic error that distorts the estimate away from truth; larger samples do not fix it. Random error shrinks with sample size. Confounding is a special distortion in which a third factor (common cause) is associated with both exposure and outcome and creates a spurious or exaggerated association.[4]

[4] [9] [11]

Time direction is examinable anatomy: cohorts move forward from exposure to outcome; case-control studies work backward from outcome to prior exposure; cross-sectional studies measure both at once and cannot establish temporal sequence.[2][3] Hill's classic considerations (strength, consistency, specificity, temporality, biological gradient, plausibility, coherence, experiment, analogy) help weigh observational associations toward causal inference without pretending they replace randomised evidence when randomisation is possible.[14]

Clinical presentation — how the exam stem arrives

Expect stems that ask: "Which design best answers this question?"; "Identify the design in this abstract"; "Name the main bias risk"; "Which reporting checklist applies?"; or "Why is an RCT unethical here?". Psychiatry vignettes favour rare harm (agranulocytosis), long-term lithium outcomes, prevalence surveys, psychotherapy trial design limits (blinding), and qualitative service-user research.[1][3]

Differential — designs examiners set as traps

[1] [2] [3] [7]

Absence of a statistically significant difference in an underpowered study is not evidence of equivalence or absence of effect.[18]

Assessment — design selection algorithm

1. Write PICO (or PEO for qualitative: population, exposure/experience, outcome/phenomenon).[16]
2. Classify question: therapy/prevention, harm, prognosis, diagnosis, prevalence, meaning.
3. If therapy/prevention and randomisation ethical and feasible → plan an RCT.
4. If rare exposure or multiple outcomes over time → cohort.
5. If rare outcome → case-control (or nested case-control inside a cohort).
6. If prevalence/frequency → cross-sectional survey with defined sampling frame.
7. If how/why/meaning → qualitative design with explicit methodology.
8. If synthesising existing evidence → systematic review ± meta-analysis when appropriate.

[1] [2] [3]

Deep dive by design

Randomised controlled trial (RCT)

An RCT assigns participants to intervention arms by a random process, aiming to balance known and unknown confounders on average.[1][5] Core elements examiners expect:

• Random sequence generation — computer, random numbers; not alternating days or clinician preference.
• Allocation concealment — central randomisation, sealed opaque envelopes properly executed; sequence protected until assignment.[5][9]
• Blinding — participants, clinicians, outcome assessors as feasible; in psychotherapy trials, assessor blinding and attention-control comparators matter because double-blinding of therapy content is often impossible.
• Baseline balance — check prognostic factors; chance imbalance still possible in small trials.
• Follow-up completeness — differential attrition is attrition bias.
• Intention-to-treat (ITT) — analyse all randomised in assigned arms; preserves randomisation benefits under non-adherence; pure per-protocol answers a different question and often inflates benefit.[10]
• Primary outcome pre-specified — patient-important where possible (remission, hospitalisation, suicide attempt), not only continuous scale drift.

Variants you must name: parallel-group (default), cluster (wards, schools, clinics — unit of randomisation is the cluster), crossover (each participant receives both treatments in sequence; washout and period effects limit use in progressive or learning-sensitive conditions), factorial, pragmatic (effectiveness in routine care) vs explanatory (efficacy under ideal conditions). Superiority seeks better than control; non-inferiority seeks not unacceptably worse than an active standard — failed superiority is not proof of equivalence.[5][18]

RoB 2 domains structure modern risk-of-bias judgment for RCTs: randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, selection of the reported result.[11]

Cohort studies

Participants are defined by exposure status and followed forward for outcomes. Prospective cohorts collect data going forward; retrospective cohorts use existing records but still move conceptually from exposure to outcome.[2] Strengths: incidence, temporal sequence, multiple outcomes, rare exposures. Weaknesses: confounding (especially confounding by indication in pharmacoepidemiology), loss to follow-up, cost/time for rare delayed outcomes, immortal time bias if follow-up windows are mishandled.[2][4][12]

Psychiatry examples: prenatal medication exposure cohorts; first-episode psychosis inception cohorts for prognosis; long-term lithium users and renal outcomes. ROBINS-I is the risk-of-bias tool for non-randomised intervention studies and foregrounds confounding.[12]

Case-control studies

Cases have the outcome; controls do not; investigators look backward at prior exposures — "research in reverse". Efficient for rare outcomes and long latency; does not directly measure incidence; vulnerable to selection of improper controls and recall bias.[3] Controls should represent the study base that gave rise to the cases. Nested case-control designs inside cohorts reduce some selection problems. Psychiatry examples: completed suicide, clozapine agranulocytosis, rare congenital anomalies after in-utero exposure.[3][4]

Cross-sectional studies

Exposure and outcome are measured at (approximately) the same time. Best for prevalence and generating hypotheses; weak for causation because temporality is unclear and reverse causation is common (does insomnia cause depression, or does depression cause insomnia?). Sampling frame and non-response determine external validity.[1][6]

Qualitative research

Addresses meaning, process, and experience — why people disengage from clozapine clinics; how families understand compulsory treatment; barriers to help-seeking in Indigenous communities. Methods (interviews, focus groups, ethnography) and analytic approaches (thematic analysis, grounded theory, phenomenology) must be named and justified; sample size follows saturation/adequacy logic, not power calculations. COREQ provides a 32-item reporting checklist for interviews and focus groups.[15] Trustworthiness language: credibility, transferability, dependability, confirmability. Do not quote qualitative themes as prevalence percentages.

Systematic reviews and meta-analysis

A systematic review uses pre-specified methods to find, select, appraise, and synthesise studies answering a focused question. Meta-analysis is the optional statistical pooling of effect estimates when studies are sufficiently similar; high clinical or statistical heterogeneity may mandate narrative synthesis instead.[7][8][20] PRISMA guides reporting (flow diagram from identification through included studies); it is not a quality certificate — transparent reporting of a poorly designed review is still a poor review.[7] Network meta-analysis compares multiple treatments via a network of direct and indirect comparisons; examiners expect awareness of the transitivity assumption, not software recipes.

Investigations — reporting standards as transparency tools

Acute exam management — first 60 seconds

1. Underline the question type in the stem.
2. Name the design (or the design the authors claim).
3. Ask: is this design capable of answering that question?
4. Name the single worst structural threat for that design (unconcealed allocation; confounding by indication; reverse causation; improper controls; pooling heterogeneous trials).
5. Name the reporting checklist that should have been used.

[1] [5] [6] [7]

Definitive blueprints (exam-ready)

Therapy RCT blueprint. Parallel-group, centrally randomised, allocation concealed, double-blind if feasible, pre-registered primary outcome (e.g. remission at 8–12 weeks), ITT primary analysis, adverse-event capture, CONSORT flow, sample size for a clinically important difference — not only for a p-value hunt.[5][10][11]

Prospective cohort blueprint. Defined exposure (e.g. new SSRI users), unexposed or alternative-exposed comparator, inception at a comparable start point, active follow-up for suicide attempts and hospitalisation, measured confounders (severity, prior attempts, comorbidity), sensitivity analyses, STROBE reporting, ROBINS-I thinking about residual confounding.[2][6][12]

Case-control blueprint. Hard case definition (e.g. agranulocytosis meeting neutrophil thresholds), controls from the same clozapine-exposed source population without the event, exposure ascertainment blinded to case status where possible, careful matching that does not match away the exposure of interest, STROBE case-control items.[3][6]

Subtypes and scenarios

• Cluster RCT for ward-level safety programmes or school mental-health interventions — analyse at correct level; contamination if individuals randomised within clusters.
• Pragmatic antidepressant trial in routine clinics vs explanatory phase-III placebo trial — external vs internal validity trade-off.
• Nested case-control of metabolic syndrome inside an antipsychotic cohort.
• Cross-sectional national survey of 12-month depression prevalence with multistage sampling.
• Qualitative study of carers' experience of Community Treatment Orders using semi-structured interviews and thematic analysis, reported with COREQ.[15]
• Systematic review of CBT for OCD with random-effects meta-analysis only if interventions and populations are poolable; report I-squared and explore heterogeneity.[20][7]

Complications and pitfalls

• Labelling any two-group comparison an "RCT" without random assignment.[1]
• Using case-control data to quote incidence rates without a known sampling fraction.[3]
• Inferring causation from a single cross-sectional association.[4][14]
• Ignoring confounding by indication when comparing two antipsychotics in claims data.[12]
• Equating non-significant underpowered results with "no difference".[18]
• Meta-analysing psychotherapy trials with incompatible controls and outcomes without exploring heterogeneity.[20]
• Treating qualitative findings as if they were prevalence estimates.[15]

Prognosis and disposition of claims

Design choice determines what claim you may make. An ideal RCT supports average causal effect for the enrolled population under trial conditions. A well-done cohort supports associations with temporal order and can estimate incidence; causation requires triangulation (Hill considerations, bias analysis, consistency). A case-control supports relative exposure odds among cases vs controls. A cross-sectional supports prevalence. Qualitative work supports transferable insight into experience, not population rates.[1][14][15] Practice change still integrates best evidence with clinical expertise and patient values — EBM's three legs.[17]

Special populations

Children and adolescents: assent/consent, developmental outcomes, school cluster designs, under-powering. Older adults: multimorbidity and polypharmacy exclusion from RCTs create external validity gaps — observational designs fill the hole but bring confounding. Perinatal psychiatry: RCTs of drug safety are rare; observational designs dominate, so confounding and exposure misclassification dominate the viva. Intellectual disability and forensic settings: systematic under-representation; adapted measures and inclusive sampling are ethical and scientific issues. Indigenous and culturally diverse communities: community governance, mixed methods, and qualitative depth are essential — hierarchical dismissal of non-RCT evidence is both methodologically and ethically wrong when RCT questions are not the ones being asked.[15][13]

Evidence and guidelines — regional deltas

Landmark methods papers for citation fluency: Grimes and Schulz Lancet series on the "lay of the land", cohorts, case-control, and observational bias; CONSORT 2010; STROBE; PRISMA 2020; Schulz empirical bias; Hollis on ITT; Hill on causation; Tong COREQ for qualitative reporting.[1][2][3][4][5][6][7][9][10][14][15]

Exam pearls

• CONSORT = trials; STROBE = observational; PRISMA = systematic reviews; COREQ = qualitative interviews/focus groups.[5][6][7][15]
• Concealment ≠ blinding; ITT ≠ completer analysis.[9][10]
• Cohort: exposure → future outcome. Case-control: outcome → past exposure. Cross-sectional: both now.[2][3]
• Confounding is a common-cause problem; randomisation is the structural solution when feasible.[4]
• Qualitative answers different questions — complementary evidence, not failed quantification.[15]
• Absence of evidence ≠ evidence of absence.[18]
• Reporting checklists create transparency, not automatic validity.[5][11]

Bottom line

For fellowship exams, research methods means you can choose, defend, and attack a design in under a minute: name the question, pick the design that can answer it, state the fatal bias if methods slip, and name the reporting standard that makes those methods inspectable. That skill sits beside — but is not the same as — calculating NNT or reading a forest plot.[1][5][6][7][17]

References

1. [1]Grimes DA, Schulz KF An overview of clinical research: the lay of the land Lancet, 2002.PMID 11809203
2. [2]Grimes DA, Schulz KF Cohort studies: marching towards outcomes Lancet, 2002.PMID 11830217
3. [3]Grimes DA, Schulz KF Case-control studies: research in reverse Lancet, 2002.PMID 11844534
4. [4]Grimes DA, Schulz KF Bias and causal associations in observational research Lancet, 2002.PMID 11812579
5. [5]Schulz KF, Altman DG, Moher D; CONSORT Group CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials BMJ, 2010.PMID 20332509
6. [6]von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies Lancet, 2007.PMID 18064739
7. [7]Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews BMJ, 2021.PMID 33782057
8. [8]Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement BMJ, 2009.PMID 19622551
9. [9]Schulz KF, Chalmers I, Hayes RJ, Altman DG Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials JAMA, 1995.PMID 7823387
10. [10]Hollis S, Campbell F What is meant by intention to treat analysis? Survey of published randomised controlled trials BMJ, 1999.PMID 10480822
11. [11]Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials BMJ, 2019.PMID 31462531
12. [12]Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ, 2016.PMID 27733354
13. [13]Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group GRADE: an emerging consensus on rating quality of evidence and strength of recommendations BMJ, 2008.PMID 18436948
14. [14]Hill AB The environment and disease: association or causation? Proc R Soc Med, 1965.PMID 14283879
15. [15]Tong A, Sainsbury P, Craig J Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups Int J Qual Health Care, 2007.PMID 17872937
16. [16]Richardson WS, Wilson MC, Nishikawa J, Hayward RS The well-built clinical question: a key to evidence-based decisions ACP J Club, 1995.PMID 7582737
17. [17]Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS Evidence based medicine: what it is and what it isn't BMJ, 1996.PMID 8555924
18. [18]Altman DG, Bland JM Absence of evidence is not evidence of absence BMJ, 1995.PMID 7647644
19. [19]Ioannidis JPA Why most published research findings are false PLoS Med, 2005.PMID 16060722
20. [20]Higgins JP, Thompson SG, Deeks JJ, Altman DG Measuring inconsistency in meta-analyses BMJ, 2003.PMID 12958120