Evidence Based Medicine: Review of the basics

Evidence Based Medicine:
Review of the basics
Moving beyond sensitivity and specificity
Hierarchy of Strength of Evidence
 N of 1 randomized controlled trial
 Systematic reviews of randomized controlled trials
 Single randomized trial
 Systematic review of observational studies
addressing patient important outcomes
 Single observational study addressing patient
important outcomes
 Physiologic studies
 Unsystematic clinical observations
Where to start?
 The Clinical Question
 Focus
Question types:
 Harm
 Prognosis
 Treatment
Next Step: Research
Basic Steps in reviewing an article
 1. Are the results of the study valid?
 2. What are the results?
 3. How can I apply these results to patient care?
 1. Are the results valid?
 Were patients randomized?
 Was randomization concealed?
 Were patients analyzed in the groups to which they were randomized?
 Were patients in treatment and control groups similar?
 Were patients aware of group allocation? (?blinded?)
 Were clinicians aware of group allocation?
 Was follow up complete?
 2. What are the results?
 How large was the treatment effect?
 How precise was the estimate of treatment effect?
 3. How can I apply the results to patient care?
 Were the study patients similar to the patient in my practice?
 Were all clinically important outcomes considered?
 Are the likely treatment benefits worth the potential harm and costs?
Therapy: Analyzing Results
 How good was the treatment?
Absolute Risk Reduction
Relative Risk
Relative Risk Reduction
Absolute Risk Reduction
 Absolute difference in outcomes between the
treatment and control groups
X% have outcome in control group, Y% have outcome in
treatment group
Relative Risk
 Risk of events among patients in the treatment group
compared to/relative to the risk in the control group
Relative Risk Reduction
 (1-Y/X) x 100
Why is this the most commonly reported measure of
treatment effect?
How can this be misleading????
RRR … small treatment effect
 Pink Potion is new on the market, and is a wonder
drug according to Big Bucks Pharmaceutical
Company. When taken every day for 20 years, it
decreases the risk of developing cell phone related
brain cancer by 50% ! (RRR is 50%)
 You find the risk of such cancers to be 0.25 % in one
high risk population. With this 20 year treatment,
0.13% of the population developed the cancer…
Risk Reduction: Take Home
 Beware RRR
 Calculate ARR whenever possible – this often will
put the overall treatment effect into perspective
 Most reports (especially when
advertising/promoting a treatment) will report
effects as RRR…
How precise was treatment effect?
 Confidence Intervals: 95%
95% CI defines a range of results that includes the true treatment
effect result 95% of the time
 P <.05
 Larger sample size – narrower the CI
 Using CI to estimate treatment importance
“positive study”– look at lower limit for CI, if that number is the true
treatment effect, is that important/beneficial when applied to your
“negative study” –look at upper limit for CI– if that number is true,
would that be clinically important? Study may not prove that a
treatment is of benefit, but at same time may not prove that it is not
of benefit…
Treatment: Applying results to patient care
 Were the study patients similar to patients in my
 Were all clinically relevant outcomes considered?
 Are the likely treatment benefits worth the potential
harm and costs?
Number Needed to Treat (NNT)
 Number of patients who must receive a
treatment/intervention to prevent one bad outcome
or produce one positive outcome
 1/ ARR
 (Pink Potion: NNT = 1/.12% = 1/.0012= 833== need
to treat 833 people for 20 years to prevent one case
of cell phone related brain cancer…)
Number Needed to Harm (NNH)
 Absolute risk of adverse outcome with treatment –
risk of adverse outcome without treatment
 NNH = 1/ absolute difference in adverse outcomes
(just like NNT)
 Weight NNT with NNH….
Reviewing studies looking at Harm
 Are the results valid?
 Was there demonstrated similarity in all known determinants of
outcome? Did they adjust for differences in their analysis?
 Were exposed patients equally likely to be identified in the two groups?
 Were outcomes measured the same way each group?
 Was follow up sufficiently complete?
 What are the results?
 How strong is the association between exposure and outcome?
 How precise is the estimate of risk?
 How can I apply the results to patient care?
 Were the study patients similar to my patients?
 Was the duration of follow up adequate?
 Was was the magnitude of the risk?
 Should I try to stop the exposure?
event status
Feasible when
not able to
Bias, limited
event status
delays, may
only need
small sample
Bias, limited
Adverse event
to bias
Assessing Harm in Case Control Studies
 Cannot use RR (RR depends upon determining the
proportion of patients with outcome – in a case
control study, the proportion of individuals with an
outcome is chosen by the investigator)
 Odds Ratio:
 Odds of a case patient being exposed / odds of a control
patient being exposed
OR and RR
Outcome: yes
Outcome: no
Exposure: yes
Exposure: no
RR= a/(a+b) / c/(c+d)
OR = (a/b) / (c/d)
If outcome is infrequent in both treatment and control groups, then OR and RR
Be nearly the same
Studies looking at Harm
 Can I apply the results to my patient?
Were the study patients similar?
Was the duration long enough?
How big is the risk?
Should I stop the exposure? (does risk outweigh any benefit?
Is there alternative therapy with less risk?)
Diagnostic Testing
 Probability of diagnosis: 0-100%
 Test threshold
 Treatment threshold
 Probability below test threshold: no testing
 Probability in between : test
 Probability above treatment threshold: no testing,
Looking at Diagnostic Tests
 Are the results valid?
 Did clinicians face diagnostic uncertainty
 Was there a blind comparison with an independent Gold Standard
applied to both Treatment and Control groups
 Did the results of the test being studied influence the decision to perform
the reference/gold standard?
 What are the results?
 What are the Likelihood Ratios associated with the range of possible test
 How can I apply the results to patient care?
 Will the reproducibility of the test result and its interpretation be doable
in my clinical setting?
 Are the results applicable to my patients?
 Will the results change my management strategy?
 Will patients be better off as a result of the test?
Sensitivity and Specificity
Disease +
Disease -
Test +
Test -
Sensitivity: If the patient has the disease, how likely is it that he will
have a + test?
“rules out” disease (not really….)
If the patient does not have the disease, how likely is he to have
a negative test?
“rules in” disease (again, not really….)
The problem with sensitivity and specificity
 In real life, the question is “the patient has a positive
test, how likely is it that he has the disease? Or the
patient has a negative test, how likely is it that he
does not have the disease?”
Predictive Values
Disease +
Disease -
Test +
Test -
Positive Predictive Value:
How many (%) people with a positive test will have the disease?
PPV = a/ a+b
Negative Predictive Value:
How many people with a negative test will NOT have the disease?
NPV = d/c+d
Problem: depend upon prevalence (low prevalence population, positive test is more
Likely to be a false positive; high prevalence, a negative more likely to be a false
Likelihood Ratios
 LR does not depend upon prevalence and can be more
easily applied to a specific patient with a known test
result to estimate the post-test probability that the
patient has “disease”
 Points:
Not affected by prevalence
Can be made specific to your patient (based upon pretest probability
of disease)
Can be linked with other LRs to come up with a post test probability
Does not rely upon test being dichotomous (positive or negative)
Likelihood Ratio
 The likelihood that disease is present given X test
result (positive, negative, intermediate, 250)
 LR: How many patients with X test result HAVE
disease compared to number of patients with X test
result who do NOT have disease
Points about the LR
 X test result can be positive, negative, intermediate,
a number
 LR always looks at ONE certain test result and
always compares likelihood of having disease to
likelihood of not having disease
LR for a positive test (Positive LR)
 Likelihood that disease is present given a “positive” test
 How many patients with a + test HAVE disease
compared to # patients with a + test who do NOT HAVE
True Positive Rate/False Positive Rate
LR + test: a /(a+c) / b/(b+d)
= sensitivity/1-specificity
Higher # (over 10) : better predicting
LR + “infinity” = 100% specificity (if the test is positive,
the patient has disease, no false positives)
LR for Negative Test (-LR)
 Likelihood that disease is present given a negative test
 How many patients with a Negative test HAVE disease compared to
# patients with a Negative Test who DO NOT HAVE disease
 False negative rate/true negative rate
 c/(a+c) / d/(b+d)
 1-sensitivity/ specificity
 Smaller number = better test (fewer patients with a negative test
HAVE disease compared to DO NOT HAVE disease)
 LR – of <.10 usually signficant
 LR – of 0 = 100% sensitivity (if the test is negative, the patient does
NOT have disease – rules “out” disease…
Review the chart again…
Disease +
Disease -
Test +
Test -
Sensitivity: a/a+c
Specificity: d/b+d
LR + test: a/(a+c) / b/(b+d)
LR – test: c/(a+c) / d/(b+d)
How to use the LR
 LR is ODDS note a %
 Determine pretest probability (ok to estimate)
 Determine pre test ODDS (odds= probability/1-
 Determine Post test ODDS: pretest odds x LR
 Convert post test ODDS back to probability
LR Nomogram
 P. 129
Looking at Summaries of Evidence
 Systematic Reviews
 Meta-analyses
Summary articles
 Are the results valid?
 Did the review explicitly address a sensible clinical question?
 Was the search for relevant studies detailed and exhaustive (publication
 Were the primary studies high quality?
 Were assessments of studies reproducible?
 What are the results?
 Were the results similar from study to study?
 Were the outcomes the same (comparing apples to apples…)
 What are the overall results?
 How precise were the results?
 How can I apply the results to patient care?
 How can I interpret the results?
 Were all clinically important outcomes considered?
 Are the benefits worth the costs and potential risks?
Developing a CAT sheet:
 Critically Appraised Topic
 Clinical Question
 Clinical Bottom Lines
 Methods
 Summary of Results (create table if needed)
 Comments (strengths, weaknesses, limitations)
 How can I apply this to my patients?
 References

similar documents