Report

Performance of a diagnostic test Steen Ethelberg 17th EPIET Introductory Course Lazareto, Menorca, Spain •Thierry Ancelle September 2011 •Marta Valenciano 1 Outline 1. Performance characteristics of a test – Sensitivity – Specificity – Choice of a threshold. 2. Performance of a test in a population – Positive predictive value of a test (PPV) – Negative predictive value of a test (NPV) – Impact of disease prevalence, sensitivity and specificity on predictive values. 2 1. Performance of a test in an experimental setting 3 Population with ill and non-ill individuals 4 Test should identify the ill only 5 In reality, tests are not perfect 6 Sensitivity & specificity Se = Identified patients All patients Se = Sp = 18 = 90% 20 Identified non-patients All non-patients Sp = 78 = 97.5% 80 7 Sensitivity of a test • Ability of a test to correctly identify affected individuals • Proportion of people testing positive among affected individuals Patients Test + － True positive (TP) False negative (FN) Sensitivity (Se) = TP / ( TP + FN ) 8 Sensitivity of a PCR for congenital toxoplasmosis Patients with toxoplasmosis Rapid test True positive False negative 54 4 58 Sensitivity = 54 / 58 = 0.931= 93.1 % 9 Specificity of a test • Ability of test to identify correctly non-affected individuals • Proportion of people testing negative among nonaffected individuals Non-affected people Test + － False positive (FP) True negative (TN) Specificity (Sp) = TN / ( TN + FP ) 10 Specificity of a PCR for congenital toxoplasmosis Individuals without toxoplasmosis Rapid test False positive True negative 11 114 125 Specificity = 114 / 125 =0.912 = 91.2 % 11 Performance of a test Yes + Disease No TP FP FN TN Test Se = TP TP + FN Sp = TN TN + FP 12 Distribution of quantitative test results among affected and non-affected people Number of people tested Ideal situation Threshold for positive result TN 0 Non affected: Affected: TP 5 10 15 Quantitative result of the test 20 13 Distribution of quantitative results among affected and non-affected people More realistic situation Non-affected: Number of people tested Threshold for positive result TN FN 0 5 Affected: TP FP 10 Quantitative result of the test 15 20 14 Effect of Decreasing the Threshold Non affected: Number of people tested Threshold for positive result Affected: FP TP TN FN 0 5 10 Quantitative result of the test 15 20 15 Effect of Decreasing the Threshold Yes + Test Se = Disease No TP FP FN TN TP TP + FN Sp = TN TN + FP 16 Effect of Increasing the Threshold Number of people tested Threshold for positive result TN Non-affected: Affected: TP FN FP 0 5 10 Quantitative result of the test 15 20 17 Effect of Increasing the Threshold Yes + Disease No TP FP FN TN Test Se = TP TP + FN Sp = TN TN + FP 18 Performance of a test and threshold • Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA) • The choice of a threshold is a compromise to best reach the objectives of the test – consequences of having false positives? – consequences of having false negatives? 19 When false diagnosis is worse than missed diagnosis • Example: Screening for congenital toxoplasmosis – One should minimise false positives – Prioritise SPECIFICITY 20 When missed diagnosis is worse than false diagnosis • Example: Testing for Helicobacter pylori infection – One should minimise the false negatives – Prioritise SENSITIVITY 21 Using several tests • One way out of the dilemma is to use several tests that complement each other • First use test with a high sensitivity • Second use test with a high specificity 22 ROC curves • Representation of relationship between sensitivity and specificity for a test • Receiver Operating Characteristics curve • Simple tool to: – Help define best cut-off value of a test – Compare performance of two tests. 23 Prevention of Blood Transfusion Malaria: Choice of an Indirect IF Threshold Sensitivity (%) 100 80 1/80 1/40 1/160 60 IIF Dilutions 1/320 40 1/640 20 0 1/20 1/10 0 20 40 60 80 100 100% - Specificity (%) 24 Comparison of Performance of ELISA and CATT Test for Screening of Human Trypanosomiasis Sensitivity (%) 100 80 ELISA CATT 60 40 20 0 0 25 50 75 100 100 - Specificity (%) 25 Comparison of Performance of ELISA and CATT Test for Screening of Human Trypanosomiasis Sensitivity (%) 100 80 ELISA CATT 60 Area under the ROC curve (AUC) 40 20 0 0 25 50 75 100 100 - Specificity (%) 26 Performance of a test • Validity – Sensitivity – Specificity • Reproducibility • Concepts may also used more broadly – Exposure status – Case definitions 27 2. Performance of a test in a population 28 Would also like to know… • As a clinician – probability that a individual with a positive test is really sick? – probability that a individual with a negative test is really healthy? • As an epidemiologist – proportion of positive tests corresponding to true patients? – proportion of negative tests corresponding to healthy subjects? 29 Predictive values PPV= Real patients Positive patients PPV= NPV= 15 = 86% 22 Real non-patients Negative patients NPV= 75 = 94% 80 30 Positive Predictive Value • Probability that an individual testing positive is truly affected – proportion of affected people among those testing positive Disease Yes No Test + TP FP PPV = TP/(TP+FP) 31 Negative Predictive Value • Probability that an individual testing negative is truly non-affected – proportion of non affected among those testing negative Disease Yes No Test FN TN NPV = TN/(TN+FN) 32 Predictive value of a positive and a negative test Disease Yes No + TP FP PPV = TP/(TP+FP) FN TN NPV = TN/(TN+FN) Test PPV = VPP = PV+ NPV = VPN = PV33 Predicted values are not constants • The predicted values depend on the sensitivity and on the specificity of the test as well as on the prevalence of the disease • Will be different in different populations. 34 Relation between predictive values and sensitivity / specificity Disease Yes No + TP FP PPV = TP/(TP+FP) FN TN NPV = TN/(TN+FN) Test 35 Step 1: Specify the prevalence (Pr) of disease Disease Yes No + Test Pr 1-Pr 36 Step 2: Use sensitivity (Se) to distribute test results among the diseased Disease Yes No + Se Pr Test (1-Se)Pr Pr 1-Pr 37 Step 3: Use specificity (Sp) to distribute test results among the non-diseased Disease Yes No + Se Pr (1-Sp)(1-Pr) (1-Se)Pr Sp(1-Pr) Pr 1-Pr Test 38 Step 4: Determine the proportion testing positive and the proportion testing negative Disease Yes No + Se Pr (1-Sp)(1-Pr) (1-Se)Pr Sp(1-Pr) Pr 1-Pr Se Pr + (1-Sp)(1-Pr) Test (1-Se)Pr+ Sp(1-Pr) 39 Step 5: Calculate PPV and NPV with appropriate expressions from Step 4 Se Pr PPV = Se Pr + (1 - Sp)(1 - Pr) Sp(1 - Pr) NPV = Sp(1 - Pr) + (1 - Se) Pr 40 Relation between predictive values and sensitivity / specificity Se Pr PPV = Se Pr + (1 - Sp)(1 - Pr) Increasing specificity increasing PPV Sp(1 - Pr) NPV = Sp(1 - Pr) + (1 - Se) Pr Increasing sensitivity increasing NPV 41 Relation between predictive values and prevalence Se Pr PPV = Se Pr + (1 - Sp)(1 - Pr) Increasing prevalence increasing PPV Sp(1 - Pr) NPV = Sp(1 - Pr) + (1 - Se) Pr Decreasing prevalence increasing NPV 42 PPV and NPV of a test according to the prevalence (80% sensitivity and specificity) Predictive value (%) 100 80 NPV 60 40 20 PPV 0 0 25 50 75 100 Prevalence (%) 43 Example: Two different populations, Se=Sp=90% Prevalence: 50% PPV = 90% Ill Test + Not ill TP FP FN TN Prevalence: 10% PPV = 50% 44 Example: Screening for human trypanosomiasis in two settings • CATT test – Sensitivity = 95% – Specificity = 75% • Endemic area – Prevalence = 20% • Low endemic area – Prevalence = 0.5% • 100,000 tests performed in each area 45 Example: Screening for human trypanosomiasis in two settings Prevalence = 20% CATT test sensitivity = 95% CATT test specificity = 75% Trypanosomiasis + CATT Yes No Total 19,000 20,000 39,000 1,000 60,000 61,000 20,000 80,000 100,000 PPV = 48.7% NPV = 98.4% 46 Example: Screening for human trypanosomiasis in two settings CATT test sensitivity = 95% CATT test specificity = 75% Prevalence = 0.5% Trypanosomiasis + CATT Yes No Total 475 24,875 25,350 25 74,625 74,650 500 99,500 100,000 PPV = 1.90% NPV = 98.97% 47 To sum up… • Sensitivity and specificity – intrinsic characteristics of a test • capacity to identify the affected • capacity to identify the non-affected – independent from the disease prevalence • Predictive values – performance of a test in real life • how to interpret a positive test • how to interpret a negative test – dependent on the disease prevalence 48 Thank you! QUESTIONS? 49