Report

Consistent Assessment of Biomarker and Subgroup Identification Methods H.D. Hollins Showalter 5/20/2014 (MBSW) 1 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 2 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 3 Tailored Therapeutics A medication for which treatment decisions are based on the molecular profile of the patient, the disease, and/or the patient’s response to treatment. • A tailored therapeutic allows the sponsor to make a regulatory approved claim of an expected treatment effect (efficacy or safety) • “Tailored therapeutics can significantly increase value—first, for patients—who achieve better outcomes with less risk and, second, for payers— who more frequently get the results they expect.”* *Opening Remarks at 2009 Investor Meeting, John C. Lechleiter, Ph.D. 5/20/2014 (MBSW) Adapted from slides presented by William L. Macias, MD, PhD, Eli Lilly 4 Achieving Tailored Therapeutics • • • • Data source: clinical trials (mostly) Objective: identify biomarkers and subgroups Challenges: complexity, multiplicity Need: modern statistical methods 5/20/2014 (MBSW) 5 Prognostic vs. Predictive Markers Prognostic Marker Single trait or signature of traits that identifies different groups of patients with respect to the risk of an outcome of interest in the absence of treatment Predictive Marker Single trait or signature of traits that identifies different groups of patients with respect to the outcome of interest in response to a particular treatment 5/20/2014 (MBSW) 6 Statistical Interactions Treatment Response Treatment by Marker Effect Treatment Treatment Effect No treatment Marker Effect - Marker + Y = 0 + 1*M + 2*T + 3*M*T + 5/20/2014 (MBSW) 7 Types of Predictive Markers Treatment Response Response Treatment No treatment - No treatment - + Marker + Marker Treatment No treatment 5/20/2014 (MBSW) + Marker Response Response Treatment No treatment - + Marker 8 Predictive Marker Example Subgroup of Interest M− Group size: 25% Trt response: -1.39 Pl response: -0.19 Group size: 50% Trt response: -1.17 Pl response: -0.09 M− M+ x2 = 1 M+ Subgroup of Interest Trt response: -0.33 Pl response: -0.20 Treatment effect: -1.20 Group size: 75% Treatment effect: -1.08 Treatment effect: -0.13 x2 = 0 Trt response: -0.23 Pl response: -0.13 Treatment effect: -0.1 x1 = 1 x1 = 0 Entire Population 5/20/2014 (MBSW) x1 = 1 x1 = 0 Entire Population 9 BSID vs. “Traditional” Analysis • Traditional subgroup analysis o o o Interaction testing, one at a time Many statistical issues Many gaps for tailoring • Biomarker and subgroup identification (BSID) o o o Utilizes modern statistical methods Addresses issues with subgroup analysis Maximizes tailoring opportunities 5/20/2014 (MBSW) 10 Simulation to Assess BSID Methods Objective Consistent, rigorous, and comprehensive calibration and comparison of BSID methods Value • Further improve methodology o o Identify the gaps (where existing methods perform poorly) Synergy/combining ideas from multiple methods • Optimize application for specific clinical trials 5/20/2014 (MBSW) 11 BSID Simulation: Three Components 1. Data generation o Key is consistency 2. BSID o “Open” and comprehensive application of analysis method(s) 3. Performance measurement o Key is consistency 5/20/2014 (MBSW) 12 Data Generation BSID Simulation: Visual Representation Performance Measurement BSID Truth Dataset 1 Dataset 2 Dataset … Dataset n Results 1 Results 2 Results … Results n Performance Metrics 1 Performance Metrics 2 Performance Metrics … Performance Metrics n 5/20/2014 (MBSW) Overall Performance Metrics 13 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 14 Data Generation • Creating virtual trial data o o o Make assumptions in order to emulate real trial data Knowledge of disease and therapies, including historical data Specific to BSID: must embed markers and subgroups • In order to measure the performance of BSID methodology the “truth” is needed o This is challenging/impossible to discern using real trial data 5/20/2014 (MBSW) 15 Data Generation Survey SIDES (2011)1 SIDES (2014)2 VT3 GUIDE4 QUINT5 IT6 n 900 300, 900 400 - 2000 100 200 - 1000 300, 450 p 5 - 20 20 - 100 15 - 30 100 5 - 20 4 continuous continuous binary binary continuous TTE binary binary continuous categorical continuous ordinal, categorical predictor correlation 0, 0.3 0, 0.2 0, 0.7 0 0, 0.2 0 treatment assignment 1:1 1:1 ? ~1:1 ~1:1 ? # predictive markers 0-3 2 0, 2 0, 2 1-3 0, 2 higher order higher order higher order N/A, simple, higher order simple, higher order simple 15% - 20% 50% N/A, ~25%, ~50% N/A, ~36% ~16% - ~50% N/A, ~25%, ? 0 0 3 0-4 1-3 0, 2 N/A N/A simple, higher order N/A, simple, higher order simple, higher order simple logit model (w/o and with subjectspecific effects linear model (on probability scale) “tree model” exponential model Attribute response type predictor type predictive effect(s) predictive M+ group size (% of n) # prognostic markers prognostic effect(s) “contribution model” 5/20/2014 (MBSW) 16 Data Generation: Recommendations • Clearly identify attributes and models o o Transparency Traceability of analysis • Make sure to capture the “truth” in a way that facilitates performance measurement • Derive efficiency and synergistic value (more on this later!) 5/20/2014 (MBSW) 17 Data Generation: Specifics • Identify key attributes o o o o o o o Sample size Number of predictors Response type Predictor type/correlation Subgroup size Sizes of effects: placebo response, overall treatment effect, predictive effect(s), prognostic effect(s) Others: Missing data, treatment assignment • Specify model 5/20/2014 (MBSW) 18 Data Generation: Recommendations • Clearly identify attributes and models o o Transparency Traceability of analysis • Make sure to capture the “truth” in a way that facilitates performance measurement • Derive efficiency and synergistic value (more on this later!) 5/20/2014 (MBSW) 19 Data Generation: Reqs • Format data consistently • Make code flexible enough to accommodate any/all attributes and models • Ensure that individual datasets can be reproduced (i.e., various seeds for random number generation) The resulting dataset(s) should always have the same look and feel 5/20/2014 (MBSW) 20 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 21 Performance Measurement • Quantifying the ability of BSID methodology to recapture the “truth” underlying the (generated) data • If done consistently, allows calibration and comparison of BSID methods 5/20/2014 (MBSW) 22 Performance Measurement: Survey SIDES (2011)1 SIDES (2014)2 Selection rate Complete match rate Partial match rate Confirmation rate Treatment effect fraction 5/20/2014 (MBSW) VT3 GUIDE4 QUINT5 IT6 Pr(complete match) Finding correct X’s (RP1a) Pr(type I errors) Pr(partial match) Closeness of to the true A Pr(selection at 1st or 2nd level splits of trees) Frequencies of the final tree sizes Pr(selecting a subset) Closeness of the size of to the size of the true A Accuracy (RP2) Rec. of tree complexity Pr(selecting a superset) Treatment effect fraction (updated def.) Power Properties of as an estimator of Pr(nontrivial tree) (RP1b) Pr(type II errors) (RP3) Rec. of splitting vars and split points. Frequency of (predictor) “hits” Bias assessment via likelihood ratio and logrank tests (RP4) Rec. of assignments of observations to partition classes 23 Performance Measurement: Recommendations Marker Level testing 5/20/2014 (MBSW) Subgroup Level Subject Level estimation prediction 24 Perf. Measurement: Survey Revisited 2 Level SIDES (2011)1 Marker SIDES (2014) Selection rate Complete match rate Partial match rate VT3 Pr(selecting Pr(completea (RP1b) Finding Pr(type superset) match) correct X’s II errors) Pr(partial Finding match)X’s correct Pr(selecting Power a subset) Pr(selection Pr(selecting at 1st or 2nda superset) level splits of trees) Treatment effect fraction Pr(nontrivial (updated tree)def.) Closeness (RP2) Rec.of tree toofthe true A complexityof Closeness (RP3) Rec. the size of of splitting vars to the size of andtrue splitA the Treatment points. Pr(complete Power effect fraction match) Frequencies Properties of Pr(partial final of theas an match) tree sizesof estimator Pr(selecting a (RP1a) Pr(type Frequency of subset) I errors) (predictor) “hits” Confirmation rate (testing) 5/20/2014 (MBSW) 4 Subgroup Level QUINT5 GUIDE 6 Subj.ITLevel Properties of (RP1a) Pr(type Frequencies Treatment Pr(selection Closeness of effect at 1stfraction or 2nd I errors) as an of to the the final true A tree sizes level splits of (RP1b) estimator of Pr(type Accuracy Treatment trees) II errors) Frequency of effect fraction (RP4) Rec. of (predictor) Accuracy (updated def.) (RP2) Rec. Bias assignments “hits” of of tree Pr(nontrivial assessment observations to Closeness of complexity tree) Bias via likelihood partition the size of assessment ratioRec. and of classes to the size of (RP3) via likelihood logrank tests splitting vars the true A ratio and and split (prediction) (estimation) logrank tests points. (RP4) Rec. of SIDES (2011)of1 assignments observations to2 SIDES (2014) partition classes VT3 GUIDE4 QUINT5 IT6 25 Contingency Table: Marker Level • • • • True False Yes True Positive False Positive No Identified as Predictive Predictive Biomarker False Negative True Negative Sensitivity = True Positive / True Predictive Biomarkers Specificity = True Negative / False Predictive Biomarkers PPV = True Positive / Identified as Predictive NPV = True Negative / Not Identified as Predictive 5/20/2014 (MBSW) 26 Performance Measures: Marker Level # and % of predictors: true vs. identified • Sensitivity • Specificity • PPV • NPV 5/20/2014 (MBSW) 27 Performance Measures: Subgroup Level • Size of identified subgroup • Treatment effect in the identified subgroup o Average the true “individual” treatment effects under potential outcomes framework • Accuracy of estimated treatment effect o Difference (both absolute and direction) between estimate and true effect 5/20/2014 (MBSW) 28 Perf. Measures: Subgroup Level, cont. • Implications on sample size/time/cost of future trials o o o Given true treatment effect, what is the number of subjects needed in the trial for 90% power? What is the cost of the trial? (mainly driven by # enrolled) How much time will the trial take? (mainly driven by # screened) 5/20/2014 (MBSW) 29 Contingency Table: Subject Level • • • • True False M+ True Positive False Positive M- Membership Classification Potential to Realize Enhanced Treatment Effect* False Negative True Negative *at a meaningful or desired level Sensitivity = True Positive / True Enhanced Treatment Effect Specificity = True Negative / False Enhanced Treatment Effect PPV = True Positive / Classified as M+ NPV = True Negative / Classified as M- 5/20/2014 (MBSW) 30 Performance Measures: Subject Level Compare subgroup membership on the individual level: true vs. identified • Sensitivity • Specificity • PPV • NPV 5/20/2014 (MBSW) 31 Conditional Performance Measures • Same metrics with Null submissions removed Markers/subgroups can be very difficult to find. When a method DOES find something, how accurate is it? Hard(er) to compare multiple methods when all performance measures are washed out by Null submissions 5/20/2014 (MBSW) 32 Cond. Subgroup Level Measures Example Group size: 50% Unconditional Treatment effect: 10 Group size: 50% Size: 0.95 Effect: 5.5 Conditional Size: 0.5 Effect: 10 M− x2 = 1 M+ BSID Method B 900/1000: Null 50/1000: x1 = 1 50/1000: x2 = 1 Unconditional Group size: 50% Treatment effect: 0 x1 = 1 x2 = 0 BSID Method A 900/1000: Null 100/1000: x1 = 1 Group size: 50% 1000 simulations x1 = 0 Size: 0.95 Effect: 5.25 Conditional Size: 0.5 Effect: 7.5 Truth (but x1 very hard to find) 5/20/2014 (MBSW) 33 Performance Measurement: Reqs For each application of BSID user proposes: • List of predictive biomarkers • The one subgroup for designing the next study • Estimated treatment effect in this subgroup In conjunction with the “truth” underlying the generated data, all of the recommended performance measures can be calculated using these elements 5/20/2014 (MBSW) 34 Considering the “Three Levels” What are the most important and relevant measures of a result? Depends on the objective… Marker Level Invest further in the marker(s) 5/20/2014 (MBSW) Subgroup Level Tailor the next study/design Subject Level Impact in clinical practice 35 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 36 Data Generation Example Attribute Value simulations (datasets) 200 n 240 p 20 response type continuous ( 0, 1.132 errors) predictor type ordinal (“genetic”) predictor correlation 0 treatment assignment 1:3 (pl:trt) placebo response -0.1 (in weakest responding subgroup) treatment effect -0.1 (in weakest responding subgroup) # predictive markers predictive effect size(s) (type) predictive M+ group size # prognostic markers prognostic effect size(s) 1 -0.45 (dominant) ~50% of n 0 N/A linear model 5/20/2014 (MBSW) 37 Data Generation Example, cont. 5/20/2014 (MBSW) 38 Data Generation Example, concl. 0 -0.05 -0.1 -0.15 -0.2 -0.25 -0.3 -0.35 -0.4 -0.45 -0.5 x_1_1_1 trt 0 trt 1 - Dataset 1 Trt 0: -0.141 Trt 1: -0.407 Effect: -0.266 5/20/2014 (MBSW) Dataset 21 Trt 0: -0.018 Trt 1: -0.427 Effect: -0.409 + 0 -0.05 -0.1 -0.15 -0.2 -0.25 -0.3 -0.35 -0.4 -0.45 -0.5 x_1_1_1 trt 0 trt 1 - + 39 BSID Methods Applied to Example Approach Traditional Virtual Twin3 TSDT7 Handling treatment-bysubgroup interaction Model Transformation Sequential Searching for candidate subgroups Exhaustive Recursive Partitioning Recursive Partitioning Addressing multiplicity Simple (Sidak Correction) Permutation Sub-sampling + Permutation Alpha controlled at 0.1 5/20/2014 (MBSW) 40 Performance Measurement Example Truth = 5/20/2014 (MBSW) + Proposal Performance Measures 41 Perf. Measurement Example, cont. 5/20/2014 (MBSW) 42 Perf. Measurement Example, concl. Measure Virtual Twin3 Traditional TSDT7 Marker Level Uncond. Cond. Uncond. Cond. Uncond. Cond. Sensitivity 0.025 0.227 0.135 0.614 0.39 0.929 Specificity 0.995 0.957 0.996 0.980 0.998 0.996 PPV 0.227 0.227 0.614 0.614 0.929 0.929 NPV 0.951 0.959 0.957 0.980 0.969 0.996 Subgroup Level Non-Identification (Null) 89% 78% 58% Subgroup Size 93.6% 41.4% 88.8% 48.9% 79.2% 50.4% Trt Effect in Subgroup -0.335 -0.388 -0.359 -0.466 -0.416 -0.535 Sensitivity 0.947 0.518 0.956 0.798 0.986 0.966 Specificity 0.076 0.689 0.180 0.820 0.406 0.966 PPV 0.523 0.639 0.576 0.814 0.702 0.967 NPV 0.592 0.592 0.805 0.805 0.965 0.965 Subject Level 5/20/2014 (MBSW) 43 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 44 Strategy • Develop framework (done/ongoing) • Present/get input (current) o o Internal and external forums Workshop • Establish an open environment (future) o o R package on CRAN Web portal repository 5/20/2014 (MBSW) 45 Predictive Biomarker Project: Vision • Access Web Portal o Reads open description (objective, models, formats etc.) • Access web interface for Data Generation o Generate data under specified scenarios, or utilize “standard”/pre-existing scenarios • Apply BSID methodology to datasets o Express results in the specified format • Access web interface for Performance Measurement o Compare performance • Encouraged to contribute to Repository o Open sharing of results, descriptions, programs 5/20/2014 (MBSW) 46 Pros and Cons Pros • More convenient and useful simulation studies to aid research • Direct comparisons of performance by methods • Optimization of methods for relevant and important scenarios for drug development • New insights and collaborations • Data sets could be applied for other statistical problems Cons • Need to develop infrastructure to support simulated data • Access and upkeep • Need experts to explicitly define the scope 5/20/2014 (MBSW) 47 Outline 1. 2. 3. 4. 5. 6. Background Data Generation Performance Measurement Example Operationalization Conclusion 5/20/2014 (MBSW) 48 Conclusion • Simulation studies are a common approach to assessing BSID methods but there is a lack of consistency in data generation and performance measurement • The presented framework enables consistent, rigorous, comprehensive calibration and comparison of BSID methods • Collaborating on this effort will result in efficiency and synergistic value 5/20/2014 (MBSW) 49 Acknowledgements • • • • • • Richard Zink Lei Shen Chakib Battioui Steve Ruberg Ying Ding Michael Bell 5/20/2014 (MBSW) 50 References 1. 2. 3. 4. 5. 6. 7. 8. 9. Lipkovich I, Dmitrienko A, Denne J, Enas, G. Subgroup identification based on differential effect search — a recursive partitioning method for establishing response to treatment in patient subpopulations. Statistics in Medicine 2011; 30:2601–2621. doi:10.1002/sim.4289. Lipkovich I, Dmitrienko A. Strategies for Identifying Predictive Biomarkers and Subgroups with Enhanced Treatment Effect in Clinical Trials Using SIDES. Journal of Biopharmaceutical Statistics 2014; 24:130-153. doi:10.1080/10543406.2013.856024. Foster JC, Taylor JMG, Ruberg SJ. Subgroup identification from randomized clinical trial data. Statistics in Medicine 2011; 30:2867–2880. doi:10.1002/sim.4322. Loh, WY, He X, Man M. A regression tree approach to identifying subgroups with differential treatment effects. Presented at Midwest Biopharmaceutical Statistics Workshop 2014. Dusseldorp E, Van Mechelen I. Qualitative interaction trees: a tool to identify qualitative treatmentsubgroup interactions. Statistics in Medicine 2014; 33:219–237. doi:10.1002/sim.5933. Su X, Zhou T, Yan X, Fan J, Yang S. Interaction trees with censored survival data. International Journal of Biostatistics 2008; 4(1):Article 2. doi:10.2202/1557-4679.1071. Battioui C, Shen L, Ruberg S. A Resampling-based Ensemble Tree Method to Identify Patient Subgroups with Enhanced Treatment Effect. Proceedings of the 2013 Joint Statistical Meetings. Zink R, Shen L, Wolfinger R, Showalter H. Assessment of Methods to Identify Patient Subgroups with Enhanced Treatment Response in Randomized Clinical Trials. Presented at the 2013 ICSA Applied Statistical Symposium. Shen L, Ding Y, Battioui C. A Framework of Statistical Methods for Identification of Subgroups with Differential Treatment Effects in Randomized Trials. Presented at the 2013 ICSA Applied Statistical Symposium. 5/20/2014 (MBSW) 51 Backup Slides 5/20/2014 (MBSW) 52 Data Generation: SIDES (2011)1 Attribute Value simulations (datasets) 5000 n 900 (then divided into 3 equal – 1 training, 2 test) p 5, 10, 20 response type continuous ( 0, 2 errors) predictor type binary (dichotomized from continuous) predictor correlation 0, 0.3 treatment assignment 1:1 (pl:trt) placebo response 0 treatment effect 0 # predictive markers 0, 1, 2, 3* predictive effect size(s) not explicitly stated predictive M+ group size 15% - 20% of n (but not explicitly stated) # prognostic markers 0 prognostic effect size(s) N/A model “contribution model” 5/20/2014 (MBSW) 53 Data Generation: SIDES (2014)2 Attribute Scenario 1 Scenario 2 Scenario 3 Scenario 4 simulations (datasets) 10000 10000 10000 10000 n 300 300 900 900 p 20, 60, 100 20, 60, 100 20, 60, 100 20, 60, 100 response type continuous ( 0, 2 errors) continuous ( 0, 2 errors) continuous ( 0, 2 errors) continuous ( 0, 2 errors) predictor type binary (dichotomized from continuous) binary (dichotomized from continuous) binary (dichotomized from continuous) binary (dichotomized from continuous) predictor correlation 0 0.2* 0 0.2* treatment assignment 1:1 (pl:trt) 1:1 (pl:trt) 1:1 (pl:trt) 1:1 (pl:trt) placebo response 0 0 0 0 treatment effect 0 0 0 0 # predictive markers 2** 2** 2** 2** predictive effect size(s) 0.35 0.35 0.6 0.6 predictive M+ group size 0.5 * n = 150 0.5 * n = 150 0.5 * n = 450 0.5 * n = 450 # prognostic markers 0 0 0 0 prognostic effect size(s) N/A N/A N/A N/A model “contribution model” “contribution model” “contribution model” “contribution model” 5/20/2014 (MBSW) 54 Data Generation: Virtual Twins3 Attribute Null Base simulations (datasets) 100 100 n 1000 1000 400 and 2000 p 15 15 30 response type binary binary predictor type continuous ( 0, 2 errors) continuous ( 0, 2 errors) predictor correlation 0 0 treatment assignment ? ? placebo response -1 -1 treatment effect 0.1 0.1 # predictive markers 0 2 predictive effect size(s) 0 0.9 for X1*X2 1.5 for X1*X2 predictive M+ group size N/A ~0.25 * n = ~250 ~0.5 * n = ~500 # prognostic markers 3 3 prognostic effect size(s) 0.5, 0.5, -0.5 for X1, X2, X7 0.5 for X2*X7 0.5, 0.5, -0.5 for X1, X2, X7 0.5 for X2*X7 model logit model logit model 5/20/2014 (MBSW) Modifications* 0.7** logit model with subjectspecific effects ai and (ai, bi) 55 Data Generation: GUIDE4 Attribute M1 M2 M3 simulations (datasets) 1000 1000 1000 n 100 100 100 p 100 100 100 response type binary binary binary predictor type categorical (3 levels)* categorical (3 levels)* categorical (3 levels)* predictor correlation 0 0 0 treatment assignment ~1:1 (pl:trt) ~1:1 (pl:trt) ~1:1 (pl:trt) placebo response 0.4 0.3 0.2 treatment effect 0 0 0.2 # predictive markers 2 2 0 predictive effect size(s) 0.2, 0.15 for X1, X2 0.05 for X1*X2 0.4 for X1*X2 N/A predictive M+ group size ~0.36 * n = ~360 in strongest M+ group (but not explicitly stated) ~0.36 * n = ~360 (but not explicitly stated) N/A # prognostic markers 0 4 2 prognostic effect size(s) N/A 0.2 for X3, X4 -0.2 for X1*X2 0.2 for X1, X2 model linear model (on probability scale) linear model (on probability scale) linear model (on probability scale) 5/20/2014 (MBSW) 56 Data Generation: QUINT5 Attribute Model A Model B*** Model C*** Model D*** Model E simulations (datasets) 100 100 100 100 100 n 200, 300, 400, 500, 1000 200, 300, 400, 500, 1000 200, 300, 400, 500, 1000 200, 300, 400, 500, 1000 200, 300, 400, 500, 1000 p 5, 10, 20 5, 10, 20 5, 10, 20 5, 10, 20 5, 10, 20 response type continuous* continuous* continuous* continuous* continuous* predictor type continuous (multivariate normal)** continuous (multivariate normal)** continuous (multivariate normal)** continuous (multivariate normal)** continuous (multivariate normal)** predictor correlation 0, 0.2 0, 0.2 0, 0.2 0, 0.2 0, 0.2 treatment assignment ~1:1 (trt 1:trt 2) ~1:1 (trt 1:trt 2) ~1:1 (trt 1:trt 2) ~1:1 (trt 1:trt 2) ~1:1 (trt 1:trt 2) treatment 1 response 20*** 20*** 20*** 18.33*** 30*** treatment 2 effect -2.5, -5, -10*** -2.5, -5, -10*** -2.5, -5, -10*** -2.5, -5, -10*** 0*** # predictive markers 1 2 3 3 1 predictive effect size(s) 5, 10, 20*** 5, 10, 20*** 5, 10, 20*** 5, 10, 20*** 2.5, 5, 10*** predictive M+ group size ~0.16 * n (but not explicitly stated)*** ~0.16 * n (but not explicitly stated)*** ~0.38 * n (but not explicitly stated)*** ~0.16 * n (but not explicitly stated)*** ~0.5 * n (but not explicitly stated)*** # prognostic markers 1*** 2*** 3*** 3*** 1*** prognostic effect size(s) 20*** 20*** 20*** 21.67*** 10*** model “tree model” “tree model” “tree model” “tree model” “tree model” 5/20/2014 (MBSW) 57 Data Generation: Interaction Trees6 Attribute Model A Model B Model C Model D simulations (datasets) 100 100 100 100 n 450 test sample method (300 for learning sample, 150 for validation sample), 300 bootstrap method 450 test sample method (300 for learning sample, 150 for validation sample), 300 bootstrap method 450 test sample method (300 for learning sample, 150 for validation sample), 300 bootstrap method 450 test sample method (300 for learning sample, 150 for validation sample), 300 bootstrap method p 4 4 4 4 response type TTE (censoring rates = 0%, 50%) TTE (censoring rates = 0%, 50%) TTE (censoring rates = 0%, 50%) TTE (censoring rates = 0%, 50%) predictor type ordinal for X1 and X3, categorical for X2 and X4 ordinal for X1 and X3, categorical for X2 and X4 ordinal for X1 and X3, categorical for X2 and X4 ordinal for X1 and X3, categorical for X2 and X4 predictor correlation 0 0 0 0 treatment assignment ? ? ? ? placebo response 0.135 0.135 0.135 0.135 treatment effect 2* 2* 2* 2* # predictive markers 0 2 2 2 predictive effect size(s) N/A 0.223 for X1* 4.482 for X2* 0.741 to 0.050 for X1* ** 1.350 to 20.086 for X2* ** 0.5 for X1* 2 for X2* predictive M+ group size N/A ~0.25 * n in strongest M+ group (but not explicitly stated) not explicitly stated** ~0.25 * n in strongest M+ group (but not explicitly stated) # prognostic markers 2 0 0 0 prognostic effect size(s) 0.223 for X1* 4.482 for X2* N/A N/A N/A model exponential model exponential model exponential model exponential model 5/20/2014 (MBSW) 58 Perf. Measurement: SIDES (2011)1 • Selection rate, that is, the proportion of simulation runs in which >1 subgroup was identified. o o • • Complete match rate: Proportion of simulation runs in which the ideal subgroup was selected as the top subgroup (computed over the runs when at least one subgroup was selected). Partial match rate: Proportion of simulation runs in which the top subgroup was a subset of the ideal subgroup (computed over the runs when at least one subgroup was selected). Confirmation rate, that is, the proportion of simulation runs that yielded a confirmed subgroup (which is not necessarily identical to the ideal subgroup). In each run, the top subgroup was identified in terms of the treatment effect p-value in the training data set (if at least one subgroup was selected). The subgroup was classified as ‘confirmed’ if the treatment effect in this subgroup was significant at a two-sided 0.05 level in both test data sets. Treatment effect fraction defined as the fraction of the treatment effect (per patient) in the ideal group, which was retained in the top selected or confirmed subgroup. The fraction was defined as follows: 5/20/2014 (MBSW) 59 Perf. Measurement: SIDES (2014)2 • Probability of a complete match • Probability of a partial match o o Probability of selecting a subset Probability of selecting a superset • Treatment effect fraction (updated definition, not weighted by group sizes): 5/20/2014 (MBSW) 60 Perf. Measurement: Virtual Twins3 • Finding correct X’s • Closeness of to the true A. This is measured using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the ROC curve (AUC). • Closeness of the size of to the size of the true A • Power. Another quantity of interest is the percentage of times methods find a null when ≠ 0 and when = 0. • Properties of as an estimator of 5/20/2014 (MBSW) 61 Performance Measurement: GUIDE4 • Probabilities that (predictive markers) are selected at first and second level splits of trees • Accuracy. Let n(t, y, z) denote the number of training samples in node t with Y = y and Z = z and define n(t,+, z) = n(t, y, z) and nt = n(t, +, z). Let be the subgroup defined by t. The value of is estimated by = |n(t, 1, 1)/n(t,+, 1) − n(t, 1, 0)/n(t,+, 0)|. The estimate of ∗ is the subgroup such that is maximum among all terminal nodes. If is not unique, is taken as their union. The “accuracy” of is defined to be / ∗ if ⊂ ∗ and 0 otherwise. • Pr(nontrivial tree) 5/20/2014 (MBSW) 62 Performance Measurement: QUINT5 • (RP1a) Probability of type I errors • (RP1b) Probability of type II errors • (RP2) Recovery of tree complexity. Given an underlying true tree with a qualitative treatment– subgroup interaction that has been correctly detected, the probability of successfully identifying the complexity of the true tree. • (RP3) Recovery of splitting variables and split points. Given an underlying true tree with a qualitative treatment–subgroup interaction that has been correctly detected, probability of recovering the true tree in terms of the true splitting variables and the true split points • (RP4) Recovery of the assignments of the observations to the partition classes 5/20/2014 (MBSW) 63 Perf. Measurement: Interaction Trees6 • Frequencies of the final tree sizes • Frequency of (predictor) “hits” • Bias assessment: the following were calculated for the pooled training and test samples and the validation samples o o the likelihood ratio test (LRT) for overall interaction the logrank test for treatment effect within the terminal node that showed maximal treatment efficacy (for presentation convenience, the logworth of the pvalue, which is defined as -log10 (p-value), was used). 5/20/2014 (MBSW) 64 Predictive Biomarker Project Data Generation • Web interface • Standard datasets 5/20/2014 (MBSW) BSID • Open methods • Standard output Performance Measurement • Web interface • Standard summary 65