Report

Pilot studies Karla Hemming The University of Birmingham Pilot or feasibility study? No consensus on the definition Pilot study: Mini version of the full trial Tests protocol Feasibility study: Tests individual components Refines / develops intervention 2 Why have a pilot study? 70% of publically funded trials fail to recruit required number of participants NIHR increasingly require demonstration of proof of principal that the TRIAL will work (not the intervention) 3 What pilot studies are not about Not around estimating effect size or CI of effect size Why? Pilot studies are small Any estimate of effect will be highly uncertain Mean estimate tells us nothing! 4 Objectives of a pilot study Show proof of principal of intervention (?) Test out data collection methods Inform outcome Acceptability of questionnaires Only insofar as what is important to the patient Estimate important parameters to inform sample size calculation (not effect size!) 5 Proof of principal Need a well defined surrogate outcome That surrogate outcome would have to have a very clear link to important clinical outcome Example – weight management programme Important clinical outcome BMI Show adherence improved in a pilot? Difficulty – to show adherence improved might need as large a sample as for BMI 6 Important parameters to estimate Examples include but are not limited to: Recruitment rate (informs how long the trial will have to last for) Consent rate (informs how amenable patients are to be randomised) Retention rate (inform drop out rate) Continuous outcomes – SD Binary outcomes – underlying event rate 7 Sample size justification Sample size should be justified Depends on objective 8 Objective: estimate SD Browne 1995 Sample size of 30 is reasonable to give an estimate of the SD BUT! Don’t just use mean estimate of SD in subsequent power calculations – use upper 95 percentile value 9 Objective: estimate a process rate Show this rate is above some value Or within some range (CI) Called a precision based approach 10 Example Want to show retention rate (p) is above some value (c) Retention should not differ between arms (blinded trial) – so include both arms in calculation Precision based approach Show retention rate above c% Need an estimate of actual retention rate p% Specify significance or width of CI (precision) 11 Example cont… Show retention rate above 60% Estimate actual retention rate 80% Recruit SS of 35, number retained 28 (80%) 95% CI would be: 63% to 92% Great! Showed retention rate above 60% and can proceed to definitive trial 12 Example cont… What if retention rate only 70% That is, recruit 35 But, only 24 are retained 95% CI would be 51% to 83% Oh no… haven't shown retention rate above 60% 13 Power of this sample size Method was motivated around CI or precision based approach (common in pilot studies) But, the power of a SS of 35 to show that the retention rate is above 60% is only 50% (assuming event rate observed is 80%) 14 Precision based methods Precision methods (CI) crucially depend on actual observed rate Associated power of this approach is much lower than 80% So, if you use the precision based approach do not incorporate stringent criteria to follow to definitive trial 15 Recommendations Application viewed unfavourably if geared around effect sizes and hypothesis testing Rather, should be focused around testing TRIAL protocol (or components) Sample size needs to be justified Don’t have stringent stop go criteria for full trial 16 Other things… More than one objective – take the larger SS Internal pilots Full trial fundable? If full trial not likely to be fundable by NIHR, RfPB unlikely to fund 17