Report

P-Values Stephen Senn Competence Centre for Methodology and Statistics (c)Stephen Senn 2012 1 Outline • Introduction to P_values – History – How they are defined and calculated – What they are – What they are not • P-values as evidence? – Distribution under null – Distribution under alternative – Relevance or irrelevance of stopping rules • Some (further) controversies – Multiple endpoints – One sided versus two-sided tests – Reproduction probabilities – Two-trials rules and combining P-values (c)Stephen Senn 2012 2 What am I trying to do? • Good question! • It’s a big field that statisticians (and others) have been arguing about for nearly a century • All I am going to be able to do is touch on some issues (c)Stephen Senn 2012 3 Introduction to P-values – History – How they are defined and calculated – What they are – What they are not (c)Stephen Senn 2012 4 John Arbuthnot (1667-1753) An argument for divine providence, taken from the constant regularity observ’d in the births of both sexes. Male and Female christenings 1629-1710 in London showed an excess of males in every single one of 82 years. “This Equality of [adult] Males and Females is not the Effect of Chance but Divine Providence”*. Arbuthnot * Quoted by Anders Hald in his book, A History of Probability and Statistics. (c)Stephen Senn 2012 5 Arbuthnot’s Data (c)Stephen Senn 2012 6 Sex Ratio Christenings (c)Stephen Senn 2012 7 The Essence of Arbuthnot’s Argument • Suppose the null hypothesis is true – The pattern is a chance pattern • Calculate the probability of the result • This probability is very small – (1/2)82 (one tailed) – 2(1/2)82 = (1/2)81(two tailed) • Therefore we choose to reject the null hypothesis (c)Stephen Senn 2012 8 Daniel Bernoulli (1700-1782) • Nephew of James Bernoulli (1654-1705), a key figure in the history of probability • Early example of significance test (1734) • Planetary orbits appeared to be closely aligned – similar planes • Was this just chance? (c)Stephen Senn 2012 9 D Bernoulli’s Data Mercury 256' Venus 410' Earth 730' Mars 549' Jupiter 621' Saturn 558' (c)Stephen Senn 2012 10 D Bernoulli’s Test • Most extreme angle is 7º30’ • Probability of any planet having inclination 7º30’ = 7º30’/90 º= 1/12 • Probability of six planets with inclination 7º30’ = (1/12)6 = 1/2985984 • This result is therefore unlikely if chance is the explanation (c)Stephen Senn 2012 11 Common Features • A null hypothesis – No difference in probability of male or female birth – Alignment of planets is random • A test statistic – Number of years excess male births – Maximum angle of declination • The probability under the null hypothesis (c)Stephen Senn 2012 12 Difference Between the Two • Exact versus tail area – Arbuthnott : probability of most extreme case – Benoulli : probability of all cases as extreme or more extreme • Interpretation – Arbuthnott: P-value & likelihood & likelihood ratio – Bernoulli: P-value only (c)Stephen Senn 2012 13 (c)Stephen Senn 2012 14 What are P-values? • They are a measure of unusualness of the data • The probability of a result as extreme or more extreme than that observed given that the null-hypothesis is true • This requires an agreed definition of what is more extreme (c)Stephen Senn 2012 15 Probability of results as extreme or more extreme than that observed = P-value (one-sided) (c)Stephen Senn 2012 Observed value of statistic 16 What P-Values are not • They are not the probability that the null hypothesis is true • They are not (usually) the likelihood under the null hypothesis (c)Stephen Senn 2012 17 Warning! • A common fallacy is to think that the Pvalue is a statement about the probability of the hypothesis. • It is a statement about the probability of the data given the hypothesis. – Strictly speaking of the data plus more extreme data • Probability statements are not reversible – An example is given on the next slide (c)Stephen Senn 2012 18 The Prosecutor’s Fallacy • There is a one in a million probability of the DNA matching • Therefore there are 999,999 chances in a million that the the suspect is guilty. – Wrong • In a population of 300 million (as say in USA) there are 300 similar profiles. • Therefore we could make the same statement about 300 individuals (c)Stephen Senn 2012 19 The Mistake • The probability of the observed DNA profile given innocence is one thing. • The probability of innocence given the DNA profile is another. • They are not the same NB P-value is not the probability of the null hypothesis being true (c)Stephen Senn 2012 20 Invalid Inversion 'You might just as well say,' added the March Hare, 'that “I like what I get" is the same thing as "I get what I like"!' Lewis Caroll, Alice in Wonderland • Common fallacy • Known to philosophers as ‘the error of the transposed conditional’ • The probability of A given B is not the same as the probability of B given A (c)Stephen Senn 2012 21 A Simple Example • Most women do not suffer from breast cancer • It would be a mistake to conclude, however, that most breast cancer victims are not women • To do so would be to transpose the conditionals • This is an example of invalid inversion (c)Stephen Senn 2012 22 Some Plausible Figures Probability breast cancer given female = 550/31,418=0.018 Probability female given breast cancer =550/553=0.995 (c)Stephen Senn 2012 23 A Little Maths PA B P B A PA B P B PA B P A U nless P B P A , P A B P B A So invalid inversion is equivalent to a confusion of the marginal probabilities. The same joint probability is involved in the two conditional probabilities but different marginal probabilities are involved (c)Stephen Senn 2012 24 P-values as evidence? – Distribution under null – Distribution under alternative – Relevance or irrelevance of stopping rules (c)Stephen Senn 2012 25 (c)Stephen Senn 2012 26 (c)Stephen Senn 2012 27 (c)Stephen Senn 2012 28 Sequential Trial Definition: A clinical trial in which the results are analysed at various intervals with the intention of stopping the trial when a 'conclusive result' has been reached. A stopping rule is usually defined in advance. In frequentist statistics the result of our analysis is a significance test (or hypothesis test). Now suppose that we propose to carry out a clinical trial in which we intend to study 500 patients at the most. However, after 250 patients have been studied we will carry out a significance test to see if the treatment is ‘already significant’. If it is we will stop. Now if we use conventional ‘fixed sample size techniques’ to examine ‘significance’ we shall claim significance rather more often than if we had run a fixed sample size trial, even when the null hypothesis is true. (c)Stephen Senn 2012 29 Illustration of ‘alpha inflation’ Circumstances under which fixed and sequentially run trials would achieve significance if there were no adjustment for multiple testing. It is assumed that 500 patients are studied if the trial runs to completion but that one looks after 250 All 500 Not Result Significant First 250 Significant Not Significant Neither Both Significant Sequential only Both (c)Stephen Senn 2012 30 Moral • Other things being equal, the chance of declaring a significant result is higher if we analyse sequentially • Therefore we should use more stringent significance levels • Or, equivalently, adjust our P-values upwards • Or should we…….? (c)Stephen Senn 2012 31 Example A Theoretical Problem? • Is there a bias in meta-analysing trials that have themselves been run sequentially? • Example on the next slide – Single look – No adjustment for repeated testing – Information fraction at look varies in steps of 0.01 from 0.01 to 0.99 – Treatment effect is 1 – Standard error for full information fraction of 1 is 1 (c)Stephen Senn 2012 32 0.5 1.0 1.5 2.0 Two approaches to weighting sequential trials 0.0 Weighted by size Weighted equally 0.0 0.2 0.4 0.6 0.8 1.0 Information fraction at first look (c)Stephen Senn 2012 33 Simple proof? • We can provide a general proof • The lines go like this – Trials that stop early will overestimate the treatment effect – Trials that don’t step early will have an early part corresponding to stopped trials that underestimates the treatment effect – Provided that these two parts are added together in the proportion in which they arise there can be no bias – This involves weighting trials as a whole according to information provided (c)Stephen Senn 2012 34 Some (further) controversies – Multiple endpoints – One sided versus two-sided tests – Reproduction probabilities – Two-trials rules and combining P-values (c)Stephen Senn 2012 35 Multiple endpoints • If you carry out many statistical tests then (unless they are all perfectly concordant) the probability that at least one is significant is greater than the probability that any one is • Hence the probabity of rejecting at least one null hypothesis is greater than the probability of rejecting any given null hypothesis • This suggests that the P-value for the tests as a family should not be the minimum (naïve) P-value • Hence if you are going to judge significance for the family by using the lowest P-value, the individual P-values should be adjusted upwards (c)Stephen Senn 2012 36 Types of multiplicity • Multiple treatments Many doses Gold standard trials • Sequential testing • Multiple outcomes The first and second usually provide cases of (approximately) known structural correlation: for example 0.5 for various treatments v placebo in the case of multiple treatments The third is (in principle) unstructured and the correlations are not known Simplest common approach in the third case is the Bonferroni correction Multiply each P-value by the number of tests (c)Stephen Senn 2012 37 Types of Power Individual power = probability to reject an individual (specific) false hypothesis Disjunctive* power = probability to reject at least one false hypothesis Conjunctive* power = probability to reject all false hypotheses Average power = average proportion of rejected false hypotheses • These have also been referred to as minimal and maximal power respectively but caution, in that case minimal power is greater than maximal power (c)Stephen Senn 2012 38 Bonferroni Basic Considerations • Protects you against worst possible configuration of correlation matrix • Is conservative • Therefore the individual power is lower • However, the disjunctive power may well be higher • Of course the conjunctive power is lower (c)Stephen Senn 2012 39 Compound Symmetry for Standardised Variable 1 1 1 Of course, this is not very realistic. It is just being used as a simple example to get some sort of impression of the effect of correlation on power. (c)Stephen Senn 2012 40 Theoretical advantage Where we have compound symmetry we can use a single latent variable (effectively the first principle component) and consider the conditional distribution of all other variables given this latent variable. The other variables are, in fact, conditionally independent given this latent variable and this considerably eases the calculation of various error rates (c)Stephen Senn 2012 41 Disjunctive power 80% Power for one outcome P ow er 0.75 The case shows 80% power for a single outcome for a 5% type I error rate two-sided 0.8 It is assumed that the noncentrality parameter is the same for all variables 0.6 0 0.2 0.4 0.6 0.8 Correlation 1 Outcome 2 Outcomes 5 Outcomes 10 Outcomes (c)Stephen Senn 2012 42 Conjunctive power 80% Power for one outcome 1 P ow er The case shows 80% power for a single outcome for a 5% type I error rate two-sided It is assumed that the noncentrality parameter is the same for all variables 0.5 0 0 0.2 0.4 0.6 0.8 Correlation 1 Outcome 2 Outcomes 5 Outcomes 10 Outcomes (c)Stephen Senn 2012 43 One sided or two-sided tests? • Should we carry out one sided or two sided significance tests? • The latter require us to consider departures from the null in both directions • This suggests that the P-value should be doubled • This simple issue has attracted a huge amount of controversy (c)Stephen Senn 2012 44 Since superiority cannot be excluded we must use two-sided tests Because our test situation is not to choose between H0 : = 0 H 1: > 0 but between H0 : = 0 H 1: 0 we must have two-sided tests (c)Stephen Senn 2012 45 Because we would never register an inferior drug we can use one-sided tests According to this point of view in reality our task is to choose between H0: 0 H1: > 0. A one sided test of size for = 0 will have type one error < for any other value of in H0, therefore there is no need to have a two-sided test. (c)Stephen Senn 2012 46 P ra ctical S itu ation P rob ab ility of R egisterin g D ru g T he dru g is harm ful < 0.05 T he dru g is useless 0.05 T he dru g brin gs benefit > 0.05 (c)Stephen Senn 2012 47 Label Invariance Requires us to Use Two-Sided Tests The advantage of a two-sided test is that it is ‘label invariant’. We can reverse the labels of the drugs and come to the same conclusion. (c)Stephen Senn 2012 48 If we use Two-Sided Tests we Cannot Conclude Superiority This is just silly. It is based on the premise that a two sided test implies that we have H0 : = 0 and H1 : 0. Rejection of H0 implies that the non-directional H1 only can be asserted. However we could equally well write H0 : = 0 and H1a : > 0. H0 : = 0 and H1b : < 0. Now if we test H0 twice, once against each of these two alternative hypotheses, our overall type one error rate will be maintained at provided each test is carried out at size /2. (c)Stephen Senn 2012 49 We Have Used Two-Sided Tests in the Past and Must Continue to do So Whatever standard we use is arbitrary. The important thing is to be consistent. Whether you regard standard tests as being two-sided at the 5% level or onesided at the 2.5% level is irrelevant: either way we must continue to do as we did. (c)Stephen Senn 2012 50 The Whole Debate is Irrelevant This is the Bayesian view. What you need is 1) Prior probabilities 2) Consequences of decisions. All the frequentist confusions arise because one tries to do without one or other of these elements. (c)Stephen Senn 2012 51 Paper by Goodman Statistics in Medicine • Considers the situation where you have P=0.05 for one trial and you then run another • What is the probability of repeating a result that is just significant at the 5% level (p=0.05)? • Answer 50% – If true difference is observed difference – If uninformative prior for true treatment effect (c)Stephen Senn 2012 52 Goodman’s Criticism • What is the probability of repeating a result that is just significant at the 5% level (p=0.05)? • Answer 50% – If true difference is observed difference – If uninformative prior for true treatment effect (c)Stephen Senn 2012 53 Sauce for the Goose and Sauce for the Gander • This property is shared by Bayesian statements • Hence, either – The property is undesirable and hence is a criticism of Bayesian methods also – Or it is desirable and is a point in favour of frequentist methods (c)Stephen Senn 2012 54 Three Possible Questions • Q1 What is the probability that in a future experiment, taking that experiment's results alone, the estimate for B would after all be worse than that for A? • Q2 What is the probability, having conducted this experiment, and pooled its results with the current one, we would show that the estimate for B was, after all, worse than that for A? • Q3 What is the probability that having conducted a future experiment and then calculated a Bayesian posterior using a uniform prior and the results of this second experiment alone, the probability that B would be worse than A would be less than or equal to 0.05? (c)Stephen Senn 2012 55 T h r e e K i n d s o f R e p l i c a t i o n P r o b a b i l i t y 1 . 0 p r o b a b i l i t y Q 1 p r o b a b i l i t y Q 2 p r o b a b i l i t y Q 3 0 . 8 Probailty 0 . 6 0 . 4 0 . 2 0 . 0 1 . 0 1 0 2 34 5 6 7 8 0 . 0 1 0 2 34 5 6 7 8 1 . 0 1 0 R a t i o o f s t a n d a r d e r r o r s (c)Stephen Senn 2012 56 Why Goodman’s Criticism is Irrelevant “It would be absurd if our inferences about the world, having just completed a clinical trial, were necessarily dependent on assuming the following. 1. We are now going to repeat this experiment. 2. We are going to repeat it only once. 3. It must be exactly the same size as the experiment we have just run. 4. The inferential meaning of the experiment we have just run is the extent to which it predicts this second experiment.” (c)Stephen Senn 2012 57 The Two-Trials Rule The FDA usually requires two trials to be significant. If only two trials are run this implies a type I error rate of 1/20 x 1/20 = 1/400. Similarly the consumer’s risk is at 1/40 x 1/40 = 1/1600. If however we wish to have a consumer’s risk of 1/1600 then a more powerful approach is to require (z1 + z2) > 2 x 3.227 for two trials of equal size. (c)Stephen Senn 2012 58 10 1.96 z score for trial 2 1.96 10 10 -10 two trials boundary z score for trial 1 pooled trials boundary (c)Stephen Senn 2012 59 So What’s the Point? • Could use more powerful pooled trials approach. • Reason why not might be random effects argument. • Trials can be struck by a ‘random gremlin’. • Then no logic, however, in making trials identical. (c)Stephen Senn 2012 60 Contour plots for Tippet boundary 0.95 0.90 0.85 0.80 0.8 0.6 0.550.650.75 0.85 0.35 0.25 0.45 0.9 0.20.30.40.5 0.6 0.7 0.8 0.95 0.75 0.70 0.65 0.4 0.60 0.55 0.50 0.2 0.45 0.40 0.2 0.4 0.6 0.35 0.8 0.30 0.25 0.20 0.15 0.10 0.05 Contour plots Fisher combined P-values 0.95 0.9 0.8 0.6 0.4 0.450.550.65 0.35 0.7 0.3 0.5 0.25 Contour plots for average Z score 0.85 0.6 0.8 0.75 0.2 0.4 0.15 0.2 0.9 0.3 0.25 0.2 0.85 0.8 0.75 0.7 0.5 0.6 0.65 0.4 0.55 0.2 0.4 0.1 0.45 0.35 0.15 0.6 0.2 0.95 0.8 0.05 0.1 0.4 0.6 0.8 0.2 (c)Stephen Senn 2012 0.4 0.6 0.8 61 In Summary • It is a very bad idea to analyse data only by Pvalues • Estimates and confidence intervals should be given as well • Be careful in interpreting P-values • Look at data from lots of different ways (c)Stephen Senn 2012 62 References 1. Senn SJ. Two cheers for P-values. Journal of Epidemiology and Biostatistics 2001; 6: 193-204. 2. Senn SJ. A comment on replication, p-values and evidence S.N.Goodman, Statistics in Medicine 1992; 11:875-879. Statistics in Medicine 2002; 21: 2437-2444. 3. Senn SJ. P-Values. In Encylopedia of Biopharmaceutical Statistics, Chow SC (ed). Marcel Dekker, 2003. (c)Stephen Senn 2012 63