Report

Licence This presentation is © 2010-11, Anne Segonds-Pichon. This presentation is distributed under the creative commons Attribution-Non-Commercial-Share Alike 2.0 licence. This means that you are free: to copy, distribute, display, and perform the work to make derivative works Under the following conditions: Attribution. You must give the original author credit. Non-Commercial. You may not use this work for commercial purposes. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a licence identical to this one. Please note that: For any reuse or distribution, you must make clear to others the licence terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights. Full details of this licence can be found at http://creativecommons.org/licenses/by-nc-sa/2.0/uk/legalcode Introduction to statistics with GraphPad Prism 5 Anne Segonds-Pichon (anne.segonds-pichon@babraham.ac.uk) Experimental design Power analysis Sample size Experimental design Think stats!! • Translate your biological question into stats. • What type of data are you going to collect? e.g. Categorical or quantitative? • Very important: Difference between technical and biological replicates. • Technical replicates involve taking one sample from one tube and analysing it across multiple conditions. • Biological replicates are different samples measured across multiple conditions. Good experimental design relies on 2 principles: • replication: the more times something is repeated, the greater the confidence of ending up with a genuine result. • randomization: experimental subjects must be allocated to treatment groups at random Common errors in the design of experiments: • experiments done on an ad hoc basis • e.g. very different group size or time variability not taken into account • control and treatment done on different days • inappropriate choice of treatment group • e.g. different age group within genotypes • experiment too large or too small • e.g. groups so small, no stat analysis can be done, or only non parametric Power analysis • Typically to estimate a sufficient Sample size • Definition of power: probability of detecting the specified effect at the specified significance level. • Also: the power of a statistical test is the probability that the test will reject the null hypothesis when the null hypothesis is false. • Arbitrarily, accepted power: 80% to 90% The power analysis depends on the relationship between 6 variables: • • • • • • the the the the the the effect size of biological interest standard deviation significance level desired power of the experiment sample size alternative hypothesis (ie one or two-sided test) Fix any five of these and a mathematical relationship can be used to estimate the sixth. 1 The effect size of biological interest •the larger the effect size, the smaller the experiment will need to be to detect it. • measure of effect size: Cohen’s d with d = Mean 1 – Mean 2 pooled SD • effect size conventions: • d = 0.20 – small • d = 0.50 – medium • d = 0.80 – large 2 The standard deviation • ideally: pilot study • if not: difficult to estimate • solutions: • literature • previous experiments • best and worst case based on lowest and highest of the available estimates 3 The significance level • usually 5% (p<0.05) • p-value is ‘the probability that a result as least as extreme as the one actually found could have been found if the null hypothesis were true’. • e.g. a difference between 2 means (treatment and control) as ‘big’ as the one actually found in the experiment could be found even if the treatment had no effect. • Don’t throw away a p-value=0.051 ! 4 The desired power of the experiment: • ~80% 5 The sample size • that’s the all point! • home office: 3 Rs: • replacement, refinement, reduction Effect size Standard deviation Sample size 6 The alternative hypothesis • is it a one or two-sided test? Good news: G*Power can do it for you … … if you know all the parameters. • with no prior knowledge, the trickiest parameters are effect size and standard deviation. Power Analysis 9.0 Variable 8.5 8.0 7.5 7.0 Sample 1 Sample 2 Sample 1 Sample 2 9.0 Variable 8.5 8.0 7.5 7.0 Alternative: the resource equation method: • more appropriate in biology than medical experiments • more about the existence of a difference than the size of it E = N –T where: E is the error degrees of freedom N is the total number of experimental units T is number of treatments combinations Exemple: 5 treatment groups and 6 mice in each group E = 30 – 5 = 25 Rule-of-thumb: E should be between 10 and 20 To be handled with care ! Common errors in the statistical analysis of experiments: • failure to do any statistical analysis on numerical data • failure to screen raw data for errors • inappropriate standardisation • misinterpretation of p-values • inappropriate or incorrect statistical analysis and in particular the Student’s t-test Remember: Stats are all about understanding and controlling variation. signal noise signal noise If the noise is low then the signal is detectable … = statistical significance … but if the noise (i.e. interindividual variation) is large then the same signal will not be detected = no statistical significance In a statistical test, the ratio of signal to noise determines the significance. “To consult a statistician after an experiment is finished is often merely to ask him to conduct a post-mortem examination. He can perhaps say what the experiment died of.” R.A.Fisher, 1938 Qualitative data • = not numerical • = values taken = usually names (also nominal) • e.g. variable sex: male or female • Values can be numbers but not numerical • e.g. group number = numerical label but not unit of measurement • Qualitative variable with intrinsic order in their categories = ordinal • Particular case: qualitative variable with 2 categories: binary or dichotomous • e.g. alive/dead or male/female Analysis of qualitative data Example of data (cats and dogs.xlsx): • Cats and dogs trained to line dance • 2 different rewards: food or affection • Is there a difference between the rewards? • Is there a significant relationship between my 2 variables? – are the animals rewarded by food more likely to line dance than the one rewarded by affection? • To answer this question: – a Chi-square test Chi-square test • In a chi-square test, the observed frequencies for two or more groups are compared with expected frequencies by chance. – With observed frequency = collected data • Example with the cats and dogs.xlsx Chi-square test (2) Expected frequency = (row total)*(column total)/grand total Did theydance? * Type of Training * Animal Crosstabulation Animal Cat Did they dance? Yes Count % within Did they dance? Count % within Did they dance? Count % within Did they dance? Count % within Did they dance? Count % within Did they dance? Count % within Did they dance? No Total Dog Did they dance? Yes No Total Type of Training Food as Affection as Reward Reward 26 6 81.3% 18.8% 6 30 16.7% 83.3% 32 36 47.1% 52.9% 23 24 48.9% 51.1% 9 10 47.4% 52.6% 32 34 48.5% 51.5% Total 32 100.0% 36 100.0% 68 100.0% 47 100.0% 19 100.0% 66 100.0% Example: expected frequency of cats line dancing after having received food as a reward: Probability of line dancing: 32/68 Probability of receiving food: 32/68 Expected frequency:(32/68)*(32/68)=0.22 22% of 68 = 15.1 Did theydance? * Type of Training * Animal Crosstabulation Animal Cat Did they dance? Yes No Total Dog Did they dance? Yes No Total Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Count Expected Count Type of Training Food as Affection as Reward Reward 26 6 15.1 16.9 6 30 16.9 19.1 32 36 32.0 36.0 23 24 22.8 24.2 9 10 9.2 9.8 32 34 32.0 34.0 Total 32 32.0 36 36.0 68 68.0 47 47.0 19 19.0 66 66.0 For the cats: Chi2 = (26-15.1)2/15.1 + (6-16.9)2/16.9 + (6-16.9)2 /16.9 + (30-19.1)2/19.1 = 28.4 Is 28.4 big enough for the test to be significant? The Null hypothesis and the error types • The null hypothesis (H0): H0 = no effect • e.g.: the animals rewarded by food are as likely to line dance as the one rewarded by affection • The aim of a statistical test is to accept or to reject H0. Statistical decision True state of H0 H0 True H0 False Reject H0 Type I error False Positive Correct True Positive Do not reject H0 Correct True Negative Type II error False Negative • Traditionally, a test or a difference are said to be “significant” if the probability of type I error is: α =< 0.05 • High specificity = low False Positives = low Type I error • High sensitivity = low False Negatives = low Type II error Chi-square test: results Dog Cat 30 20 Counts Counts 30 10 0 Dance Yes Dance No 20 10 0 Food Affection Food Affection • In our example: cats are more likely to line dance if they are given food as reward than affection (p<0.0001) whereas dogs don’t mind (p=0.908). Quantitative data • They take numerical values (units of measurement) • They can be discrete (values vary by finite specific steps) or continuous (any values) • They can be described by a series of parameters: – Mean, variance, standard deviation, standard error and confidence interval The mean • Definition: average of all values in a column • It can be considered as a model because it summaries the data – Example: a group of 5 persons: number of friends of each members of the group: 1, 2, 3, 3 and 4 • Mean: (1+2+3+3+4)/5 = 2.6 friends per person – Clearly an hypothetical value • How can we know that it is an accurate model? – Difference between the real data and the model created The mean (2) • Calculate the magnitude of the differences between each data and the mean: – Total error = sum of difference From Field, 2000 =0 • No errors ! – Positive and negative: they cancel each other out. Sum of Squared errors (SS) • To avoid the problem of the direction of the error: we square them – Instead of sum of errors: sum of squared errors (SS): • SS gives a good measure of the accuracy of the model • But: dependent upon the amount of data: the more data, the higher the SS. • Solution: to divide the SS by the number of observations (N) • As we are interested in measuring the error in the sample to estimate the one in the population we divide the SS by N-1 instead of N and we get the variance (S2) = SS/N-1 Variance and standard deviation • Problem with variance: measure in squared units – For more convenience, the square root of the variance is taken to obtain a measure in the same unit as the original measure: • the standard deviation – S.D. = √(SS/N-1) = √(s2) = s • The standard deviation is a measure of how well the mean represents the data Standard deviation Small S.D: data close to the mean: mean is a good fit of the data Large S.D.: data distant from the mean: mean is not an accurate representation SD and SEM (SEM = SD/√N) • Many scientists are confused about the difference between the standard deviation (SD) and the standard error of the mean (SEM). – The SD quantifies how much the values vary from one another (scatter or spread). • The SD does not change predictably as you acquire more data. – The SEM tells you how much variability there is in this statistic across samples from the same population. • The SEM gets smaller as your samples get larger, – the mean of a large sample is likely to be closer to the true mean than is the mean of a small sample. SD and SEM The SD quantifies the scatter of the data. The SEM quantifies how far the sample mean is from the true population mean. SD or SEM ? • If the scatter is caused by biological variability, it is important to show the variation. – Report the SD rather than the SEM. • Better, show a graph of all data points, or perhaps report the largest and smallest value there is no reason to only report the mean and SD. • If you are using an in vitro system with no biological variability, the scatter can only result from experimental imprecision (no biological meaning). – Report the SEM since the SD is less useful here. • Instead, report the SEM to give your readers a sense of how well you have determined the mean. Confidence interval • 95% of observations in a normal distribution lie within +/- 1.96 SEM - So limits of 95% CI: [Mean - 1.96 SEM; Mean + 1.96 SEM] - SEM = SD/√N Error bars Type Description Standard deviation (SD) Descriptive Typical or average difference between the data points and their mean. Standard error (SEM) Inferential A measure of how variable the mean will be, if you repeat the whole study many times. Confidence interval (CI), usually 95% CI Inferential A range of values you can be 95% confident contains the true mean. SE gap ~ 4.5 n=3 SE gap ~ 2 n=3 16 Dependent variable Dependent variable 13 12 11 ~ 2 x SE: p~0.05 10 9 15 14 13 ~ 4.5 x SE: p~0.01 12 11 10 8 A 9 B A B SE gap ~ 2 n>=10 SE gap ~ 1 n>=10 12.0 11.0 ~ 1 x SE: p~0.05 10.5 10.0 9.5 A B Dependent variable Dependent variable 11.5 11.5 11.0 ~ 2 x SE: p~0.01 10.5 10.0 9.5 A B CI overlap ~ 1 n=3 CI overlap ~ 0.5 n=3 Dependent variable Dependent variable 14 12 ~ 1 x CI: p~0.05 10 8 15 ~ 0.5 x CI: p~0.05 10 6 A B A CI overlap ~ 0.5 n>=10 CI overlap ~ 0 n>=10 12 11 ~ 0.5 x CI: p~0.05 10 A B Dependent variable Dependent variable 12 9 B 11 ~ 0 x CI: p~0.01 10 9 A B Analysis of quantitative data • Check for normality • Choose the correct statistical test to answer your question: – They are 2 types of statistical tests: • Parametric tests with 4 assumptions to be met by the data, • Non-parametric tests with no or few assumptions (e.g. Mann-Whitney test) and/or for qualitative data (e.g. χ2 test). Assumptions of Parametric Data • All parametric tests have 4 basic assumptions that must be met for the test to be accurate. 1) Normally distributed data – Normal shape, bell shape, Gaussian shape • Transformations can be made to make data suitable for parametric analysis Assumptions of Parametric Data (2) • Frequent departure from normality: – Skewness: lack of symmetry of a distribution – Kurtosis: measure of the degree of peakedness in the distribution • The two distributions below have the same variance approximately the same skew, but differ markedly in kurtosis. Assumptions of Parametric Data (3) 2) Homogeneity in variance • The variance should not change systematically throughout the data 3) Interval data • The distance between points of the scale should be equal at all parts along the scale 4) Independence • Data from different subjects are independent – Values corresponding to one subjects do not influence the values corresponding to another subject. – Important in repeated measures experiments Analysis of quantitative data • Is there a difference between my groups regarding the variable I am measuring? – e.g.: are the mice in the group A heavier than the one in group B? • Tests with 2 groups: – Parametric: t-test – Non parametric: Mann-Whitney/Wilcoxon rank sum test • Tests with more than 2 groups: – Parametric: Analysis of variance (one-way ANOVA) – Non parametric: Kruskal Wallis • Is there a relationship between my 2 (continuous) variables? – e.g.: is there a relationship between the daily intake in calories and an increase in body weight? • Test: Correlation (parametric or non-parametric) Remember: Stats are all about understanding and controlling variation. signal noise signal noise If the noise is low then the signal is detectable … = statistical significance … but if the noise (i.e. interindividual variation) is large then the same signal will not be detected = no statistical significance In a statistical test, the ratio of signal to noise determines the significance. Comparison between 2 groups: t-test • Basic idea: – When we are looking at the differences between scores for 2 groups, we have to judge the difference between their means relative to the spread or variability of their scores • Ex: comparison of 2 groups control and treatment t-test (2) t-test (3) t-test (4) • 3 types: – Independent t-test • it compares means for two independent groups of cases. – Paired t-test • it looks at the difference between two variables for a single group: – the second sample is the same as the first after some treatment has been applied – One-Sample t-test • it tests whether the mean of a single variable differs from a specified constant (often 0) Example: coyote.xlsx • Question: are the males coyote bigger than the females? • First step: how do my data look like? – 4 assumptions for parametric tests – Plot the data Assumptions for parametric tests Histogram of Coyote:Freq. dist. (histogram) 10 Counts 8 Counts OK here but if several groups of different sizes, go for percentages Female Male 6 4 2 0 15 707274767880828486889092949698100 102 104 106 707274767880828486889092949698100 102 104 106 Bin Center Female Male Counts 10 5 Normality 0 15 69 72 75 78 81 84 87 90 93 96 99 102105 69 72 75 78 81 84 87 90 93 96 99 102105 Bin Center Female Male Counts 10 5 0 68 72 76 80 84 88 92 96 100 104 108 68 72 76 80 84 88 92 96 100 104 108 Bin Center Coyote 110 Maximum 100 Length (cm) Upper Quartile (Q3) 75th percentile Interquartile Range (IQR) 90 Lower Quartile (Q1) 25th percentile Median 80 Smallest data value > lower cutoff Cutoff = Q1 – 1.5*IQR 70 60 Outlier Male Female Independent t-test: example coyote.pzf 120 Standard error 95 90 85 Body length (cm) Body Mass 100 110 100 90 80 70 60 80 Female Females Male Males 95 Standard deviation 94 93 Length (cm) Body Mass 100 95 90 85 92 91 90 89 88 87 86 80 Female Male Confidence interval 85 Male Female Independent t-test: results coyote.xlsx Males tend to be longer than females but not significantly so (p=0.1045). What about the power of the analysis? Homogeneity in variance What about the power of the analysis? You would need a sample 3 times bigger to reach the accepted power of 80%. But is a 2.3 cm difference between genders biologically relevant? Another example of t-test: height husband wife.xlsx height husband wife.xlsx 20 15 Husband and Wife Difference Height 200 180 10 5 0 160 -5 140 Husband Wife -10 Normality Dependent t-test: example height husband wife.xlsx 200 15 180 Difference Height (cm) 20 160 140 10 5 0 -5 Husband Wife -10 Paired t-test = One sample t-test Results No test for homogeneity of variances Comparison of more than 2 means • Why can’t we do several t-tests? – Because it increases the familywise error rate. • What is the familywise error rate? – The error rate across tests conducted on the same experimental data. Familywise error rate • Example: if you want to compare 3 groups and you carry out 3 ttests, each with a 5% level of significance • The probability of not making the type I error is 95% (=1 – 0.05) – the overall probability of no type I errors is: 0.95 * 0.95 * 0.95 = 0.857 – So the probability of making at least one type I error is 1-0.857 = 0.143 or 14.3% – The probability has increased from 5% to 14.3% ! – If you compare 5 groups instead of 3, the familywise error rate is 40% !!!!! (=1-(0.95)n) • Solution for multiple comparisons: Analysis of variance Analysis of variance • Extension of the 2 group comparison of a ttest but with a slightly different logic: – If you want to compare 5 means, for example, you can compare each mean with another • It gives you 10 possible 2-group comparisons – Complicated ! So, the logic of the t-test cannot be directly transferred to the analysis of variance (=ANOVA) • Instead the ANOVA compares variances: – If variance between the 5 means > variance within the 5 groups (random error) • then the means must be more spread out than it would have been by chance. Analysis of variance • The statistic for ANOVA is the F ratio. • F= • F= Variance between the groups Variance within the groups (individual variability) Variation explained by the model (= systematic) Variation explained by unsystematic factors (= random variation) • If the variance amongst sample means is greater than the error/random variance, then F>1 – In an ANOVA, you test whether F is significantly higher than 1 or not. Analysis of variance Source of variation Sum of Squares df Mean Square F p-value Between Groups 2.665 4 0.6663 8.423 <0.0001 Within Groups 5.775 73 0.0791 Total 8.44 77 • Variance (= SS / N-1) is the mean square – df: degree of freedom with df = N-1 Hypothetical model Between groups variability Within groups variability Total sum of squares Parametric tests assumptions Normality Analysis of variance: Post hoc tests • The ANOVA is an “omnibus” test: it tells you that there is (or not) a difference between your means but not exactly which means are significantly different from which other ones. – To find out, you need to apply post hoc tests. – These post hoc tests should only be used when the ANOVA finds a significant effect. Analysis of variance: example protein expression.xlsx Homogeneity in variance F=0.6702/0.07896=8.49 Post hoc tests Protein expression 10 10 8 8 6 6 4 4 2 2 0 A B C D E 0 A B Cell groups E 1.5 Protein expression (Log) Protein expression (Log) D Cell groups 1.5 1.0 0.5 0.0 -0.5 -1.0 C A B C D E 1.0 0.5 0.0 -0.5 -1.0 A B C Cell groups D E 1 0.1 A B C Cell groups D E 0.4 Log(Protein Expression) Protein expression 10 0.2 -0.0 -0.2 -0.4 A B C Cell groups D E Correlation • A correlation coefficient is an index number that measures: – The magnitude and the direction of the relation between 2 variables – It is designed to range in value between -1 and +1 Correlation • Most widely-used correlation coefficient: – Pearson product-moment correlation coefficient “r” • The 2 variables do not have to be measured in the same units but they have to be proportional (meaning linearly related) – Coefficient of determination: • r is the correlation between X and Y • r2 is the coefficient of determination: – It gives you the proportion of variance in Y that can be explained by X, in percentage. Correlation: example roe deer.xlsx • Is there a relationship between parasite burden and body mass in roe deer? 30 Male Body Mass Female 25 20 15 10 1.0 1.5 2.0 2.5 Digestive Parasites 3.0 3.5 Correlation: example roe deer.xlsx There is a negative correlation between parasite load and fitness but this relationship is only significant for the males (p=0.0049 vs. females: p=0.2940). Exercises • Arachnophobia – Is it as scary to look at the picture of a spider than at a real one? • Cane toad – Is the proportion of cane toads infected by intestinal parasites the same in 3 different areas of Queensland? • Colorectal cancer (CRC) – Are DNA scores higher in people with colorectal cancer? • Migration of neutrophils – Do neutrophils go further depending on which inhibitor you use? Arachnophobia 30 Difference 20 10 0 ff di -10 -20 Answer: If you are an arachnophobe, it is scarier to look at a real spider than at the picture of one (p=0.0310). Cane toad Cane toad Number of toads 20 Infected Uninfected 15 Answer: The proportion of cane toads infected by intestinal parasites varies significantly between the 3 different areas of Queensland (p=0.0359), the animals being more likely to be parasitized in Rockhampton and Mackay than in Bowen. 10 5 0 Rockhampton Bowen Mackay Colorectal cancer 80 CRC no CRC Frequencies 60 40 20 0 0 7 14 21 28 35 Bin Center Answer: Higher DNA scores appear to be associated with a greater likelihood of CRC (p=0.0007). Migration of neutrophils Migrated neutrophils (%of total) A B C 20 15 10 5 0 Migrated neutrophils (%of total) 25 25 20 15 10 5 0 A B Genotypes C A B C Genotypes Answer: There is significant difference between the 3 inhibitors, with inhibitor C being the more effective and inhibitor A the least one (p<0.0001). * Outcome ** Predictor Choosing the Correct Statistical Test (adapted from “Choosing the correct stastitics” developed by James D. Leeper, Ph.D)