Report

SUMMARY Hypothesis testing Self-engagement assesment = 7.8 = 0.76 Null hypothesis song Null hypothesis: I assume that populations without and with song are same. At the beginning of our calculations, we assume the null hypothesis is true. no song Hypothesis testing song • population = 7.8, = 0.76 • sample = 30, = 8.2 = Because of such a low probability, we interpret 8.2 as a significant increase over 7.8 caused by undeniable pedagogical qualities of the 'Hypothesis testing song'. 8.2 − 7.8 = 2.85 0.76 30 corresponding probability is 0.0022 7.8 8.2 Four steps of hypothesis testing 1. Formulate the null and the alternative (this includes one- or two-directional test) hypothesis. 2. Select the significance level α – a criterion upon which we decide that the claim being tested is true or not. --- COLLECT DATA --3. Compute the p-value. The p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true. 4. Compare the p-value to the α-level. If p ≤ α, the observed effect is statistically significant, the null is rejected, and the alternative hypothesis is valid. One-tailed and two-tailed one-tailed (directional) test two-tailed (non-directional) test Z-critical value, what is it? NEW STUFF Decision errors • Hypothesis testing is prone to misinterpretations. • It's possible that students selected for the musical lesson were already more engaged. • And we wrongly attributed high engagement score to the song. • Of course, it's unlikely to just simply select a sample with the mean engagement of 8.2. The probability of doing so is 0.0022, pretty low. Thus we concluded it is unlikely. • But it's still possible to have randomly obtained a sample with such a mean mean. Four possible things can happen Decision State of the world Reject H0 Retain H0 H0 true 1 3 H0 false 2 4 In which cases we made a wrong decision? Four possible things can happen Decision Reject H0 State of the world H0 true H0 false Retain H0 1 4 In which cases we made a wrong decision? Four possible things can happen Decision Reject H0 State of the world H0 true H0 false Retain H0 Type I error Type II error Type I error • When there really is no difference between the populations, random sampling can lead to a difference large enough to be statistically significant. • You reject the null, but you shouldn't. • False positive – the person doesn't have the disease, but the test says it does Type II error • When there really is a difference between the populations, random sampling can lead to a difference small enough to be not statistically significant. • You do not reject the null, but you should. • False negative - the person has the disease but the test doesn't pick it up • Type I and II errors are theoretical concepts. When you analyze your data, you don't know if the populations are identical. You only know data in your particular samples. You will never know whether you made one of these errors. The trade-off • If you set α level to a very low value, you will make few Type I/Type II errors. • But by reducing α level you also increase the chance of Type II error. Clinical trial for a novel drug • Drug that should treat a disease for which there exists no • • • • • • therapy If the result is statistically significant, drug will me marketed. If the result is not statistically significant, work on the drug will cease. Type I error: treat future patients with ineffective drug Type II error: cancel the development of a functional drug for a condition that is currently not treatable. Which error is worse? I would say Type II error. To reduce its risk, it makes sense to set α = 0.10 or even higher. Harvey Motulsky, Intuitive Biostatistics Clinical trial for a me-too drug • Drug that should treat a disease for which there already • • • • • exists another therapy Again, if the result is statistically significant, drug will me marketed. Again, if the result is not statistically significant, work on the drug will cease. Type I error: treat future patients with ineffective drug Type II error: cancel the development of a functional drug for a condition that can be treated adequately with existing drugs. Thinking scientifically (not commercially) I would minimize the risk of Type I error (set α to a very low value). Harvey Motulsky, Intuitive Biostatistics Engagement example, n = 30 H0 : = HA : ≠ = 7.8 = 0.76 = 30 = 8.06 = 7.91 Z = 1.87 Z = 0.79 = 0.05 two-tailed test =0 www.udacity.com – Statistics Engagement example, n = 30 Which of these four quadrants represent the result of our hypothesis test? Decision Reject H0 State of the world Retain H0 H0 true H0 false www.udacity.com – Statistics Engagement example, n = 30 Which of these four quadrants represent the result of our hypothesis test? Decision Reject H0 State of the world H0 true H0 false Retain H0 X Engagement example, n = 50 H0 : = HA : ≠ = 7.8 = 0.76 = = 8.06 = 7.91 Z = 2.42 Z = 1.02 = 0.05 two-tailed test =0 www.udacity.com – Statistics Engagement example, n = 50 Which of these four quadrants represent the result of our hypothesis test? Decision Reject H0 State of the world Retain H0 H0 true H0 false www.udacity.com – Statistics Engagement example, n = 50 Which of these four quadrants represent the result of our hypothesis test? Decision Reject H0 State of the world H0 true Retain H0 X H0 false www.udacity.com – Statistics population of students that did not attend the musical lesson parameters are known 0 0 population of students that did attend the musical lesson unknown sample statistic is known Test statistic test statistic − 0 = 0 Z-test We use Z-test if we know the population mean 0 and the population s.d. 0 . New situation • An average engagement score in the population of 100 students is 7.5. • A sample of 50 students was exposed to the musical lesson. Their engagement score became 7.72 with the s.d. of 0.6. • DECISION: Does a musical performance lead to the change in the students' engagement? Answer YES/NO. • Setup a hypothesis test, please. Hypothesis test • H0: 0 = • H1: 0 ≠ • In this case doing two-sided test is the only way to test the null. You compare the sample mean of 7.72 with the population mean of 7.5. It seems that sample mean is larger than the population mean (7.72 > 7.5), but the sample s.d. is 0.6. You can't setup the onetailed test as you can't guess the correct direction of the relationship. Actually, you could very easily miss the correct direction. • = 0.05 Formulate the test statistic − 0 = 0 population of students that did not attend the musical lesson 0 known 0 unknown but this is unknown! • Instead of 0 we only know the sample s.d. • We can use it as the point estimate of population s.d. • However, this will estimate s.d. for the population exposed to the musical lesson, 0 in the above formula is for "unperturbed" population. • In this case, it is common to make an assumption that both populations have the same standard deviation. population of students that did attend the musical lesson unknown sample t-statistic − 0 = one sample t-test jednovýběrový t-test Choose a correct alternative in the following statements: 1. The larger/smaller the value of , the strongest the evidence that > 0 . 2. The larger/smaller the value of , the strongest the evidence that < 0 . 3. The further the value from 0 in either direction, the stronger/weaker evidence that ≠ 0 . t-distribution One-sample t-test − 0 = 0 : = 0 : < 0 > 0 ≠ 0 level Quiz − 0 = • What will increase the t-statistic? Check all that apply. 1. A larger difference between and 0 . 2. Larger . 3. Larger . 4. Larger standard error. Z-test vs. t-test • Use Z-test if • you know the standard deviation of the population. • If you know the sample AND you have large sample size (traditionally over 30). In addition, you assume that the population standard deviation is the same as the sample standard deviation. • Use t-test if • you don't know the population standard deviation (you know only sample standard deviation ) and have a relatively small sample size. • Tip: If you know only the sample standard deviation, always use t-test. • For two sided test and = 0.05, what are the critical values at Z- and t-distributions? Typical example of one-sample t-test • You have to prepare 20 tubes with 30% solution od NaCl. When you're finished, you measure the strength of 20 solutions. The mean strength is 31.5%, with the s.d. of 1.15%. • Decide if you have 30% solution or not? • 0 = 30% • 0 : = 30%, 1 : ≠ 30% • You use t-test in such a situation. • You could use Z-test if you have a large sample (e.g., you prepared 100 tubes), but generally it is always correct to use t-test. Dependent t-test for paired samples • Two samples are dependent when the same subject takes the test twice. • paired t-test (párový t-test) • This is a two-sample test, as we work with two samples. • Examples of such situations: • Each subject is assigned to two different conditions (e.g., use QWERTZ keyboard and AZERTY keyboard and compare the error rate). • Pre-test … post-test. • Growth over time. Example • 25 students attended a normal lesson. Their mean engagement is = 5.08. • The same 25 students then heard the „Hypotheses testing song“. Their mean engagement score is = 7.80. student 1 student 2 ⋮ student n 1 1 1 2 2 2 ⋮ ⋮ ⋮ song no song − Do the hypothesis test • Now we follow the same procedure as for the one-sample t-test, except that we use values of differences . • What will be the null? = 25, = 5.08, = 7.8 • 0 ∶ = • But this is equivalent to stating 0 ∶ − = 0 • And the alternative? • 0 ∶ ≠ • What is our point estimate for − ? • − = 5.08 − 7.8 = −2.72 Do the hypothesis test • What else do we need to calculate a t-statistic? • Wee need the standard deviation of mean differences. • We have a paired samples table, so we know each value, and we can easily calculate (do not forget, you're dividing by − 1!). • Let's say it is = 3.69. • The t-statistic = − −2.72 = 3.69 = −3.68 25 • Do we reject the null or do we fail to reject the null at the = 0.05? • Critical values for . . = − 1 = 24 for two-tailed = 0.05 are ±2.064. • We reject the null. Dependent samples • e.g., give one person two different conditions to see how he/she reacts. Maybe one control and one treatment or two types of treatments. • Advantages • we can use fewer subjects • cost-effective • less time-consuming • Disadvantages • carry-over effects • order may influence results Independent samples • Disadvantages of dependent samples become advantages of dependent samples and vice versa. • We need more subjects, it's generally more time consuming and more expensive. • No carry-over effects (each subject only gets one treatment). • Everything else is same • 0 ∶ 1 − 2 = 0, 1 ∶ 1 ≠ 2 • = 1 −2 SE • Reject 0 if < , fail to reject 0 if > . Independent samples • However, the standard error changes because it is based on two sample sizes and two standard deviations. • If we subtract normally distributed data from another normally distributed data, we get a new data set 1 , 1 − 2 , 2 = 1 − 2 , 12 + 22 • Similarly, for the sample: . . = 12 + 22 This is true only if two samples are independent! • standard error . . = 12 + 22 = 12 + 22 = 12 22 + Independent samples • However, the standard error changes because it is based on two sample sizes and two standard deviations. • If we subtract normally distributed data from another normally distributed data, we get a new data set 1 , 1 − 2 , 2 = 1 − 2 , 12 + 22 • Similarly, for the sample: . . = 12 + 22 • standard error . . = 12 + 22 = 12 + 22 = 12 22 + 1 2 An example • Again, the musical lesson. • Let's teach nN = 10 students without the musical performance, and expose different n = 20 students to the song. • What will be the null and the alternative? • 0 : = , : ≠ • Which direction will we use? • two-tailed An example • = 10, = 20 • = 5.08, = 2.65 • = 7.80, = 2.18 • Standard error = 2 2 + = 2.652 2.182 + = 0.97 10 20 • Calculate t-statistic − 5.08 − 7.80 = = = −2.80 0.97 • How will you proceed further? • calculate d.f., define , find the critical t-value, compare the t- statistic with the t-critical, decide about the null An example • . . = 10 + 20 − 2 = 28 • t-critical value for = 0.05 is ±2.048 • Reject or fail to reject the null? • Reject the null. Summary of t-tests • one-sample test (jednovýběrový test) • you test H0 : = 0 • two-sample test (dvouvýběrový test) • you test H0 : 1 − 2 = 0 • dependent samples • paired t-test (párový test) • independent samples • equal variances 1 ~2 • unequal variances 1 ≠ 2 two-sample tests F-test of equality of variances • How to know if our variances are equal or not? • var.test() in R, 0 : 1 = 2 • Test statistic is a ratio of two variances. It has an F- distribution. Each numerator and denominator has certain number of d.f. source: Wikipedia t-test in R • t.test() • Let's have a look into R manual: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/t.test.html • See my website for link to pdf explaining various t-test in R (with examples). Assumptions 1. Unpaired t-tests are highly sensitive to the violation of the independence assumption. 2. Populations samples come from should be approximately normal. • This is less important for large sample sizes. • What to do if these assumptions are not fullfilled 1. Use paired t-test 2. Let's see further Check for normality – histogram Check for normality – QQ-plot qqnorm(rivers) qqline(rivers) Check for normality – tests • The graphical methods for checking data normality still leave much to your own interpretation. If you show any of these plots to ten different statisticians, you can get ten different answers. • H0: Data follow a normal distribution. • Shapiro-Wilk test • shapiro.test(rivers): Shapiro-Wilk normality test data: rivers W = 0.6666, p-value < 2.2e-16 Nonparametric statistics • Small samples from considerably non-normal distributions. • non-parametric tests • No assumption about the shape of the distribution. • No assumption about the parameters of the distribution (thus they are called non-parametric). • Simple to do, however their theory is extremely complicated. Of course, we won't cover it at all. • However, they are less accurate than their parametric counterparts. • So if your data fullfill the assumptions about normality, use paramatric tests (t-test, F-test). Nonparametric tests • If the normality assumption of the t-test is violated, and the sample sizes are too small, then its nonparametric alternative should be used. • The nonparametric alternative of t-test is Wilcoxon test. • wilcox.test() • http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html