Report

Experimental Design 1 http://thefifthlevel.blogspot.com/2011/05/prologue-genesis.html Experimental Design • Strongest design with respect to internal validity 2 If X then Y and If not X, then not Y or If the program is given, then the outcome occurs and If the program is not given, then the outcome does not occur 3 Dilemma • 2 identical groups • 2 identical contexts • Same time • …. similarity 4 Course of Action • Randomly assign people from a pool to the 2 groups – probabilistically equivalent • One group gets the treatment and the other does not 5 Random Selection and Assignment • Random selection is how you draw the sample of people for your study from a population. • Random assignment is how you assign the sample that you draw to different groups or treatments in your study. Probabilistic Equivalence • Means that we know perfectly the odds that we will find a difference between two groups. • When we randomly assign to groups, we can calculate the chance that the two groups will differ just because of the random assignment. 7 External validity • Experiments are difficult to carry out • artificial situation high internal validity • Limited generalization to real contexts –> limited external validity ? 8 Two-Group Experimental Design • Simplest form: two-group posttest-only randomized experiment • No pretest required • Test for differences: t-test or ANOVA 9 Advantages • Strong against single-group threats and multigroup threats (except selection-mortality) • Strong against selection testing and selectioninstrumentation 10 Classifying Experimental Designs • Two components: signal and noise • signal-enhancing experimental design (factorial design) • Noise-reducing experimental design (covariance designs or blocking designs) 11 Factorial Designs • A factor is a major independent variable – Time and setting • A level is a subdivision of a factor. – Time (1h/4h), setting (pull-out/in-class) • 2 x 2 factorial design 12 Factorial Design • • • • X11 = 1h and in-class X12 = 1h and pull-out X21 = 4h and in-class X22 = 4h and pull-out 13 The Null Outcome • The null case is a situation where both treatments have no effect. 14 The Main Effects A main effect is an outcome that is a consistent difference between levels of a factor. 15 Main Effects 16 Main Effects 17 Interaction Effects • An interaction factor exists when differences on one factor depending on the level of the other factor. 18 How do you know if there is an interaction in a factorial design? • Statistical analysis • When it can be talked about one factor without mentioning the other factor • In graphs of group means – the lines are not parallel 19 Interaction Effects 20 Interaction Effects 21 Factorial Design Variations • 2 x 3 Example Factor 1: Treatment – psychotherapy – behavior modification Factor 2: Setting – inpatient – day treatment – outpatient 22 Main Effects 23 Main Effects 24 Interaction Effect 25 Interaction Effect 26 Factorial Design Variations • A Three-Factor Example (2 x 2 x 3) Factor 1: Dosage – 100 mg – 300 mg Factor 2: Treatment – Psychotherapy – Behavior modification Factor 3: Setting – Inpatient – Day treatment – Outpatient 27 2 x 2 x 3 Design 28 Incomplete Factorial Design • Common use is to allow for a control or placebo group that receives no treatment 29 Randomized Block Design • Stratified random sampling • To reduce noise or variance in the data • Division into homogeneous subgroups • Treatment implemented to each subgroup • Variability within each block is less than the variability of the entire sample or each block is more homogenous than the entire group 30 Randomized Block Design • Stundents are a homogenous group with exception of semester freshman sophomore junior senior 31 How blocking reduces noise? 32 Covariance Designs (ANCOVA) • Pretest-posttest randomized design • Pre-program measure = covariate • Covary it with the outcome variable • Covariates are the variables you adjust for – Effect is going to be removed 33 How does a Covariate reduce Noise? 34 How does a Covariate reduce Noise? 35 How does a Covariate reduce Noise? 36 How does a Covariate reduce Noise? 37 How does a Covariate reduce Noise? 38 Hybrid Experimental Designs • Are new strains that are formed by combining features of more established designs. 39 The Solomon Four-Group Design • Is designed to deal with a certain testing threat • 2 groups are pre-tested, 2 are not • 2 groups get a treatment, 2 do not 40 The Solomon Four-Group Design • T = Treatment Group, C = Control Group 41 The Solomon Four-Group Design • T = Treatment Group, C = Control Group 42 Switching Replication Design • The implementation of the treatment is repeated or replicated. • In the repetition, the two groups switch roles • Finally, all participants have received the treatment • Reduces social threats 43 Switching Replication Design • Period 1 – group 1 gets the treatment • Period 2 – group 2 gets the treatment 44 Switching Replication Design • Longterm treatment effect group 1 improves even though no further treatment was given 45