COLLECTING AND ANALYZING DATA: MEASURING STUDENT SUCCESS Rebecca Orr, Ph.D. Professor of Biology Agenda • Efforts made to improve student success: • Online homework using MasteringBiology • Peer tutoring • Online quizzing • • Strategies for data collection: • Preparing to begin • Modifying as collection progresses • Involving your students • Case study: Impact of pre-exam quizzing on student outcomes • Lessons learned Student Success Challenges • Have you ever heard: • “I studied so hard, and it just didn’t work!” • My question: • How can student success be improved in the introductory biology (majors) course? My Evolution of Mastering Use • Began using Mastering to create assignments and made them available for optional student use. • Student feedback indicated a trend: • Successful students were using Mastering • Course evaluation surveys by students said: “Make Mastering mandatory!” • Less successful students did not tend to utilize Mastering • Moved to making Mastering mandatory by assigning a portion of lecture credit (10%) to Mastering. Is There a Significant, Positive Correlation Between Completing MasteringBiology and Exam Scores? • Students required to complete MasteringBiology homework assignments (10% lecture grade). • Overall exam average was compared to percentage MasteringBiology completed by end of semester. • Spearman's Rank Order correlation was run: (rs(138) = 0.571, P = .000) • A statistically significant, positive correlation between percentage of MasteringBiology homework completed and exam average was found. Results • Students completing more work in MasteringBiology were more successful on exams. Problem: How to Increase Student Success in Introductory (Majors) Biology? • My Strategy, Spring 2011 semester: • Evaluate the efficacy of providing peer tutors vs. requiring MasteringBiology homework (using exam scores to measure) • Offered optional, online pre-exam quizzes to all using MasteringBiology • Results: • Peer tutoring appears to have no benefit over offering required MasteringBiology homework. • Significant, positive correlation between participating in optional quizzing and increased exam score observed. Cognitive Science of Learning • Introduction of Desirable Difficulties: “Conditions of instruction that appear to create difficulties for the learner, slowing the rate of apparent learning, often optimize long-term retention and transfer.” Testing Serves as Learning Events • Testing as a learning vehicle vs. an assessment vehicle. • Evidence shows: Practice of recalling information is more powerful than re-presentation of information. • Reality is: Students re-present information to themselves (i.e., reading a chapter over and over again, underlining things, etc.), focusing on storage rather than retrieval. Can An Online Homework Platform Be Used For Quizzing to Increase Student Success? • Introduced required pre-exam quizzing delivered via MasteringBiology • Students required to take pre-exam quizzes during the Fall 2011 semester (n=199) and Spring 2012 semester (n=174) • My Question: Does taking pre-exam quizzes using Mastering result in significantly higher exam scores? Preparation • Before beginning, set up spreadsheet to collect data necessary to answer questions. • My predesigned columns included: • Name (deleted prior to sending to statistician) • Exam # and grade • Number of pre-exam quizzes taken • Grade on each pre-exam quiz • Set up color coding to designate “drops” and “not in my course” • Data gathered by exporting grade book from Mastering. Student Participation Compliance in Required Pre-exam Quizzing Fall 2011 & Spring 2012 % Students Completing Quizzes prior to Exam % Quizzes Exam 1 Completed n= 362 Exam 2 Exam 3 Exam 4 n= 339 n= 321 n= 305 100% 78.5 59.9 62.9 59.3 67% 16.3 22.4 * * 50% * * 26.5 25.9 33% 3.0 11.5 * * 0% 2.2 7.4 10.3 14.8 Comparison of Exam Averages 100% Quiz Takers to 0% Quiz Takers Fall 2011 Spring 2012 Comparison of Exam Averages by Percentage of Quiz Taking: Spring 2012 Analysis by Groups • Scores on Exams 1 and 2 were averaged, and students were grouped by these averages. • Groups Established: • Pass: 70%+ • Fail: <70% • Low (“weak student”): <60% • Middle (“average student”): 60%-80% • High (“strong student”): >80% • Student scores were compared by group for 100% quiz takers vs. 0% quiz takers. Performance of Passing vs. Failing Students: 100% Quiz Takers vs. 0% Quiz Takers Exam 3 Exam 4 Performance by Low, Middle and High Students: 100% Quiz Takers vs. 0% Quiz Takers Exam 3 Exam 4 Data Analysis Often Generates New Questions! • Can students’ compliance with pre-exam quiz requirement be increased? • If percent pre-exam quiz compliance increases, will significant gains in student success in the course be observed? Impact of Data Collection on Course Approach • Tell the students about the data! • Ask them to predict! Who earns higher exam scores: • Quiz takers or non quiz takers? • Quiz takers that fail their quizzes or non quiz takers? • Show them the data, and ASK: • “What group do you want to be in?!” The blue group? Or the red group?! Pre-exam Quiz Participation Rate Increased Significantly in Fall 2012 vs. Spring 2012 85 80 75 70 65 60 55 50 45 40 35 30 25 20 15 10 5 0 Exam 1 Exam 2 Exam 3 Exam 4 Spring: 100% Participation Fall: 100% Participation Spring: 0% Participation Fall: 0% Participation Percentage of ABC vs. DFW Grades Earned in Spring 2012 vs. Fall 2012 70.00% 60.00% 50.00% 40.00% Spring Fall 30.00% 20.00% 10.00% 0.00% ABC DFW Study Results • Student participation in pre-exam quizzing results in significantly higher exam grades. • Benefit realized to student is based on participation (rather than performance) in quizzing. • Participation in pre-exam quizzing results in significantly higher exam scores for students of diverse abilities. • Stronger students may see slightly greater benefits from pre-exam quizzing (as evidenced by exam averages). • Significant increase in student compliance with required quizzing was accompanied by significantly higher percentage of ABC rates vs. DFW rates. Student Perception of Mastering: Homework and Pre-Exam Quizzing 100 90 80 70 60 50 40 30 20 10 0 Mastering Mastering Quizzes Quizzing Homework Homework Increased Results Led to Helped Me Increased Understanding + Studying Keep up with Understanding Material Plan to continue quizzing in future Study Conclusions • Pre-exam quizzing using online platform seems to provide a viable testing environment for student learning. • Using online platform allows instructors more time to teach! • All students showed positive gains in exam scores as a result of participating in pre-exam quizzing. Lessons learned • Begin process of gaining IRB approval right away. • Consult with your statistician about what you want to investigate before you start. • Speak with colleagues about your investigation to get a feel for questions they may raise. • If introducing a new teaching method or technology, consider giving a student survey to gather their thoughts and comments. Lessons learned (cont.) • Design your data collection tools before beginning the study. • Gather and organize more than you think you need! • Stay current in updating master spreadsheet throughout the semester: • Falling behind can become overwhelming! • New questions often arise as you review the data. • Modifications in what you record may be necessary. Much easier to do this “as you go” than to go back and wade through the entire semester’s worth of data! • Tell students about your study- involve them in the process when possible. Benefits of Studying Student Success Data • Allows you to identify what you are doing that is truly effective. • Sharing results with students motivates them and helps them to feel that your requirements will yield positive gains for them. • Provides intellectual challenge to you that provides additional purpose and interest! Acknowledgements • Shellene Foster Professor of Mathematics, Collin College • Pushpa Ramakrishna, Ed.D. Professor of Biology, Chandler-Gilbert Community College References • 1. Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-64). New York: Worth Publishers. • 2. Halamish, V., & Bjork, R. A. (2011). When does testing enhance retention? A distributionbased interpretation of retrieval as a memory modifier. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37, 801-812.