Report

Longitudinal Analysis of Effects of Reclassification, Reporting Methods, and Analytical Techniques on Trends in Math Performance of Students with Disabilities Yi-Chen Wu, Martha Thurlow, & Sheryl Lazarus National Center on Educational Outcomes University of Minnesota This paper was developed, in part, with support from the U.S. Department of Education, Office of Special Education Programs grants (#H373X070021and #H326G110002). Opinions expressed herein do not necessarily reflect those of the U.S. Department of Education or Offices within it. NCEO Web site (http://www.cehd.umn.edu/nceo/) Outline Background Achievement gap Explanations Ysseldyke and Bielinski (2002) study Questions Method Data source Analytical Techniques Results Conclusions Achievement gap Focused on race/ethnicity or poverty. Less attention on achievement gaps between SPED vs. Non-SPED Research on Achievement Gap (Chudowsky, Chudowsky, & Kober, 2009a; 2009b) Examined gaps for subgroups by proficiency rate & mean SS, but no comparison between SPED and NonSPED Examined the achievement over time for SWD, but not the gap between SWD vs. SWOD over time 4 Explanations on gap increasing over time between SWD and SWOD SPED drop out of school=high achievement (McMillen & Kaufman, 1997) Tests given in higher grades are less valid for SWD (Thurlow & Ysseldyke, 1999; Thurlow, Bielinski, Minnema, & Scott, 2002) Students with lower performance moved in SPED and students with higher performance move out SPED (Ysseldyke and Bielinski, 2002) 5 Ysseldyke and Bielinski (2002) study Explored the extent to which reclassification impacts the size of the achievement gap between GED and SPED across grades. to compare the effects of different reporting methods, and to examine the effects of reclassification They argued that fair comparisons involved using clearly defined and consistent comparison groups, and that special education status complicates the reporting because status changes over time. Ysseldyke and Bielinski (2002) study They used three methods to analyze trends in performance (cross-sectional, cohort-static and cohortdynamic), and found that gap trends depended on the method used examined how the use of scaled scores and effect size could be used for reporting results. Purpose The Ysseldyke and Bielinski (2002) study did not use proficiency to examine the reporting results is now more than a decade old was completed prior to the implementation of ESEA 2001. There is a need to take a new look at how achievement gap trends are affected by the method used to calculate them. Research Questions Reporting Methods: How does the use of crosssectional, cohort-static, and cohort-dynamic data analysis methods affect interpretation of trends in the performance of students with disabilities? Analytical Techniques: How does the score used in the analyses (proficiency level, scaled score, effect size) affect interpretation of trends and achievement gaps? Reclassification: To what extent do students move in and out of special education each year, and what are the achievement characteristics of those who do and do not move? Method Data source used math assessment data for grades 3-8 from a midwestern state Cross-sectional 2005-06 to 2009-10 305,819 records Cohort 2005-06 to 2009-10+ 2004-05 (G3-8) 8,231 students with 6-yr records Method- Methods Used to Measure Gap Cross-sectional five years of data were used to calculate the average performance to reduce year-to-year variations that might affect results if data from a single year were selected. Cohort-static A cohort across six years Group membership stayed the same across years. Cohort-dynamic group membership was redefined every year Method- Analytical Techniques Results—RQ1 How does the use of cross-sectional, cohort-static, and cohort-dynamic data analysis methods affect interpretation of trends in the performance of students with disabilities? Using PF to show the trend over time among the three methods used to measure gap Results—Comparing reporting methods Percentage of Students above Proficiency Level 100 90 80.2 81.3 79.4 78.9 80 70 59.1 60 76.1 76.7 54.0 21-->47 47.4 50 38.4 40 30.0 29.4 7 8 30 20 Non-SPED 10 SPED 0 3 4 5 6 Grade Figure 1. Cross-sectional method: Percentage of students above proficiency level on math assessment by SPED and non-SPED Results—Comparing reporting methods Percentage of Students above Proficiency Level 100 90 82.1 84.3 83.2 60.1 61.6 60.5 81.8 81.0 83.3 80 70 56.6 60 61.6 54.7 50 22->21 40 30 20 Non-SPED 10 SPED 0 3 4 5 6 7 8 Grade Figure 2: Cohort-static method: Percentage of students above proficiency level on math assessment by SPED and Non-SPED Results—Comparing reporting methods Percentage of Students above Proficiency Level 100 90 82.1 84.6 83.6 81.9 81.6 84.2 80 70 60.1 60 56.3 52.6 45.6 50 34.6 40 38.8 22-->45 30 20 Non-SPED 10 SPED 0 3 4 5 6 7 8 Grade Figure 3. Cohort-dynamic method: The percentage of students above proficiency level on math assessment by SPED and Non-SPED Results—Comparing reporting methods Cross-sectional Quit different Quite similar Steady Percentage of Students above Proficiency Level 100 90 80.2 81.3 79.4 78.9 80 70 59.1 60 76.1 76.7 30.0 29.4 7 8 54.0 47.4 50 38.4 40 30 20 Non-SPED 10 SPED 0 3 4 5 6 Grade Cohort-dynamic Cohort-static Percentage of Students above Proficiency Level 90 82.1 84.3 60.1 61.6 83.2 81.8 81.0 83.3 80 70 60.5 56.6 60 61.6 54.7 50 40 30 20 Non-SPED 10 SPED 0 Percentage of Students above Proficiency Level 100 100 90 82.1 84.6 83.6 81.9 81.6 84.2 80 70 60.1 60 56.3 52.6 45.6 50 34.6 40 38.8 30 20 Non-SPED 10 SPED 0 3 4 5 6 Grade 7 8 3 4 5 6 Grade 7 8 Results—RQ2 How does the score used in the analyses (proficiency rate, scaled score, effect size) affect interpretation of trends and achievement gaps? Results—Comparing Analytical Techniques 60 Percentage of Students above Proficiency Level 50 47.4 47.0 40.6 40 32.0 30 20 28.4 22.0 21.1 27.3 22.6 46.2 45.4 36.3 31.0 22.7 25.1 26.2 21.7 Cross Sectional 10 Cohort Dynamic Cohort Static 0 3 4 5 6 7 8 Grade Figure 4. Percent proficient: Achievement gap (difference between non-SPED and SPED) in percent proficient on math assessment Results—Comparing Analytical Techniques 50 43.6 45 38.8 40 Scale Score 35 35.4 30 25 20 41.8 40.5 26.8 22.7 20.9 26.0 22.0 28.5 26.0 23.4 24.9 21.3 20.2 15 10 Cross Sectional 5 Cohort Dynamic Cohort Static 0 3 4 5 6 7 8 Grade Figure 5. Scaled score: Achievement gap (difference between nonSPED and SPED) in mean scaled score on math assessment Results—Comparing Analytical Techniques 0.0 Cohort-Static Cohort-Dynamic Effect Size -0.2 -0.4 -0.6 Cross-Sectional -.51 -.63 -.57 -0.8 -.68 -.77 -.60 -.62 -.55 -.65 -.74 -.76 -1.0 -.94 -1.00 -.99 -.96 -1.14 -1.2 3 4 5 Grade 6 7 -1.10 8 Figure 6. Effect size: Achievement gap (difference between nonSPED and SPED) in effect size on math assessment Results—Comparing analytical techniques Proficiency Level 60 Percentage of Students above Proficiency Level 50 47.4 47.0 40.6 40 32.0 30 28.4 22.0 20 21.1 27.3 22.6 46.2 45.4 36.3 31.0 25.1 22.7 26.2 21.7 Quit different Quite similar Steady Cross Sectional 10 Cohort Dynamic Cohort Static 0 3 4 5 6 7 8 Grade Effect size Scaled Score 0.0 50 38.8 40 Scale Score 35 35.4 30 25 20 41.8 40.5 26.8 22.7 20.9 26.0 22.0 28.5 26.0 23.4 24.9 21.3 20.2 15 10 Cross Sectional 5 Cohort Dynamic Cohort-Static Cohort-Dynamic -0.2 Effect Size 43.6 45 -0.4 -0.6 -.51 -.63 -.57 -0.8 -.68 -.77 -.60 -.62 3 4 5 6 Grade 7 8 -.55 -.65 -.74 -.76 -1.0 -.94 -1.00 -.99 -.96 Cohort Static 0 Cross-Sectional -1.14 -1.2 3 4 5 Grade 6 7 -1.10 8 Results—RQ3 To what extent do students move in and out of special education each year, and what are the achievement characteristics of those who do and do not move? Results—Reclassification Note: NS1 = Students who remained in non-special education in both of two consecutive years; NS2 = Students who moved from non-special education to special education in the second of two consecutive years; S1 = Students who remained in special education in both of two consecutive years; S2 = Students who moved from special education to non-special education in the second of two consecutive years. Figure 7. Mean math scaled scores by special education status across years Results—Reclassification Non-SPED only Students stayed in non-SPED for six years Non-SPED to SPED Students moved from non-SPED to SPED only once over six years SPED to Non-SPED Students moved from SPED to non-SPED only once over six years Back and forth Students moved between SPED and non-SPED more than once over six years SPED only Students stayed in SPED for six years Effect Size Results—Reclassification 0.0 -0.2 -0.4 -0.6 -0.8 -1.0 -1.2 -1.4 -0.28 -0.32 -0.29 -0.30 -0.31 -0.25 -0.94 -1.02 -1.01 -1.16 -0.94 -1.00 -0.94 -1.22 -1.15 -1.11 -0.74 -1.06 -1.16 -1.18 -1.09 -1.22 -1.29 3 5 6 7 8 Grade SPED to Non-SPED (N=730) Back and Forth (N=180) -1.25 4 Non-SPED to SPED (N=251) SPED only (N=434) Figure 8. The effect size between different reclassification groups in math assessment by using non-SPED only group as the reference group Discussion and Conclusion Different methods of reporting data present different pictures of the gap between SPED and non-SPED This study was undertaken to update the work done more than a decade ago by Ysseldyke and Bielinski (2002) Replicated + proficiency level Confirmed Suggestions Discussion and Conclusion Suggestions The choice of method affects what the results look like and the possible interpretation of findings. Tracking individual student performance provides a better indication of how well schools are educating their students than cross-sectional models where the grade remains the constant but the students change. Cross-sectional models should not be used when examining trends across grades. Cohort-static and cohort-dynamic methods enable educators to make comparisons among individual students Discussion and Conclusion Specific situation for each reporting method If the goal is to know how well students do yearly without considering changing students => cross-sectional http://www.schooldigger.com/go/MN/schools/3243001386 /school.aspx If states and districts want to account with precision for the reclassification of students each year. => cohort-dynamic When the goal is to account for individual student performance over time without regard to the nature of services received=> cohort-static