Assessing Student Learning Outcomes in Writing Courses Lisa Marchand Cosumnes River College Sacramento, California ACCJC Accreditation Standards California Community Colleges are required to establish “well-documented, ongoing processes for evaluating” effectiveness at all levels of the institution. The foundation of this evidence must be data collected and analyzed in a consistent way, over time. By the Year 2012 Engage in an ongoing dialog about student learning Have a well documented process for evaluating Student Learning Outcomes Show tangible evidence that your program is continuously moving toward improved student learning. Think about your own classes or program… How do you assess students writing skills? Are Student Learning Outcomes in place? Are they measurable? How are they measured? Take five minutes to talk it over! Cosumnes River College Questions: How well can students demonstrate that they have met the Student Learning Outcomes for writing courses that they have passed? How well can students who have placed into a level through the assessment process demonstrate the skills specified in the prerequisite course’s SLO’s? Existing Diagnostics Not specifically aligned to course SLO’s (Either too general or too specific) Mining data from these diagnostics would be too difficult Concerns We wanted to assess SLO skill sets but we didn’t want to have these assessments affect the students’ grades. We didn’t want to have the SLO assessments be used to evaluate the teachers in any way. Assessing SLO’s at the end of a course wouldn’t provide the best information on students who entered through the placement process. Our Goal: Sustainable Continuous Quality Improvement Measure Student Learning Outcomes Ramping Up Define metrics to measure SLO’s Develop student learning outcomes Experiment with various measures and discuss observations Creating Departmental Diagnostic Writing Assessments Measure expressive writing skills A holistically scored writing sample Measure meta-linguistic abilities An objective multiple choice assessment Analyze SLO’s to Identify Metrics Measurable elements of writing include: ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ Syntactic structures and variety Accuracy of verb tense form and use Appropriate word forms Range of vocabulary Rhetorical patterns Development Use of cohesive devices Other …include measures of meta skills Meta-linguistic skills are included in SLO’s Examples: ◦ ◦ ◦ ◦ ◦ Identify X Label X Analyze X Recognize X Differentiate between X and Y An objective instrument can provide discrete measures of each of these Build a Scoring Guide CRC chose an ETS style holistic scoring guide model with 5 levels: ◦ ◦ ◦ ◦ ◦ Insufficiently prepared = 1 Under-prepared = 2 Fairly prepared = 3 Well prepared for the level = 4 Demonstrates skill above level = 5 …write descriptors for each level Start with the SLO’s – write a descriptive narrative to match expressive skills they describe This will be the score 4 – “well prepared for the course” To write descriptors for scores 1 – 3 and for score 5, examine the sample papers already marked by faculty Select A Writing Prompt For the lowest levels, we chose a picture and guiding questions: “Look at the picture. Who is the man? Where is he? What happened? How does he feel? Why does he feel that way? Where is he going? What will happen next? Write the story!” For intermediate SLO’s and beyond, we selected this prompt: “Some people like to live in big cities, others do not. Tell where you prefer to live and explain why.” Create an Objective Instrument Align assessment items directly to SLO Make more than one item to measure each skill…vary the contexts Use multiple choice format & SCANTRON© Check to see that the terms used are consistent with what students have encountered in classes and texts Collaborate to check for validity, bias, and clarity Scrutinize the Process Validity Check According to Test Better,Teach Better by W. James Popham, content-related evidence of validity can be determined by relying on “human judgement” A team of expert colleagues agrees that each test item measures the skill it was designed to reflect. The team must also agree that the test items are accessible and free of bias. Check for Clarity Conduct item-by-item study of the test Be sure that the language level matches the texts Be sure that vocabulary is appropriate and standard Be sure the content doesn’t unfairly affect a student’s ability to respond—it is not biased. Check for Bias Content Bias ◦ Gender ◦ Culture ◦ Experiential Linguistic Ambiguity ◦ Vague wording ◦ Confusing grammar ◦ Low-frequency vocabulary Difficulty Create a Score Tally in Excel student id Verb Identifyin Identifyin Verb Tenses in g g Parts of Tense Question SLO #2 Sentence Speech Forms s SLO SLO #1 Logic Parts SLO #6 SLO #7 #2 SLO's 1 - Evidence and SLO #3 SLO #5 Items 8- Items 15- Items 214 Holistic of Develop SLO #4 Mechanic Items 1-7 14 20 24 Score Planning ment Syntax s (7Q's) (7 Q's) (6 Q's) (4 Q's) 475549 4 2 3 2 2 5 4 1 2 485646 2 2 2 1 2 4 4 4 3 922766 3 2 2 2 2 4 4 6 4 987991 3 2 2 2 1 5 6 5 3 1020104 3 2 3 3 2 2 3 3 2 1036656 2 2 1 1 1 4 3 4 2 …include Columns for ◦ ◦ ◦ ◦ Student ID #’s Holistic Writing Score Scores for component skills evaluation Scores from each section of the objective assessment Analyze Data Collect descriptive statistics Make correlations Focus on evidence of student learning Provide data sets to all teachers for use in self-studies and program review Use the assessments over several semesters to enable longitudinal study of a significant sample of student work Continue the Dialog Use collective observations to suggest and implement changes to ◦ Curriculum ◦ Methods ◦ Materials Select appropriate data to support resource requests like facilities and FTE Have faculty experienced in the process mentor and norm their colleagues Other than ACCJC Standards, Why Assess SLO’s? Find out what you’re “teaching” that the students aren’t learning. Move faculty dialogue away from anecdotal discussion Confirm intuition (or not) Gather substantive support for resource requests Example SLO’s SLO 1: Identify and apply basic essay components SLO 2: Employ the writing process SLO 3: Interpret and respond to writing prompts SLO 4: Develop paragraphs appropriate to rhetorical mode SLO 5: Identify and correct grammatical errors through editing SLO 6: Write simple, compound, and complex sentences with correct punctuation SLO 7: Identify and correctly use grammar structures and word forms Example of an SLO with Metrics SLO 7: IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS identify and use all basic verb tenses with very rare errors in form (simple and progressive aspect of present, past and future) identify and use some perfect verb tenses with occasional error in form and usage (present perfect, present perfect progressive, past perfect) identify and use singular, plural and non-count forms and appropriate quantifiers and articles without errors in form identify parts of speech and use correct word forms in writing W 50 SLO Attainment Fall ‘09 average percent success 80 70 60 50 40 30 20 10 0 SLO 1 SLO 2 SLO 3 SLO 4 SLO 5 SLO 6 SLO 7 W 50 SLO#7 IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS 10 9 8 7 6 Total Possible 5 Average 4 Best Score 3 Lowest Score 2 1 0 Holistic Score Grammar Correcting Errors Convert to a Comparable Form Do some simple arithmetic: ◦ Divide the total possible into 100 100÷5 = 20 ◦ Multiply the answer by the average score 20 x 2.18 = 43.6% W 50 SLO#7 IDENTIFY AND USE GRAMMAR STRUCTURES AND WORD FORMS Student Success 80 70 60 50 40 Student Success 30 20 10 0 Holistic Score Grammar in Writing Correcting Errors on objective test Observations from the Data Neither the students’ ability to correct grammar errors on the objective assessment nor the teachers’ evaluation of students’ grammar in the writing sample were significantly correlated with their success on the holistically graded writing sample. There was no significant correlation between students writing sample and the objective assessment. Significant Correlations The students holistic writing scores were directly correlated with the score the rater gave them on their essay organization. The students holistic writing scores were directly correlated with the score the rater gave them on their essay development. Thank you for your participation!