Developing an Assessment Plan Owens Community College Assessment Day Workshops November 13-14, 2009 Anne Fulkerson, Ph.D. Institutional Research Agenda What is assessment? Why assess? Types of assessment Basic steps in the assessment planning process Identifying outcomes Defining measures Evaluating measures Other important considerations OCC assessment plan template & rubric Sample assessment plan How IR can help What is Assessment? “Assessment is the ongoing process of: Establishing clear, measurable expected outcomes of student learning. Ensuring that students have sufficient opportunities to achieve those outcomes. Systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches our expectations. Using the resulting information to understand and improve student learning.” Suskie (2004), pg. 3 Why Assess? Demonstrate effectiveness Improve student learning Establish Learning Outcomes Provide Learning Opportunities Implement Change Measure Learning Adapted from Suskie (2004) and Maki (2004) Types of Assessment Formative—assessment that takes place while learning is in progress (or while a new program is being developed) in order to provide feedback for improvement Summative—assessment that takes place at the end of a course or program to document that learning has occurred or to make judgments about the efficacy of a program Establish Mission Statement Identify Student Learning Outcomes Student Learning Outcomes: Specifiable activities, products, skills, abilities, behaviors, attitudes, or pieces of knowledge that students attain as a result of their involvement in a particular set of educational experiences. Represent the most important competencies that all students should possess Do not reflect ALL learning that might occur Often aligned with accrediting body standards No magic number, but keep them manageable Expressing Student Learning Outcomes Too vague: Students will demonstrate information literacy skills Too specific: Students will be able to use institutional online services to retrieve information Better: Students will locate information and evaluate it critically for its validity and appropriateness Expressing Student Learning Outcomes Fuzzy Terms: Know Understand Become aware of Appreciate Think critically Demonstrate knowledge Learn Write proficiently Action Words: Remember Identify Perform Create Define Summarize Explain Discuss Describe Solve Find/Locate Analyze Evaluate Apply Debate/Argue Synthesize Integrate Research Choose Construct Design Develop Organize Use Thinking Skills Bloom’s Taxonomy (Revised) Based on an adaptation of Anderson & Krathwohl (2001) http://uwf.edu/cutla/assessstudent.cfm Sample Student Learning Outcomes Not Measurable: Measurable: Recognize a need for lifelong learning and plan for personal and professional growth Describe and adopt a plan for ongoing professional development and lifelong learning Demonstrate an historical knowledge of the symphonic, string orchestra, and chamber ensemble repertoire Describe the historical development of the symphonic, string orchestra, and chamber ensemble repertoire http://uwf.edu/cutla/assessstudent.cfm Identify Opportunities for Students to Achieve Each Learning Outcome Course / Activity / Experience Learning Outcome 1. 2. 3. 4. 5. Define Measures for Each Outcome Types of Measures: Quantitative—numeric (e.g., test scores, placement rates, GPA, structured surveys) Qualitative—described by words rather than numbers (e.g., interviews, focus groups, observations, rubrics) Direct—require students to display knowledge or skill (e.g., tests, performances, assignments) Indirect—second-hand evidence (e.g., surveys, course evaluations, journal reflections) Word of Caution Don’t measure everything, just because you can Pick the best measures Keep it manageable Determine How Measures will be Evaluated Criterion-based benchmarks—compares student performance with a pre-established standard. Norm-referenced benchmarks—compares student performance with a standardized norm or a group of peers. Value-added approach—compares student performance at two points in time to see if they have improved. Longitudinal/historical approach—compares current students with prior students. Other Things to Consider What resources are needed? Who’s responsible for collecting and analyzing the data? Build support through participation in the planning process Assessment plans are perpetual drafts OCC Assessment Plan Template https://intranet.owens.edu/committees/outcomes/index.html Program Mission: Program Vision: Program Level Student Learning Outcomes Program Student Learning Outcomes # 1: Where taught? Where Direct Where measured? measure(s) 1. 1. 1. 2. 2. 2. Program SLO # 2. Where taught? Where measured? 1. 1. 2. 2. Program SLO # 3 Where taught? Where measured? 1. 1. 2. 2. Program SLO # 4 Where taught? Where measured? 1. 1. 2. 2. Program SLO # 5 Where taught? Where measured? 1. 1. 2. 2. Indirect measure(s) Resources needed Person(s) responsible 1. 2. Schedule Benchmark(s) Direct measure(s) 1. 2. Indirect measurer(s) Resources needed Person(s) responsible 1. 2. Schedule Benchmark(s) Direct measure(s) 1. 2. Indirect measurer(s) Resources needed Person(s) responsible 1. 2. Schedule Benchmark(s) Direct measurer(s) 1. 2. Indirect measurer(s) Resources needed Person(s) responsible 1. 2. Schedule Benchmarks(s) Direct measurer(s) 1. 2. Indirect measurer(s) Resources needed Person(s) responsible 1. 2. Schedule Benchmark(s) OCC Assessment Plan & Report Rubric https://intranet.owens.edu/committees/outcomes/index.html Elements (1) Program goals and intended student learning outcomes (Appropriateness) Developing Lack of or incomplete inclusion of some established criteria. Established Describe how the goals & SLO align with College &/or School mission & vision. Describe how the goals & SLO link to accreditation standards. if app Describe how the goals & SLO link to industry and/or academic standards. Exemplary Provides evidence supporting the statements in “Established” category. Evidence is appropriate for the measurement of student learning (Evidence) Lack of or incomplete inclusion of some established criteria. Evidence that the measurement instrument is aligned with goals and/or SLO. Benchmarks in place and evaluated regularly Artifacts when utilized are evaluated with rubrics, when appropriate. At least one direct & one indirect measure as appropriate. Normed evidence Provide information on validity & reliability of instrumentation. Rubrics are constructed and/or agreed upon by most stakeholders. Multiple direct & indirect measures. Individuals responsible for delivering the program engage in systematic analysis of evidence. (Analysis of Evidence & Shared Responsibility) Lack of or incomplete inclusion of some established criteria. Describe how the data was analyzed. Provide evidence that appropriate stakeholders were involved in the analysis of data. Describe how the data was used to make decisions about program improvement. Describe action steps needed to improve data collection and/or program. Provide a summary of the analysis with references to data. Provide evidence that the previous plan of action was implemented. Provide evidence that the previous action plan was assessed. Provide results of the implementation of the action plan. Results are shared with essential stakeholders Lack of or incomplete inclusion of some established criteria. Evidence that the report was shared with all faculty and advisory committee members where applicable. Evidence that the report was shared with additional stakeholder groups. (Shared Responsibility) Sample Assessment Plans Owens— https://www.owens.edu/portrait/index.html College of Charleston, SC— http://spinner.cofc.edu/~oap/docs.html Supplemental Instruction How IR Can Help Data support and analysis Assessment consulting Survey development Rubric development Identifying appropriate metrics Identifying existing data sources References Maki, P. (2004). Assessing for Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing: Herndon, VA. Palomba, C. & Banta, T.W. (1999). Assessment Essentials. Planning, Implementing, and Improving Assessment in Higher Education. John Wiley & Sons: San Francisco, CA. Suskie, L. (2004). Assessing Student Learning: A Common Sense Guide. Jossey-Bass: San Francisco, CA. Owens Community College, Student Learning Assessment Committee (2009). Building your SLAC assessment plan: Instructions for the SLAC assessment plan template version #1: intranet.owens.edu/committees/outcomes/index.html. University of West Florida, Center for University Teaching, Learning, & Assessment. http://uwf.edu/cutla/Assessres.cfm.