Opportunities for Postsecondary Success Project Center for Community Partnerships Department of Occupational Therapy Colorado State University Goal Attainment Scaling AHEAD 2012 Conference 7/11/12 Cathy Schelly, M.Ed., OTR Assistant Professor Director, Center for Community Partnerships Patti Davies, PhD., OTR Professor Research Director Opportunities for Postsecondary Success (OPS) Implementation site: Center for Community Partnerships (CCP) OPS Students: ◦ CSU & Front Range Community College students with Asperger’s/ASD, TBI, ID ◦ Transitioning youth with ID from local school district (partnership with Project Search) ◦ Young adults with ID seeking community employment Establishing measurable goals with participants: Goal Attainment Scaling Goal Attainment Scaling Goal attainment scaling (GAS): an individualized approach for measuring the achievement of goals (King et al., 1999). Originally developed to assess adults in a community mental health setting, but has since been applied to numerous practice areas, including education, health, and social work (MacKay, Somerville, & Lundie, 1996). Original goal attainment scaling method The original GAS scale uses a 5-point scale, ranging from -2 to +2, with zero representing the expected level of performance after intervention. Levels Kiresuk, Smith, & Cardillo (1994) -2 Much less than expected outcome -1 Somewhat less than expected outcome 0 Projected level of performance +1 Somewhat more than expected outcome +2 Much more than expected outcome Alternative methods Levels King et al. (1999) -2 Baseline -1 Less than expected 0 Expected level +1 Greater than expected +2 Much greater than expected Levels Steenbeek et al. (2007) +2 A lot more +1 A little more 0 As expected -1 Partially achieved -2 Same as baseline -3 Worse Levels 1 No change 2 Small improvement 3 Interest in goal 4 Takes some action towards goal 5 Maintains new behavior 6 Committed to achieving goal 7 Achieves goal OPS Scale Numerical Level of Goal Attainment I lost ground. 0 Where I am right now. (Baseline) 1 I’m getting close to my goal. 2 I reached my goal! 3 I did even better than expected! 4 Descriptive Level of Goal Attainment OPS Goal Attainment Scaling GOAL: Implement effective study strategies. Numerical Level of Goal Attainment Descriptive Level of Goal Attainment I lost ground. Not willing or interested in discussing/considering effective study strategies. 0 Where I am right now. 1 Willing to discuss/consider potential effective study strategies but not implementing any strategies. I’m getting close to my goal. 2 Implementing effective study strategies with most difficult course, with cueing and reminders. I reached my goal! Implementing effective study strategies with all courses, with cueing and reminders. 3 I did even better than expected! 4 Independently implementing effective study strategies with all courses. Applying GAS in practice 1. 2. 3. 4. 5. 6. 7. Identify the overall objective. Identify the specific problem area(s) that the individual wants/needs to address. Identify the behaviors or events that will indicate improvement in each area. Determine the methodology to collect the desired information. Identify outcomes for scaling. Determine the individual’s current status (baseline). Determine how progress will be documented. What is the overall goal? Setting your expected level of outcome or the desired goal: ◦ For example, you and the student may agree that developing self-advocacy skills is the overall program goal. ◦ Document the student’s current status (baseline). ◦ Establish incremental changes in behavior from baseline to depict improvements or ‘losing ground’ as student pursues goal. What are the specific problem areas? Prioritize areas of concern and formulate SMART goals. Prioritized goals are given a weight, which is used to convert the scale score into a standard score. What are SMART goals? Specific= a specific goal has a much greater chance of being achieved. It is the who, what, where, when, which, and why of the goal. Measurable=establish concrete criteria for measuring progress. Attainable=help client’ set goals that are meaningful to them and that they want to achieve. Relevant=a goal must represent an objective toward which you are both willing and able to work. Time-specific=a goal should be grounded within a time-frame. Tips for scaling goals… Select the expected level of performance. Identify the least favorable outcome and the most favorable outcome. Identify the intermediate levels of performance. Set SMART goals. Develop scale with equally interval levels. Important considerations Where is my client’s current level of performance (baseline)? When/how often should I follow-up to determine and document client’s change in status? What behaviors and events will indicate improvement? Create a scale that will be useful for measuring changes in performance. For the student working on self-advocacy skills, this may involve creating a scale based on the number of times the student attends his/her professor’s office hours. For student who is wanting to make friends in the residence hall (RH), this may involve creating a scale based on the number of RH activities he participates in. GAS Goal: Recognize and address issues that result in lack of persistence spring semester. Numerical Level of Goal Attainment Descriptive Level of Goal Attainment I lost ground. Withdraws from all courses spring semester with no warning, no support. 0 Where I am right now. (Baseline) 1 Planning on withdrawing from challenging courses spring semester – seeking support but unwilling to discuss issues that are causing urge to withdraw. I’m getting close to my goal. 2 Willing to discuss issues that cause the urge to withdraw from challenging courses. I reached my goal! Recognizes and begins addressing issues that are causing urge to withdraw from challenging courses. 3 I did even better than expected! 4 Proactively addresses issues that cause urge to withdraw from challenging courses and persists with all courses. GAS Goal: Implement study strategies. Numerical Level of Goal Attainment Descriptive Level of Goal Attainment I lost ground. Not willing to develop study strategies. 0 Where I am right now. (Baseline) 1 Not participating in Academic Advancement Center (AAC) to learn study strategies and not using any study strategies, but willing to consider development of strategies. I’m getting close to my goal. 2 Attending AAC study strategy sessions to learn about and develop study strategies. I reached my goal! Fully implementing study strategies with OPS support during study sessions. 3 I did even better than expected! 4 Independently implementing study strategies. GAS Goal: Self-advocacy for learning accommodation needs. Numerical Level of Goal Attainment Descriptive Level of Goal Attainment I lost ground. Not willing to speak with professor. 0 Where I am right now. (Baseline) 1 States a willingness to speak with professor regarding accommodation needs, but afraid to follow through. I’m getting close to my goal. 2 Made list of learning accommodation needs in preparation for meeting with with professor(s). I reached my goal! With mentor assistance, set up appointment(s) with professor and kept appointment(s), advocating for learning accommodation needs and with mentor attending meeting(s). 3 I did even better than expected! 4 Independently set up appointment(s) with professors and kept appointments, advocating for learning accommodation needs with behind the scenes encouragement from mentor. GAS Goal: Completes new tasks when assigned jobs are finished. Numerical Level of Goal Attainment Descriptive Level of Goal Attainment I lost ground. Leaves work area when assigned jobs are finished. 0 Where I am right now. (Baseline) 1 Sits down and stops working when assigned jobs are finished, even when additional tasks are not completed. I’m getting close to my goal. 2 Asks job coach what to do next when assigned jobs are finished and works on additional tasks as directed. I reached my goal! Asks supervisor and/or natural support staff if there are additional tasks to be done and then works on these tasks as directed. 3 I did even better than expected! 4 Recognizes additional tasks after assigned jobs are finished and independently completes these tasks. Process we use Develop relationship with student. Connect student with mentor. Gather information re: needs, wants of student, and areas where there are challenges (assessment process). Develop semester goal with student – review with team to ensure that goal is SMART. Enter goal into GAS database. Begin goal attainment activities, documenting incremental changes. At end of semester, regroup and establish new goal(s) for next semester. CCP Assessment Process Functional, community-referenced process ◦ Identify strengths, challenges, barriers in all domains ◦ Identify support needs ◦ Formulate student/participant driven goal(s) Challenges/Support Needs Evaluation Observed Indicator (actual behavior, action or circumstance) Fatigue (physical Student describes inability to stay focused and and cognitive) awake in afternoon classes. Student complains that fatigue is unrelenting, even after 8 hours of sleep each night. Student states that mornings are best for being alert and focused. Confusion (short-term memory challenges) Student spends unusually long periods of time studying for exams and completing assignments, but complains of inability to remember content even after reading numerous times. Assessment process is ongoing • Assessment is the critical starting point that paves the way to successful planning, educational mentoring, and provision of necessary supports that facilitate success. • Functional assessment continues after the student is working towards goals to continually guide planning, solve problems, improve academic performance and, ultimately, persistence, retention and graduation. • Same guidelines apply for students pursuing employment/careers. Canadian Occupational Performance Measure (COPM) Under consideration, to complement GAS. Assesses individual’s perceived occupational performance in life areas. Process: ◦ Conduct semi-structured interview to identify wants/needs of client. ◦ Based on wants/needs, create goals. ◦ Supplement interview information through direct observation, environmental assessment, or standardized tests. Benefits of COPM Measures client’s perceived progress in goal attainment between pre-COPM, intervention and post-COPM (designed as outcome measure). Easy to administer ~20 minutes. Demonstrated reliability and validity. Similar to GAS, useful with a wide variety of clients in many different settings. Can weight and prioritize goals (again, consistent with GAS). Audience Discussion What type of assessment process(es) are you using? ◦ What are the advantages of your assessment process? ◦ What are the disadvantages, if any, of your assessment process? What population do you serve? Practice session on writing goals Review GAS worksheets. Develop goals as per worksheet, OR, for student you are working with. ◦ Connect goals to domain areas. ◦ Develop SMART goals. Discussion ◦ Actual goals that we established with our students/participants How will data be collected? What process will your group implement to collect and analyze data? How will information be collected and who will collect it? Role of the Transition Coordinator Role of the Mentee Role of the Mentor Our GAS Research Exciting results from pilot study fall 2011 and spring 2012 Noted progress in following areas: ◦ ◦ ◦ ◦ ◦ ◦ ◦ Academics Self-advocacy Socialization Persistence Retention Graduation Employment Preliminary Data Participants - total 23 ◦ 12 students at CSU ◦ 5 students at FRCC ◦ 6 students enrolled in Project Search Number of goals written – total 43 ◦ 15 in Fall semester 2011 ◦ 28 in Spring semester 2012 Students enrolled in Project Search 6 students Goals written for a 3 month internship 8 goals written ◦ 1.3 goals on average per student 83% of the students made steady improvement on their goals during their internships 16% of the students displayed a decline in their GAS score at the end of the internship 100% of students were hired after completing their internships Students in College (FRCC) and University (CSU) 17 students Goals written for a semester 35 goals written ◦ 1.8 goals on average per student 47% of the students made steady improvement on their goals across the semester 24% of the students displayed a decline in their GAS score at the end of the semester Changes in Goal Attainment Scores Before and After Intervention T Score Mean Standard Deviation Pre Intervention 38.12 2.23 Post Intervention 58.90 7.92 Changes in Goal Attainment Scores Before and After Intervention ANOVA Statistic: F(1,20) = 136.17, p < .0005 Effect Size: Partial η² = .87 Converting Goal Attainment Scales Scores to T Scores Sum the GAS score for the goal(s) ◦ If a participant only has one goal the sum will be the score for that one goal. ◦ If a participant has more than one goal the score for each goal will be added together. Convert the summed score to a T Score ◦ Formula in Kiresuk & Sherman (1968) More precision ◦ Tables in Kiresuk, Smith, & Cardillo (1994) Less precision Advantages of Using T Scores Helps to equate scores across participants when they have different number of goals. Provides variability to pre-intervention scores which is necessary for conducting statistical analyses. Goals can be Weighted To reflect ◦ Relative value of each goal based on intervention outcomes ◦ Relative importance of each goal based on participants’ perspectives ◦ Relative to the difficulty of the goal Weights do not need to sum to a particular fixed total Examples ◦ 1, 2, 3 or 2, 4, 6 or 10, 20, 30 Equal weighting of goals lose little information Goal Attainment Scaling as an Intervention Tool Weighting and selecting goals based on client’s desires Setting goals in collaboration with client Using improvement on GAS scores as motivation for the client to work hard Can be used in combination with the COPM ◦ Determine goal areas ◦ Weight goals Goal Attainment Scaling as an Research Tool The evaluation and establishment of goals may be conducted by someone different than the person providing the intervention. Follow-up assessment on goals may be conducted by someone other than the person providing the intervention. Consider comparing intervention programs or using a control group. Consider using other evaluation tools in addition to GAS. Goal: University Graduation Success! Goal: College Graduation Success! Advantages of using GAS with the OPS project Provides a quantitative measure of outcome. Can be used to compare a student’s progress over time. Can be used to compare performances across clients in the same program but who have different skills, challenges (Ottenbacher & Cusick, 1989). Allows for a collaborative approach-working with the student to set realistic goals. Provides flexibility for measuring diverse outcomes (Brown, 2009). Limitations of GAS Scaling can be a time-consuming process. Biases can occur in goal setting, scaling, and rating. Temptation to modify goals throughout the course of intervention. NOTE: Good goals are really hard to write!! References King, G.A., McDougal, J., Palisano, R.J., Gritzan, J., & Tucker, M.A. (1999). Goal attainment scaling: its use in evaluating pediatric therapy programs. Physical and Occupational Therapy in Pediatrics, p. 31-52. Kiresuk, T. & Sherman, R. (1968). Goal attainment scaling: A general method of evaluating comprehensive mental health programs. Community Mental Health Journal, 4 (6), 443-453. Kiresuk, T.J., Smith, A., & Cardillo, J.E. (1994). Goal Attainment Scaling: Application,Theory, & Measurement. Hilldale, N.J.: Lawrence Erlbaum Associates. Mailloux, Z., May-Benson, T.A., Summers, C.A., Miller, L.J., Brett-Green, B., Burke, J.P., et al. (2007).The Issue Is-Goal Attainment Scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61, 254-259. McKay, G., Somerville, W., & Lundie, J. (1996). Reflections on goal attainment scaling (GAS): cautionary notes and proposals for development. Education Research, 38,2, 161-172. Ottenbacher, K.J. & Cusick, A. (1990) Goal attainment scaling as a method of clinical service evaluation. American Journal of Occupational Therapy, 44, 6, 519-525. Steenbeek, D., Ketelaar, M., Galama, K., & Gorter, J.W. (2007). Goal attainment scaling in pediatric rehabilitation: a critical review of the literature. Dev Med Child Neurol, 49, 550-556. Turner-Stokes, L. & Williams, H. (2010). Goal attainment scaling: a direct comparison of alternative rating methods. Journal of Clinical Rehabilitation, 24, 66-73.