SOCIAL WORK EDUCATION ASSESSMENT PROJECT (SWEAP): Responding to the Challenges of Assessment Pre-Conference Workshop BPD Annual Conference Louisville, KY March 19, 2014 • WORKSHOP OUTLINE Introductions o o o • Overview of Accreditation o o o o • EPAS 2008 (&2015) Competencies Characteristic Knowledge, Values & Skills Practice Behaviors SWEAP Instruments o o o o o o • History of Accreditation Direct vs. Indirect Measures Implicit Curriculum Multiple Measures Organizational Framework for Program Assessment o o o o • Introducing SWEAP /Goodbye BEAP The SWEAP Team Workshop Participants Entrance Exit Alumni/Graduate Employer Curriculum Instrument (FCAI) Field Instrument (FPPAI) BREAK WORKSHOP OUTLINE (CONTINUED) • Unique Benefits of SWEAP • Tying SWEAP to Program Assessment o o o o Linking SWEAP to EPAS Matrix Program Integration Example Using SWEAP beyond EPAS • Sharing Assessment Ideas o Group Work • Navigating SWEAP o o o o o Website Ordering Raw Data Policy Processing Reports • Questions INTRODUCING SWEAP • Was BEAP • Now SWEAP o Not just for undergraduate programs anymore o Can assess foundation year for graduate programs as well o Can be modified for program specific advanced year assessment THE SWEAP TEAM Vicky Buchan Colorado State University [email protected] Kathryn Krase LIU Brooklyn [email protected] Brian Christenson Lewis-Clark State College [email protected] Phil Ng [email protected] Tobi DeLong Hamilton Lewis-Clark State College, CDA [email protected] Ruth Gerritsen-McKane University of Utah [email protected] Sarah Jackman [email protected] Patrick Panos University of Utah [email protected] Roy (Butch) Rodenhiser Boise State University [email protected] INTRODUCING WORKSHOP PARTICIPANTS • • • • Your Name Your Program Your Role Have you used SWEAP? o If so, which instrument(s)? OVERVIEW OF ACCREDITATION: HISTORY OF CSWE ACCREDITATION o o o o What was the process like before BEAP? How has it changed over time? How do changes to EPAS impact it? Move to competency based educational standards in EPAS 2008 OVERVIEW OF ACCREDITATION: DIRECT VS. INDIRECT MEASURES o Direct Measures • “Student products or performances that demonstrate that specific learning has taken place.” • Examples (SWEAP and Non-SWEAP) o Indirect Measures • “May imply that learning has taken place (e.g., student perceptions of learning) but do not specifically demonstrate that learning or skill.” • Examples (SWEAP and Non-SWEAP) o Which do you need and why? OVERVIEW OF ACCREDITATION: IMPLICIT CURRICULUM o What is Implicit Curriculum? • “Educational environment in which the explicit curriculum is presented. It is composed of the following elements: the program’s commitment to diversity; admissions policies and procedures; advisement, retention, and termination policies; student participation in governance; faculty; administrative structure; and resources.” (EPAS, 2008) • How do you measure it? • Why should you measure it? OVERVIEW OF ACCREDITATION: MULTIPLE MEASURES o The Importance of Multiple Measures o SWEAP alone is not enough ORGANIZATIONAL FRAMEWORK FOR PROGRAM ASSESSMENT EPAS 2008 (& 2015) • Working under EPAS 2008 • EPAS 2015 underway… expected changes o Fewer competencies o Fewer practice behaviors ORGANIZATIONAL FRAMEWORK FOR PROGRAM ASSESSMENT COMPETENCIES • EPAS 2.1—Core Competencies o Competency-based education o Outcome performance approach to curriculum design. o Measurable practice behaviors comprised of knowledge, values, & skills. o Need to demonstrate integration & application competencies in practice with individuals, families, groups, organizations, and communities. o 10 competencies listed along with description of characteristic knowledge, values, skills, & practice behaviors that may be used to operationalize the curriculum and assessment methods. o Programs may add competencies consistent with their missions and goals. ORGANIZATIONAL FRAMEWORK FOR PROGRAM ASSESSMENT COMPETENCIES • 2.1.1—Identify as a professional social worker and conduct oneself accordingly. • 2.1.2—Apply social work ethical principles to guide professional practice. • 2.1.3—Apply critical thinking to inform and communicate professional judgments. • 2.1.4—Engage diversity and difference in practice. • 2.1.5—Advance human rights and social and economic justice. • 2.1.6—Engage in research-informed practice and practiceinformed research. • 2.1.7—Apply knowledge of human behavior and the social environment. • 2.1.8—Engage in policy practice to advance social and economic well-being and to deliver effective social work services. • 2.1.9—Respond to contexts that shape practice. • 2.1.10— Engage, assess, intervene, and evaluate with individuals, families, groups, organizations, and communities. ORGANIZATIONAL FRAMEWORK FOR PROGRAM ASSESSMENT: CHARACTERISTIC KNOWLEDGE, VALUES & SKILLS • Current focus on measuring practice behaviors. • Don’t forget about knowledge, values & skills. ORGANIZATIONAL FRAMEWORK FOR PROGRAM ASSESSMENT PRACTICE BEHAVIORS • Multiple practice behaviors per competency • Each practice behavior MUST be measured for selfstudy/reaccreditation • TWO measures required for each practice behavior o At least one measure must be DIRECT SWEAP INSTRUMENTS WHAT ARE THEY AND HOW DO I USE THEM? • • • • • Entrance Exit Alumni/Graduate & Employer Curriculum Instrument (FCAI) Field Instrument (FPPAI) ENTRANCE PURPOSE • Provides demographic profile of entering students. • Completed at time of entrance into the program (Program Defined). • Provides overview of financial resources students are using or plan to utilize. • Provides employment status & background information regarding both volunteer and paid human service experience. • Helps track planned or unplanned changes in the profile of students in the program. • Evaluate impact of policy changes, such as admissions procedures, over time. ENTRANCE QUESTIONS o o o o o o o o o o o o o Student tracking: ID number & Survey completion date Gender Year in school Overall GPA & GPA in major & Highest possible GPA at school Length of current social work-related work experience (volunteer & paid) Citizenship/ length of residence in USA Employment plans during social work education Hours per week expected to work during education Sources of financial aid expected Language fluency Expected date of graduation Race/ Ethnicity Disabilities/Accommodation EXIT PURPOSE • Completed by students just prior to graduation. o Often administered in field seminar or capstone seminar. • Feedback from students about their experiences while in the program. • Addresses: o Evaluation of curriculum objectives based on EPAS. o Post-graduate plans, related to both employment and graduate education, are addressed. • Collects demographic information to compare with the entrance profile. EXIT QUESTIONS • Educational Experience o Including implicit curriculum assessment • Current Employment • Employment-seeking activities • Current & anticipated Social Work Employment o Primary function & major roles • Post Graduate Educational Plans EXIT QUESTIONS • Students Evaluate how well program prepared them to perform practice behaviors • Professional Activities o Use of research techniques to evaluate client progress & Use of program evaluation methodology • Personal Demographic Information o Gender, Citizenship, Language fluency, & Disabilities ALUMNI/GRADUATE PURPOSE • Intended for completion two years after graduation o Standardized timing for administration is essential to create a reliable dataset for comparison over time. • Alumni evaluate how well program prepared them for professional practice. • Alumni employed in social work and those not employed in social work are surveyed. • Also gathers information on current employment, professional development activities, and plans/ accomplishments related to further education. ALUMNI/GRADUATE QUESTIONS • Current Employment • Current Social Work Employment • Evaluation of preparation by the Program in the 10 EPAS competency areas (using Likert-type scale) • Educational Activities • Professional Activities • Demographics EMPLOYER PURPOSE • Intended for completion two years after graduation • Addresses both accreditation and university concern for feedback from the practice community. • Measures graduate’s preparation for practice based on supervisor’s assessment. • Alumni/ae request employer complete the survey: o Addresses primary concern of confidentiality. o Use of student identified allows connection to other instruments. EMPLOYER QUESTIONS • Educational background of supervisor/employer • Twelve (12) items which evaluate alumni/ae proficiency in all EPAS competencies. BREAK TIME Foundation Curriculum Assessment Instrument CURRICULUM INSTRUMENT (FCAI) PURPOSE 1. Provides Pre/Post test in seven major curricular areas of the foundation year. 2. Provides a direct measure to assist programs with evaluation of their curriculum. 3. Assists with identification of curricular areas that may need attention. 4. Provides national comparative data. Curricular Components Curriculum Area Number of Questions Practice 13 Human Behavior & Social Environment 10 Policy 9 Research 9 Ethics and values 8 Diversity 8 Social and Economic Justice 7 Sample HBSE Question • The concept “person-in-environment” includes which of the following: a. Clients are influenced by their environment b. Clients influence their environment c. Behavior is understood in the context of one’s environment d. All of the above Sample Practice Question • Determining progress toward goal achievement is one facet of the _____ stage. o a. Engagement o b. Evaluation o c. Assessment o d. Planning Overview of FCAI Respondents Instrument 2011 2012 2013 Total Entrance 1,986 2,009 8,432 12,427 964 1,714 5,369 8,047 2,950 3,723 13,801 20,474 Exit Totals Reliability Testing • Version 9 – Tested in two junior practice classes – Students tested twice, 2 weeks apart – Pearson’s correlation coefficient • r = .86 Item difficulty index • Overall difficulty or average should be around .5 (Cohen & Swerdlik, 2005) • FCAI = .523 (n=415) • “This is a very good difficulty level for the test. Not likely to misrepresent the knowledge level of test takers”. Reliability & Effect Size • Cronbach’s alpha = .784 • Effect Size d = 6.87 Current Data 1/13 to 9/13 • Number of Schools using the FCAI in 2013 110 • Number of Respondents in 2013: o Pre-test: 8432 o Post-test: 5369 School X / National Comparison PRE TEST POST TEST SD SIG School X 31.24 (48.81%) 39.35 (61.48%) 6.19 .000 National 30.38 (48.10%) 37.54 (58.60%) 7.59 .000 Overall Scores Pre-Post BSW Entering BSW Exiting (N = 8432) (N =5369) Raw Score 32.8 40.05 Mean % correct 51.2% / 64 62.5%/64 T-test 380.2 S.D. 7. 8 Sig .000 BSW Student Scores by Curricular Area One Program FCAI Entrance & Exit Curricular Area Question Suggested Competency Correct responses Correct and % Entrance responses and % Exit Practice Question 1 2.1.1C 14/64 21.8% 39/64 61% Question 2 2.1.10A 46/64 71.8% 60/64 93.7% Question 4 2.1.10H 21/64 32.8% 43/64 67.2% Expansion beyond BSW • Based upon CSWE assertions related to educational levels in social work education, we expanded testing to three additional groups: o MSW foundation students: • entering • exiting o Advanced standing students: • entering Points to keep in Mind about the FACI 1. Purpose of this instrument: to review and improve curriculum 2. Program will want to “monitor” scores over several years (or several cohorts) for trends. 3. The FCAI can be considered a measure of “value added” from program entry to exit. 4. Benchmarks: can be set two ways, a. by competency b. overall score Field Practicum/Placement Assessment Instrument (FPPAI) FIELD PRACTICUM/PLACEMENT ASSESSMENT INSTRUMENT (FPPAI) • Responds to need for a standardized field/practicum assessment instrument that measures student achievement of practice behaviors. FIELD INSTRUMENT (FPPAI) PILOTING PHASE • • • • Initial Piloting for BSW in May 2008 Second Pilot in Fall 2008 /Spring 2009 Third Pilot in Fall 2009 Reliability Analysis o Chronbach’s Alpha of 0.91 or higher in each practice behavior • Full implementation in BSW: Fall 2010 • Piloting in MSW: Spring-Fall 2012 • Full implementation in MSW (Foundation): Fall 2013 FIELD INSTRUMENT (FPPAI) METHODOLOGY • 58 Likert Scale questions measuring practice behaviors linked to the EPAS 2008 competencies. • Qualitative feedback form for each domain available for program use. • Available online and in print format. • Individual program outcomes report with national comparisons available. • Individual program outcomes report with national comparisons for EPAS 2008 Competencies & Practice Behaviors including CSWE benchmark reporting. • Can be used as a final field assessment and midtest/post test design. FIELD INSTRUMENT (FPPAI) SCALE FIELD INSTRUMENT (FPPAI) CURRENT STATUS • • • • 84 programs currently use this instrument More than 3,132 administrations to date National data comparisons are available Cronbach’s Alpha reliability test of internal consistency at midpoint (Pilot and first two years Testing): 0.969 • Cronbach’s Alpha reliability test of internal consistency at final (Pilot and first two years testing): 0.975 BREAK TIME SWEAP UNIQUE BENEFITS Student demographics Numerous data points for comparison Explicit and Implicit curriculum assessment Doesn’t end at graduation Peer comparison by region, program type, auspice & nationally • …and more • • • • • TYING SWEAP TO PROGRAM ASSESSMENT LINKING SWEAP TO EPAS • All instruments updated to reflect 2008 EPAS • All instruments will be updated and available as soon as 2015 EPAS goes into effect • Competency Matrix for 2008 EPAS(Handout) TYING SWEAP TO PROGRAM ASSESSMENT Educational Policy 2.1.1—Identify as a professional social worker and conduct oneself accordingly Practice Behavior use supervision and consultation Measures SWEAP-Exit SWEAP-FPPAI SWEAP-Employer SWEAP-Alumni TYING SWEAP TO PROGRAM ASSESSMENT: MATRIX Competency 2.1.7—Apply knowledge of human behavior and the social environment.A. utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation; B. critique and apply knowledge to understand person and environment. Alumni Exit Employer FCAI FPPAI TYING SWEAP TO PROGRAM ASSESSMENT PROGRAM INTEGRATION EXAMPLE • Program Integration Example (Handout) TYING SWEAP TO PROGRAM ASSESSMENT USING SWEAP BEYOND EPAS • Curricular Development • Addressing Implicit Curriculum Concerns • Understanding student body o Changes in student body over time SHARING ASSESSMENT IDEAS Break into small groups, tackle the assigned competency discussing the following: How can/do you measure competencies in your programs? Offer SWEAP & NonSWEAP options SHARING ASSESSMENT IDEAS Break into small groups, tackle the assigned competency discussing the following: How can/do we measure Implicit Curriculum? Offer SWEAP & Non-SWEAP Measures NAVIGATING SWEAP THE WEBSITE http://sweap.utah.edu o Navigating the site o Locating the results of your instruments NAVIGATING SWEAP ORDERING o Online through website. o Pay by credit-card, or send check. o Receive email with .pdfs to be printed, or links to online instruments. o Printing .pdfs • Can be double-sided, BUT DO NOT STAPLE For questions about your order or status: [email protected] Online Instruments • All SWEAP instruments are available in electronic format. • eTickets (electronic tickets) are simple to use. You order them via the SWEAP order page on our website, just like you would order instruments to print on paper. • Each eTicket is an individual instrument and cannot be copied. • The simplest way to administer instruments with eTickets is to send an email to each student, field educator or alumnus/a who is expected to complete the instrument. In that email, you provide an individual eTicket link. • The respondent enters the letters and numbers that follow the equal sign (=) on the eTicket link. For instance, to access the email example, the code to enter in the pink box is RITNFN26197 (see next slide). NAVIGATING SWEAP RAW DATA POLICY • We offer return of de-identified data to programs with the following data removed: o o o o o o Year of Birth Day of Birth Parts of Social Security Numbers ZIP code of Employment or Residence. Sex / Gender Ethnic Identity • In addition, specific answers to the FCAI instruments will not be returned, nor specific answers to the Values at Exit or Entrance. These will be reported in an aggregate only. • FPPAI instruments, due to their nature, will return all identifying information. NAVIGATING SWEAP RAW DATA POLICY • There will be a surcharge to utilize this service, it will be a one time charge of $25.00 to have all permissible data returned to the program. NAVIGATING SWEAP PROCESSING • Paper instruments need to be mailed to Utah. o Processing involves scanning forms into system and then calculating data. o Can take up 3-6 weeks, depending on time of year. • Online instruments o Processing is very quick. o Can be days, instead of weeks. o No need to be scanned at our end. NAVIGATING SWEAP REPORTS • Receive email when reports are ready. • Retrieve reports through website. • Can request reports with tailored comparisons. Coming in 2015! • Bridging the gap for direct measures: New direct measures in-line with 2015 EPAS • Outcome Reports on your webpage as required by CSWE • All current instruments updated for EPAS 2015 QUESTIONS?