Florida Interim Assessment Item Bank and Test Platform (FL IBTP) Local Assessment Part 2 February 17 and 18, 2014 – Lake Mary, FL February 20 and 21, 2014 – Tallahassee, FL Learning Goals At the end of this workshop, you will understand how to: • Part One • Create assessment blueprints • Part Two • Build test maps • Part Three • Create and schedule assessment forms • Administer forms • Enter scores for open response items • View assessment data and reports Training Overview Today, participants will work in groups to create portions of a sample blueprint, test map, and assessment. Participants will learn the necessary skills and processes to create blueprints, test maps, and assessments. They will leave with the knowledge, skills, and resources to train district personnel to create assessments. Agenda • • • • Introduction to materials Training overview Demonstration of the assessment development process Guided practice – Creating a blueprint – Creating a test map and searching for items – Creating, scheduling, administering, and scoring an assessment List of Materials • Participants’ Workbook – Tab 1: Blueprints – Tab 2: Test maps and searching for items – Tab 3: Assessment form creation – Tab 4: Blueprint template, sample blueprint, test map template, and sample test map • Job Aids Materials available for download at: http://flassessments.com/html-redirects/IBTP/index.htm Step 1: Create a Blueprint Grade K math fall interim assessment example: Step 2: Create a Test Map The following is an example of the first columns of the grade K mathematics test map: Create a Test Map The following is an example of the last columns of the grade K mathematics test map: Step 3: Create an Assessment Using the test map, create the assessment in the system. Then schedule, administer, and score the assessment. Take an Assessment Now take a three-question test to see how students will see the test in the system. Break into Groups • Groups of five or six PART ONE: CREATING BLUEPRINTS Assessment Blueprints Use a blueprint to see the content and structure of an assessment. The blueprint is based on the standards or benchmarks listed in the course description. Blueprints include the following: • Total number of items • Number of items by standard or benchmark • Percentage of items at each cognitive complexity level (Depth of Knowledge, or DOK level) • Percentage of items at each difficulty level (if item difficulty statistics are available in the FL IBTP) Blueprint Considerations When building an assessment blueprint, consider the following: • • • • • • Purpose of the assessment Overall assessment design Overall time allocated to assessment administration Time considerations for item types DOK levels of individual standards and benchmarks DOK levels of items Step 1: Category Breakdown Before drafting the blueprint, determine the purpose and type of assessment. In this example, create a blueprint for a grade K mathematics interim assessment that will be administered in the fall to gauge how students are progressing towards mastery of the course standards. Based on the instructional time spent on each strand of standards or benchmarks (reporting category), determine the percentage of items by reporting category. Grade K math fall interim assessment example: • Counting and Cardinality = 70–80% • Geometry = 20–30% Step 2: Define the Item Range Next, determine the number and types of items to be included in the assessment. Consider the typical time allocation for each item type when designing the assessment. Item range • Approximate the number of items • Use increments of 5 when determining an item range (i.e., 5–10 items) Approximate Time Allocations Item Type Minutes Allocated Point Value Selected Response (SR) 1–3 minutes 1 Gridded Response (GR) 1–3 minutes 1 Short Response (SHR) 3 minutes 1 Constructed Response (CR) 5 minutes 2 Extended Response (ER) 10–15 minutes 4 Essay Response (ESR) 20–30 minutes 6 Performance Task (PT) Varies 1–10 (depending on complexity and content area) Step 3: Percentage of Items by DOK Level Determine the percentage of items by DOK level. • DOK level: 1, 2, 3, 4 • Should total 100% • Give minimum range of 5–10 percentage points (i.e., 20–30% DOK level 1) Grade K math fall interim assessment example: • DOK level 1 = 50–60% • DOK level 2 = 40–50% • DOK level 3-4 = 10–20% Step 4: Number of Items by DOK Level Multiply the cognitive complexity level percentage by the total number of items to get the item range for each DOK level. Start with the low complexity items. Grade K math fall interim assessment example: 50–60% DOK level 1 x 10 items = 5–6 DOK level 1 items Number of Items by DOK Level Multiply the cognitive complexity level percentage by the total number of items to get the item range for the DOK level 2 items. Grade K math fall interim assessment example: 40–50% DOK level 2 x 10 items = 4–5 DOK level 2 items Number of Items by DOK Level Multiply the cognitive complexity level percentage by the total number of items to get the item range for the DOK level 3 or 4 items. Grade K math fall interim assessment example: 10–20% DOK level 3–4 x 10 items = 1–2 DOK level 3–4 items Number of Items by Reporting Category Multiply the reporting category percentage by the total number of items to get the item range for each reporting category. Grade K math fall interim assessment example: 70–80% Counting and Cardinality x 10 items = 7–8 Counting and Cardinality items 20–30% Geometry x 10 items = 2–3 Geometry items Completed Example Blueprint Grade K math fall interim assessment example: Step 5: Guided Practice Complete the assessment blueprint using a blueprint template such as the one shown here: District Presentation Hands-On Practice: Create a Blueprint Using the course information from www.cpalms.org, work in groups to create an assessment blueprint. Create blueprints for one of the following courses: • Grade 8 American History (Social Studies) • Grade 7 Life Science (Science) Create at least four lines of the blueprint. Hands-On Practice: Create a Blueprint 1. Review the course description on www.cpalms.org. 2. Add the standard or benchmark identifiers to the blueprint template. 3. Based on the instructional time spent on each strand of standards or benchmarks (reporting category), determine the percentage of items by reporting category (use district resources, such as a curriculum map, pacing guide, or scope and sequence). 4. Determine the number and types of items to be included in the assessment. 5. Determine the percentage of items by DOK level. 6. Multiply the cognitive complexity level percentage by the total number of items to get the item range for each DOK level. Share Your Learning • Share what you’ve learned about creating blueprints. • Record notes about the process on your T-chart. PART TWO: BUILDING TEST MAPS What is a test map? A test map provides item-level details so that the assessment developer is able to search for and select specific items to build an assessment. While the test blueprint is a high-level overview of the contents of the assessment, the test map is a list of specific items in the sequence in which they should appear on the assessment. Test Planning • Discuss naming conventions for assessments. For example: Subject Season Year Grade Form Math Spring 2014 Gr K Form 1 Math Spring 2014 Gr K Form 2 • Discuss workflow. Assign responsibility for blueprint and test map creation, test creation, scheduling the test, assigning the test, and monitoring the collection of test results. 32 Test Map Template The following is an example of a test map template: Step 1: Complete the First Columns Using the information from the blueprint, complete the first columns on the test map. These columns are completed without going into the system. • • • • • • • • • Test Name Subject Grade Level or Grade Band Course Code Course Title Test ID (to be completed later) Standards Source Document Standard or Benchmark Code Sequence Example The following is an example of the first columns of the grade K mathematics test map. Step 2: Search for Items Now go into the system to search for items. In the Florida Item Bank and Test Platform, search for the items to be included in the assessment by subject, grade, or standard. Items will then be listed on the test map. Complete the Next Columns Complete these columns by going into the system: • • • • UIN (Name) Test ID Manipulatives (math tools) Item Type Step 3: Locate the Item UIN Locate the Unique Identification Number (UIN) of the item under Properties on the right side of the Item Central page. Step 4: Record the UIN Record the item UIN in the appropriate column on the test map. Step 5: Test ID Column The Test ID column cannot be completed until the system generates the assessment. The system then assigns a test ID number and displays it on the Test Detail page. Be sure to fill in the Test ID column on the test map after the assessment is created. Math Fall 2014 Gr 6 Form 1  Step 6: Manipulatives Indicate whether or not tools will be used to answer this item. Step 7: Item Type Each item lists the item type. Record the abbreviation for the item type on the test map. Step 8: Complete the Last Columns Complete the last six columns of the test map. • Key (correct response) • DOK Level (Webb) • Item Difficulty • Maximum Points • Associated Passage UIN • Passage Type Key For Gridded Response and Selected Response items, record the correct response in the Key column on the test map. For open-ended items, enter “NA” for not applicable. DOK Level The DOK level, or context complexity level, is associated with the standard or benchmark listed in the course description for the CCSS in mathematics and English language arts, and for the NGSSS benchmarks in science. Find the content complexity/depth of knowledge rating in CPALMS. Determine the DOK level of the item and record it in the DOK Level column. Item Difficulty Item difficulty is a statistical measure of the proportion of students who have correctly answered an item. Item difficulty statistics may not be available initially in the FL IBTP system. Maximum Points Determine the maximum points for the item and record that number in the Max Points column. Associated Passage UIN Look for the Passage ID on the right side of each passage and record it on the test map, as appropriate. Passage Type Look for the passage type in the same list and record it on the test map. Repeat the Process Repeat the process for all the items on the assessment until the test map is complete. Guided Practice: Search for Items Use this URL: https://flademo.schoolnet.com To start an item search, roll the cursor over Assessment Admin and select Find an Item, Passage or Rubric. District Presentation Hands-On Practice: Create a Test Map Using the blueprint you created earlier, work in groups to create a test map. Create test maps for one of the following courses: • Grade 8 American History (Social Studies) • Grade 7 Life Science (Science) Create at least four lines of the test map. Hands-On Practice: Create a Test Map 1. 2. 3. 4. 5. 6. 7. 8. 9. Sign in to the training site. Roll the cursor over Assessment Admin and click Find an Item, Passage or Rubric. Use the subject, grade, and standards filters to refine the search. Use Advanced Search to search for multiple items by standard group. When the search is narrowed sufficiently, click View Results to view the items that match the selected criteria. Review the items that resulted from the search. Record the appropriate information found under Properties on the right side of the screen. To change the search criteria, click Open Filters and change the selections. To start a new search, roll the cursor over Assessment Admin and select Find an Item, Passage or Rubric. Share Your Learning • Share what you’ve learned about creating test maps. • Record notes about the process on your T-chart. PART THREE: ASSESSMENTS – CREATING, ADMINISTERING, SCORING, AND REPORTING PART THREE (A): CREATING AND SCHEDULING A FORM/ASSESSMENT Benchmark Assessments In the Florida IBTP system: • In Florida, the term “benchmark” refers to the NGSSS • • learning targets, but in the FL IBTP system, “benchmark assessments” is a generic term that describes all interim assessments not loaded as standardized tests. Tests are created at the district or school level for the purpose of institution-wide data collection. Results appear in the School & District Data and Classrooms modules. Using Assessment Admin • Use Assessment Admin to build tests and monitor the collection of results. • View data and report options in the Classrooms and School & District Data modules. Ways to Create a Test Express test Manual test Create a Test Manually Use the test map to search for and select test items. Building a Test Math Fall 2014 Gr 6 Form 1  Hands-On Practice: Create an Assessment Create a test using the items from the sample test maps using grade 8 American History and grade 7 Life Science. Use the website: https://flademo.schoolnet.com Test Stages Private Draft Public Draft Ready for Scheduling Scheduled In Progress Complete Scheduling the Test Math Fall 2014 Gr 6 Form 1  Test Schedule Dates Math Fall 2014 Gr 6 Form 1  Online Test Options Assigning a Test Hands-On Practice: Schedule an Assessment Preview the online test, then schedule the test. PART THREE (B): ADMINISTERING A FORM/ASSESSMENT Test Administration The assessments created in the FL IBTP system will be administered to students online. Online assessments are administered in two ways: • Secure Tester • Browser-based Proctor Dashboard: Monitor an Online Test Math Fall 2014 Gr 6 Form 1  • Graphic representation of the progress of the class • Links to individual student scores, answer sheets, and PDF test booklets The Student Online Test Experience • Make sure computers are ready for testing. • Students sign in and enter online passcodes. The Student Online Test Experience PART THREE (C): ENTERING SCORES FOR OPEN RESPONSE ITEMS Entering Scores Open-ended items must be scored manually. After scoring each open response item, click Save at the bottom of the Score page. PART THREE (D): REPORTING Finding the Results View assessment results using the data and reports found in the School & District Data and the Classrooms modules. Benchmark Tests Dashboard Additional Reports Pre-formatted Reports Save and Access Your Reports So data is always current, save the report parameters rather than the results. Hands-On Practice: View Data and Report Options View available assessment data and reports in the School & District Data and the Classrooms modules. Using IBTP in Your District In district groups, discuss what resources are needed to plan and deliver professional development in your district. Evaluation Your feedback is valuable. We invite you to respond to a short set of questions. Thank you very much for your time. Thank You NOTE: These materials were developed under a grant from the U.S. Department of Education. However, the contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. Copyright © 2014 Florida Department of Education. All rights reserved.