Achievement Level Descriptors

Report
Achievement Level
Descriptors and Cut Scores
December 2014
Achievement Level Descriptors
• Achievement level descriptors (ALDs) are a means
of describing performance on a standardized test in
terms of levels or categories of performance.
• For the Smarter Balanced assessments, outcomes
will be reported in terms of four levels of
achievement: Level 1, Level 2, Level 3, and Level 4.
• The ALDs are text descriptions of the knowledge,
skills, and processes demonstrated by students in
each category of performance.
ALDs and CCSS
“…the ALDs reflect the depth and rigor of the
CCSS as well as the way in which Smarter Balanced
intends to assess the CCSS.”
ALDs
• Smarter Balanced developed multiple levels
• Four interrelated levels
• Used to
• Guide development of the assessment
• Set cut scores
• Interpret results
ALDs
• Labeled as “initial” because they will be refined and
finally adopted by Smarter Balanced after student
performance data are collected through field test
and after standard setting.
• Will be augmented to include the reporting ALDs.
Range ALD Example: Mathematics
• Target D: Apply and extend previous
understandings of numbers to the system of
rational numbers.
Level 1 Student
• Level 1 students should be able to place all integers on a
number line and integer pairs on a coordinate plane
with one-unit increments on both axes.
• The Level 4 student demonstrates thorough ability to
produce compelling, well-supported writing for a diverse
range of purposes and audiences.
Threshold ALD Example:
Mathematics
• The student who just enters level 3 should be able
to:
• Generate measurement data by measuring length using
rulers marked with quarter-inch intervals and represent
the data on a line plot marked with quarter-inch
intervals.
• Solve word problems involving perimeters of polygons.
Range ALD Example: ELA
• Target 12: Interpret, explain, or connect
information presented within or across texts (e.g.
compare/contrast, cause/effect, integrate
information).
Level 3 and 4 Students
• Level 3 students should be able to adequately interpret,
explain, or connect information presented within or
across texts of moderate-to-high complexity.
• Level 4 students should be able to thoroughly interpret,
explain, or connect information presented within or
across texts of unusually high complexity.
Threshold ALD Example: ELA
• The student who just enters level 3 should be able
to:
• Use details and information from texts of moderate
complexity to support answers and inferences.
• Identify or summarize central ideas/key events in texts
of moderate complexity.
• Use supporting evidence to justify/explain own
inferences in texts of moderate complexity.
ALDs
• Labeled as “initial” because they will be refined and
finally adopted by Smarter Balanced after student
performance data are collected through field test
and after standard setting.
• Will be augmented to include the reporting ALDs.
Next Steps
• Edits/refinements to draft ALDs based on field test
data and standard setting
• Development of Reporting ALDs
Review of Smarter Balanced
Achievement Level Setting
Joe Willhoft, Ph.D., Executive Director
Smarter Balanced Assessment Consortium
California State Board of Education
November 13, 2014
Recruiting Activities
•
•
•
Online Panel
– Over 10,000 registered
In-Person Panel
– 504 panelists confirmed
40+ alternates
482 participants
– Demographics
Representation from each state in each panel
In-Person panel composition matched SB teacher workforce
Cross-Grade Panel
– 64 panelists from In-Person panels
Composition of In-Person Panels
Demographics:
Representation from each state in each panel
In-Person panel matched SB teacher workforce by
gender and ethnicity
Completion of the Online Panel
•
•
•
•
Opening of the Window
– October 6-17
– Individual windows and total window extended
Level of Participation
– 10,099 registered
– 5,840 logged in
– 2,660 submitted
Support Provided
Results Shared with In-Person Panel
Completion of the In-Person Panel
•
•
Training Activities
– Software
– Common Core and ALDs
– Ordered Item Booklet
Panel Activities
– Bookmark placement
– Discussion
•
Table Leader training before Day 1 begins
Training
As you study each item in the OIB, discuss two questions with your fellow
panelists:
22
21
20
19
1. What do you know about a student who
responds successfully to this item; that is, what
skills must a student have in order to know the
correct answer?
18
17
16
15
14
13
12
11
10
2. What makes this item
more difficult than preceding
items?
9
8
7
6
5
4
3
2
1
Ordered
Item
Book
1. What do you know about a student who responds
2. What
makes this
itemitem;
morethat
difficult
thanskills
preceding
successfully
to this
is, what
must aitems?
student have in order to know the correct answer?
41
Threshold Achievement Level Descriptors
The student who just enters Level 3 should be able to:
Targets
1–7:
Reading
Literary
Text







Use details and information from texts of moderate complexity to support
answers and inferences.
Identify or summarize central ideas/key events in texts of moderate
complexity.
Begin to determine the intended meanings of words, including words with
multiple meanings, based on context, word relationships, word structure, and
use of resources in texts of moderate complexity.
Use supporting evidence to justify/explain own inferences in texts of
moderate complexity.
Interpret, specify, or compare how information is presented across texts of
moderate complexity.
Begin to relate knowledge of text structures, genre-specific features, or
formats to obtain, interpret, explain, or connect information within texts of
moderate complexity.
Determine or interpret figurative language, literary devices, or connotative
meanings of words and phrases used in context and partially explain the
impact of those word choices on meaning and tone in texts of moderate
complexity.
Items 18 – 22:
less than a 50% chance of
success.
22
21
20
L3
19
17
18
16
Items 1-17:
At least 50% chance
of success
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Ordered
Item
Book
Ask yourself: Would a student at
the threshold of Level 3 have at
least a 50% chance of earning this
point?
Yes: Move on to the next item.
No: Place your bookmark here.
24
Results of the In-Person Panels
•
Each grade-by-content panel recommended 3 “cut
scores” to define 4 achievement levels
– Group median
– 10th, 25th, 75th, 90th percentile of panel also recorded
Evaluations
How confident are you about the three bookmarks you just
entered? (At end of Round 3)
Bookmark
Very
Confident
Level 2
222 (47%) 237 (51%)
10 (2%)
0 (0%)
469
Level 3
234 (50%) 220 (47%)
15 (3%)
0 (0%)
469
Level 4
245 (52%) 217 (46%)
7 (1%)
0 (0%)
469
Confident
Very
Uncertain Uncertain Total
Overwhelming endorsement of process (92-99%
positive) on 14 separate measures.
After In-Person Panels
Chiefs, K-12 Leads, Higher Ed Leads
(Round 5 – Nov. 6)
•
•
•
Consider recommendations through policymaker lens
Two principles to balance:
–
–
Honor the work/advice of the Ach. Level Setting panels
Consider external information about student readiness for
college-level coursework
Conversations with states are continuing
State-by-state Adoption
•
•
After Smarter Balanced states approve Consortiumlevel cut scores…
Each state follows its own processes for adoption of
scores for its purposes/uses
What Do You Notice?
• As you review the Threshold scale scores, what do
you notice?
• As you review the percentage of students scoring at
each achievement level?
• What are the similarities and differences between
the ELA and mathematics scale scores and
achievement levels?
Remember!
• Scale scores from the Smarter Balanced
assessments cannot and should not be compared
with scale scores from the previous STAR/CST
assessments.
• Different standards are being assessed
• Different assessment methods are being used
• Different levels of cognitive rigor involved
Resources
• http://www.smarterbalanced.org/achievementlevels/
• http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/11/Smarter-Balanced-ELALiteracy-ALDs.pdf
• http://www.smarterbalanced.org/wordpress/wpcontent/uploads/2012/11/Smarter-Balanced-MathALDs.pdf
Contact Information
Kathy Caric, Senior Assessment Fellow
[email protected]
661-703-4575

similar documents