Slide 1

Report
Creating an Aligned Assessment
System to Promote Work Training
Readiness and Career Success
Steve Robbins
Vice President, Research
ACT, Inc.
Though designed to meet a wide array of needs,
all ACT programs and services have one guiding purpose:
Helping people achieve education
and workplace success.
© 2010 by ACT, Inc. All rights reserved.
1
The Challenges
• Create a comprehensive assessment system
that predicts success in education and work
• Tailor assessments to critical transition points
and context
• Understand the complexity and interplay of
cognitive and noncognitive factors across time
and jobs
© 2010 by ACT, Inc. All rights reserved.
2
Do noncognitive factors matter?
• What we know from the world of work:
– Project A from the military
– Training and work outcomes
© 2010 by ACT, Inc. All rights reserved.
3
Combining Personality &
Cognitive Ability Tests
Creating opportunity for incremental validity especially
as criteria vary
Correlations between general cognitive ability and personality tests
and measures of job performance in Project A
Cog
Pers
Both
Criteria
.63
.26
.67
Core technical proficiency
.65
.25
.70
General Soldiering Proficiency
.31
.33
.44
Effort and Leadership
.16
.32
.37
Personal Discipline
.20
.37
.42
Physical fitness and military
bearing
(McHenry, Hough, Toquam, Hanson, & Ashworth, 1990)
© 2010 by ACT, Inc. All rights reserved.
4
Training and Work Outcomes
• Noncognitive factors are predictive of important job
performance criteria (e.g., Rotundo & Sackett, 2002;
Casillas et al., 2009)
– Task Performance
– Organizational Citizenship
– Counterproductive Work Behaviors
– Safety
• Noncognitive factors meet a variety of employer needs
– Screening
– Selection
– Training
© 2010 by ACT, Inc. All rights reserved.
5
Validity of Cognitive and Noncognitive
Measures for Training and Work Outcomes
Training
Outcomes
Work
Performance
General Mental Ability
.54 a
.62 b
Math
.48 a
.52 b
Reading
.44 a
.35 b
Conscientiousness
.17 c
.22 c
Emotional Stability
.10 c
.11 c
Notes. a Brown, Le, & Schmidt (2006). b Salgado et al. (2003).
c Schmidt et al. (2007) corrected for indirect range restriction.
© 2010 by ACT, Inc. All rights reserved.
6
No Unified Theory of Key Characteristics
• However, the literature suggests that several
factors are important:
– Motivation, Social Engagement,
and Self-Regulation
– Personality characteristics (the Big Five)
• Role of behavior ratings vs. self-report
– Different perspectives
– Coaching/training
– Monitoring progress
© 2010 by ACT, Inc. All rights reserved.
7
Common and Specific Factors
• Are the same characteristics (i.e., common
factors) associated with educational and work
outcomes?
• Are there unique (i.e., specific) characteristics
that contribute to the prediction of educational
and work outcomes?
• In particular, how do age and setting impact
the outcomes of interest?
© 2010 by ACT, Inc. All rights reserved.
8
Integrated Pyramid for Success
Career
Development
(Exploration,
Crystallization,
Choice & Match)
Behavioral Development
(Motivation,
Social Engagement, &
Self-Regulation)
Cognitive Development & Acquisition of
Foundational Skills
(Academic Learning & Achievement)
Allen & Robbins (2010)
© 2010 by ACT, Inc. All rights reserved.
9
The
human
side of the
pyramid
© 2010 by ACT, Inc. All rights reserved.
10
Set of Solutions
ACT’s noncognitive programs are part of a
comprehensive and integrated set of solutions,
which address the three broad areas essential for
academic and work success across the lifespan:
 Academic achievement & foundational skills
 Behavior (noncognitive)
 Career planning (noncognitive)
© 2010 by ACT, Inc. All rights reserved.
11
Comprehensive & Integrated Academic
& Work Solutions: Grade 6 - Work
6
Achievement &
Foundational Skills
Behavior
Career Planning
7
Grades
8 9 10 11 12
EXPLORE® PLAN® ACT®
College
Work
COMPASS®
WorkKeys®
SRI and BMS
ACT Interest Inventory
SRI
Performance,
Talent & BMS
College Fit
Fit
© 2010 by ACT, Inc. All rights reserved.
12
Creating a Comprehensive Set of Behavior
(Self & Other Report) Assessments
• Role of meta-analysis and validity
generalization
– Robbins et al. (2004). Do psychosocial and study skill
factors predict college outcomes? A meta-analysis.
Psychological Bulletin, 130, 261-288.
– Robbins et al. (2009). Intervention effects on college
performance and retention as mediated by motivational,
emotional, and social control factors: Integrated metaanalytic path analyses. Journal of Applied Psychology, 94,
1163-1184.
© 2010 by ACT, Inc. All rights reserved.
13
Behavior: Self-Report
Domain
Middle School
High School & College
Work*
Motivation
(Getting work
done)
 Academic Discipline
 Commitment to School
 Optimism






Social
Engagement
(Getting along)
 Family Attitude toward
Education
 Family Involvement
 Relationships w/ School
Personnel
 School Safety Climate
 Social Activity
 Social Connection





Self-Regulation
(Keeping your
cool)
 Managing Feelings
 Thinking Before Acting
 Orderly Conduct
 Academic Self-Confidence
 Steadiness
 Optimism
 Stability
Academic Discipline
Commitment to College
Goal Striving
General Determination
Study Skills
Communication Skills
 Carefulness
 Discipline
 Order
Cooperation
Goodwill
Influence
Sociability
Striving
Note. *Creativity and Savvy do not fully map onto these domains.
© 2010 by ACT, Inc. All rights reserved.
14
Behavior: Other Report
Domain
Middle School & High School
Work
Motivation
(Getting work done)




Social Engagement
(Getting along)
 Communication
 Working with Others
 Communication
 Working with Others
Self-Regulation
(Keeping your cool)
 Managing Feelings
 Conduct




Initiative
Planning & Organizing
Sustained Effort
Performance




Initiative
Planning & Organizing
Persistence
Responsibility
Organizational Citizenship
Stress Management
Following Rules
Adaptability
© 2010 by ACT, Inc. All rights reserved.
15
Career Planning
Domain
Middle School & High School
Work
Interests
 ACT Interest Inventory
 DISCOVER Career
Information Decision System
 Fit Inventory
Values
 DISCOVER Career
Information Decision System
 Fit Inventory
- Neuman et al. (2009). Job congruence, academic achievement, and earnings
- Tracey & Robbins (2006). The interest-major congruence and college success relation:
A longitudinal study.
© 2010 by ACT, Inc. All rights reserved.
16
Tailoring Assessments to Critical
Transition Points and Context
• The role of noncognitive measures in
promoting work training readiness and career
success
– Talent supply chain
– Meeting various needs
© 2010 by ACT, Inc. All rights reserved.
17
Talent Supply Chain
Pre-K
Primary
(K-8)
Secondary
(9-12)
Post
Secondary
(2 & 4-yr)
Work
• Noncognitive factors are important at all of the
key transitions that lead individuals to the
world of work.
© 2010 by ACT, Inc. All rights reserved.
18
Noncognitive Measures Meet Various
Workforce Needs
Purpose
Needs
ACT Noncognitive Solutions
Pre-Selection
-- Screen people in most cost effective way WorkKeys Performance
-- Find honest/dependable employees
-- Save time in the screening process
Recruitment
-- Identify people who fit the work
environment
-- Identify people with skills that match
the job
WorkKeys Fit
Selection
-- Select employees with skills that best fit
the job
-- Save time in selection process
-- Select people in most cost-effective way
-- Find honest/dependable employees
-- Certifying employees
WorkKeys Performance
WorkKeys Talent
WorkKeys Fit
© 2010 by ACT, Inc. All rights reserved.
19
Noncognitive Measures Meet Various
Workforce Needs (cont.)
Purpose
Needs
ACT Noncognitive Solutions
Coaching/
Development
-- Identify other jobs that an employee can
fit
-- Develop employees for future company
needs
-- Employees identify areas of improvement
WorkKeys Talent
WorkKeys Fit
Behavioral Monitoring Scales
Succession/
Leadership
Planning
-- Identify candidates for top-level positions
-- Develop employees for future needs of
company
-- Retain top performers
WorkKeys Talent
WorkKeys Fit
Training/
Development
-- Identifying work readiness
-- Identify basic workplace skill levels
-- Educating about career planning
-- Job Placement
WorkKeys Talent
WorkKeys Fit
Behavioral Monitoring Scales
© 2010 by ACT, Inc. All rights reserved.
20
Critical Issues
• Self vs. other report
• Changing attitudes, behaviors, and skills
• Complexity of personality & career
relationships
– Different personality characteristics are important
for different jobs (whether across or within jobs)
– Moderation effects of personality on ability
– Curvilinear effects of personality on performance
© 2010 by ACT, Inc. All rights reserved.
21
Self vs. Other Report
• Connelly & Ones (2010) conducted meta-analyses of
self-other ratings based on 44,000+ individuals across
263 independent samples.
– Other ratings are strong predictors of behaviors,
particularly for academic achievement and job
performance.
– In some cases, other ratings yielded predictive validities
substantially greater than self-ratings.
– When other-ratings are added to self-ratings, results show
considerable increases in validity (with gains more
pronounced for Conscientiousness, Emotional Stability, and
Extraversion).
© 2010 by ACT, Inc. All rights reserved.
22
Changing Attitudes, Behaviors, and Skills
Behavior modeling training (BMT), based on Bandura’s social
learning theory, has become a widely used psychologically-based
approach to training in work environments.
A recent meta-analysis by Taylor et al. (2005) summarizing 117
studies found substantial effects of BMT on a variety of training
outcomes.
Outcome
d
Attitudes
.56
Behaviors
.33
Declarative knowledge
.69
Procedural knowledge-skills
1.21
© 2010 by ACT, Inc. All rights reserved.
23
“It was about here, wasn’t it, Ed, when you came on board as sales manager?”
Harvard Business Review. March 2007. p. 90
© 2010 by ACT, Inc. All rights reserved.
24
Complexity of Personality & Career
• Different personality characteristics are
important for different jobs (whether across or
within fields).
– Conscientiousness appears to be important for all
jobs.
– Extraversion is relevant to some jobs (e.g., sales)
but not others (e.g., computer programmers).
© 2010 by ACT, Inc. All rights reserved.
25
Complexity of Personality & Career
Across Fields
• Position
Computer Programmers
• O-NET Job Zone
Four: Considerable
Preparation Needed
• Education
Most occupations in this
zone require a four-year
bachelor's degree, but
some do not.
http://online.onetcenter.org/link/summary/15-1021.00
• Position
Sales Representatives
• O-NET Job Zone
Four: Considerable
Preparation Needed
• Education
Most occupations in this
zone require a four-year
bachelor's degree, but
some do not.
http://online.onetcenter.org/link/summary/41-4011.00
© 2010 by ACT, Inc. All rights reserved.
26
Complexity of Personality & Career:
Comparing Scores Across Fields
Computer Programmers and Sales Representatives
Computer Programmers and Sales Representatives
Carefulness
Carefulness
Cooperation
Cooperation
Creativity
Creativity
Discipline
Discipline
Goodwill
Goodwill
Influence
Influence
Optimism
Computer
Programmers
Optimism
Order
Sales
Representatives
Order
Savvy
Savvy
Sociability
Sociability
Stability
Stability
Striving
Striving
0
10
0
20
10
30
20
40
30
50
40
50
60
60
70
70
80
80
Scale Scores
Scale Scores
© 2010 by ACT, Inc. All rights reserved.
27
Complexity of Personality & Career
Moderation
Postlethwaite et al. (2009) found that conscientiousness was a
stronger predictor of safety behavior for individuals with lower levels
of cognitive ability.
© 2010 by ACT, Inc. All rights reserved.
28
Complexity of Personality & Career
Curvilinear Relationships
Le et al. (in press) found curvilinear relationships between
Emotional Stability and job performance (task, OCB, CWB), as
well as between Conscientiousness and task performance.
© 2010 by ACT, Inc. All rights reserved.
29
Questions?
Correspondence regarding this presentation
should be addressed to:
Steve Robbins
Vice President, Research
ACT, Inc.
[email protected]
© 2010 by ACT, Inc. All rights reserved.
30
References
Allen, J. & Robbins, S. (2010). Effects of interest–major congruence, motivation, and
academic performance on timely degree attainment. Journal of Counseling
Psychology, 57, 23–35.
Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there
incremental validity for training performance? International Journal of Selection and
Assessment, 14, 87-100.
Casillas, A., Robbins, S. B., McKinniss, T., Postlethwaite, B., & Oh, I.S. (2009). Using
narrow facets of integrity to predict safety: A test validation study. International
Journal of Selection and Assessment, 17, 119-125.
Connelly, B.S. & Ones, D.S. (2010). An Other Perspective on Personality: Meta-Analytic
Integration of Observers’ Accuracy and Predictive Validity. Psychological Bulletin,
136, 1092–1122.
Le, H., Oh, I.-S., Robbins, S. B., Ilies, R., Holland, E., & Westrick, P. (in press). Too Much
of a Good Thing: Curvilinear Relationships Between Personality Traits and Job
Performance. Journal of Applied Psychology.
McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990).
Project A validity results: The relationship between predictor and criterion domains.
Personnel Psychology, 43, 335-354.
Neumann, G., Olitsky, N., & Robbins, S. (2009). Job congruence, academic achievement,
and earnings. Labour Economics, 16, 503-509.
Postlethwaite, B., Robbins, S., Rickerson, J., & McKinniss, T. (2009). The moderation of
conscientiousness by cognitive ability when predicting workplace safety behavior.
Personality and Individual Differences, 47, 711-716.
© 2010 by ACT, Inc. All rights reserved.
31
References (cont.)
Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do
psychosocial and study skill factors predict college outcomes? A meta-analysis.
Psychological Bulletin, 130, 261-288.
Robbins, S. B., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college
performance and retention as mediated by motivational, emotional, and social
control factors: Integrated meta-analytic path analyses. Journal of Applied
Psychology, 94, 1163-1184.
Rotundo, M. & Sackett, P. R. (2002). The relative importance of task, citizenship, and
counterproductive performance to global ratings of job performance: A policycapturing approach. Journal of Applied Psychology, 87, 66–80.
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003).
International validity generalization of GMA and cognitive abilities: A European
community meta-analysis. Personnel Psychology, 56, 573-605.
Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of
Cognitive Ability and Personality in Job Performance and Training Performance:
Some Surprising New Research Findings. Paper presented at the 2007 ATP
conference, Palm Springs, CA. Feb. 6.
Taylor, Paul J., Russ-Eft, D.F., & Chan, D.W.L. (2005). A Meta-Analytic Review of
Behavior Modeling Training. Journal of Applied Psychology, 90, 692-709.
Tracey, T.J.G., & Robbins, S.B. (2006). The interest-major congruence and college
success relation: A longitudinal study. Journal of Vocational Behavior, 69, 64-89.
© 2010 by ACT, Inc. All rights reserved.
32

similar documents