Adapting EBP*s AI/NA Prevention Practice: Critique and Application

Peering into the black box
between pre and post:
A theory of change
related to interactive/
experiential delivery
Susan Carter
Evaluation Coordinator
National Indian Youth Leadership Project
[email protected]
J. Fred Springer, EMT Associates
[email protected]
A presentation for the 6th Annual Research
and Evaluation in Adventure Programming
Gaithersburg, MD
March 17-19, 2010
48 site quasi-experimental design
10,500 youth
4 point longitudinal study
Common instrument
Comprehensive process data
Hierarchical statistical design
American Evaluation Association Study of the
Year award, 2000
Largest direct service ATOD prevention
evaluation ever implemented
The Multi-Site High Risk Youth Study
funded by SAMHSA
Fred Springer, EMT, Co-PI
Common Factors Reducing
30-day Drug Use
0 .1 8
0 .1 3 5
Effect Size
0 .0 9
0 .0 4 5
C o nne ct io n B uilding
B e ha viora l L ife- S kills F ocu se d
Intro sp ec tive Le arnin g
P rogra m C ohe ren ce
Inte ns e C ont ac t
Promoting introspection / connectedness are
critical common factors
Out of school more effective
Manualized programs less effective
Intensity counts
Promoting behavioral skills more effective
than focus on knowledge and attitude change
HRY Common Factor Findings
Project Venture identified as one of eight
best programs in HRY study
◦ PV included all of the common factors
Incorporates connectedness building and
 Uses interactive experiential program
delivery strategy
Application: Project Venture
Program Components
◦ Middle school aged American Indian youth
◦ High school aged AI service staff
◦ Mix of risk levels (at risk + high risk)
Experiential education
Connecting to the natural world
Physical and social-emotional challenge
Service learning
Focus on developing positive relationships
Positive Youth Development approach
Essential Elements of Project
Experiential, adventure-based positive
youth development program for American
Indian youth
Recognized by NREPP as evidence-based
The only American Indian ATOD
prevention program currently in NREPP
Widely disseminated throughout the US
and Canada
Twenty year implementation and
evaluation history
Project Venture: Overview
Outdoor, adventure activities:
ropes course, rappelling, canoeing,
backpacking, camping, mountain biking, etc.
Service Learning
Meaningful projects that are youth centered
and designed, address cultural,
environmental and other community needs.
In a research-supported, year long sequential
Project Venture: Strategies
Positive substance abuse prevention
findings (prevent, reduce, delay onset)
especially related to alcohol abuse
 Increased internal assets
 Increased external assets, especially
community and peer domains
Project Venture: Evaluation
We propose to advance knowledge
concerning how adventure based
strategies and activities can be
strengthened, promoted, and replicated.
Purpose of proposed study
Interactive /experiential activities that
clearly incorporate reflective learning
skills and promote connectedness to
positive people and environments will
promote intended behavioral change more
effectively than interactive/experiential
activities that have less focused emphasis
on these interim outcomes.
 Baseline, exit, six and twelve month
follow up
 Assess similar participants in Project
Venture programs:
◦ 6 programs using an enhanced version of PV
◦ 6 “regular” PV programs
◦ (perhaps 6 groups with no PV)
Method notes
Behavioral outcomes (ATOD, etc.)
 Connectedness (e.g., Michael Karcher’s
Hemmingway scales)
 Reflective learning (perhaps adapted from
education literature)
Rigorous development and testing
program to develop the measures
 Rigorous statistical methodology including
propensity scores for improving
comparison equivalence
 Hierarchical modeling to account for
nesting effects, treatment interactions,
treatment settings
 Strong fidelity/coherence measurements
Analysis notes
Characteristics of Most
Successful Prevention
How Project Venture Incorporates these Characteristics through
Interactive Experiential Delivery Strategies
Cultural content, leadership, values / experiential learning / connecting to
the natural world / physical and emotional challenge
150 + hours year (3 + hours per week)
68 sessions / In & after school, weekends, summer
Reflective Orientation
Experiential learning cycle, debriefing; Outdoor adventure: problem
solving; Service learning: student led, problem solving; Camp, treks:
journaling, values, norms, natural world, spiritual awareness; Native
cultural values: spiritual traditions; In school: Problem solving, skill
building, socialization
Connectedness Building
Outdoor adventure: team building, trust building; Service learning: youth
led, meaningful, relationship emphasis; service ethic; Camp, treks: build
caring, cooperative community; Native cultural values; family events,
cultural traditions
Life-skills Focused
Outdoor adventure: cooperation, leadership, efficacy ; Service learning:
problem solving, leadership, cooperation; Camp, treks: leadership, efficacy;
In school: problem solving, skill building
Positive Environment
Cultural content; adult participation, interaction, trust, support;
Clear purpose and process; training; collaboration; implementation project
Community Support
Guided by Native Elders; Native cultural values; positive youth
development approach; collaboration.
High Risk Youth Study, SAMHSA mapped onto Project Venture
Secondary data analysis of 5 years of
pre/post survey data from Project Venture 7th
grade participants (600) and matched
comparison group (400) collected between
 Longitudinal follow-up study of PV program
participants who were in 7th grade in 2006 –
including original program survey (adapted
for older youth cohort) AND life history
Other study ideas

similar documents