ngITEA_Seminar_12_10_07 - International Technology and

Report
Nebraska 4-H Robotics
Effectiveness of Educational Robotics
in the Classroom
Presenters
• Dr. Bradley S. Barker, 4-H Science and
Technology Specialist
– Email: [email protected]
• Dr. Neal Grandgenett, Professor of
Mathematics Education
– Email: [email protected]
• Dr. Gwen Nugent, Associate Research
Professor, Nebraska Center for Research on
Children, Youth, Families and Schools
– Email: [email protected]
Presentation Overview
• Provide background of using robots in nonformal education
• The three iterations (pilot study, large scale
study, NSF ITEST program) and impact data
from the 4-H robotics program.
• Embedded assessment (Reasoning/
Communication)
• Effects of robotics on attitudes
• Summary
• Questions
Purpose of Program
• To address the shortage of students pursuing
careers in science, technology engineering, and
mathematics (STEM).
• Goals
– promote youths’ interest in STEM
fields (including IT),
– introduce basic STEM skills,
– foster problem solving and inquiry,
– and encourage teamwork
Background – Why Robotics?
• Integrate many content areas
(CS, Engineering, Mathematics)
• LEGOs are familiar to youth
• Durable and reusable
• Relative low-cost
• Programming language works
on Macs and PCs
• Widely available
• Motivating
NXT Robot with GPS trailer
RCX Robot
Review of Literature
• Fundamental gap in literature –
what is the impact of using
robotics on STEM learning?
• Most studies are qualitative,
examine self-evaluations, project
descriptions, and student reactions.
• We set out to examine the impact on STEM
learning in a nonformal environment.
Pilot Study: Gibbon Elementary
• Purpose was to develop assessment instrument
and examine impact on learning STEM
concepts.
• Developed 24-item multiple choice test one
item from each unit in the curriculum.
– Test items reviewed by CMU robotic experts and
revised.
• Robotics group met twice a week for six
weeks. Pre test prior to intervention and post
on last day for both groups.
Pilot Study Participants
• Participants
– 32 students in study ages 9-11
median age was 9.0
– 14 students in (9 male,
5 female) experimental group
– 18 students in control group (11 male, 7 female)
– Control group selected by instructor in same grade
but not part of the afterschool program.
Pilot Study Results
• Cronbach’s alpha score of 0.86 on posttest.
• LEGO Robotics questions removed
• STEM assessment items alpha was calculated
at 0.76
• The assessment instrument seemed to be valid
and reliable.
• Used Pell and Jarvis instrument to measure
attitudes towards science.
Pilot Study Results
• Observed impact on learning
– No significant difference (t(30) = 11.60, p = .70 on
pretest scores between groups (M=7.50, SD = 2.58,
control) and M = 7.93, SD=3.71, experimental).
– Significant difference (t(22,17) = 12.93, P < .000
between groups on posttest (M=7.44, SD = 2.98,
control) and M = 17.00, SD= .88, experimental).
• No change in attitudes
Pilot Study
Pilot Results
• Refined the assessment instrument.
• Felt confident students had an increase in
STEM content areas as well as specific robotic
concepts.
• Used instrument for larger study.
Large Scale Study (RCX)
• Participants
– 121 students ages 7-14 from 6 afterschool
programs and 3 4-H clubs.
– 36 youth ages 11-14 acted as a control group from
3 afterschool programs
• Interventions (not concurrent) lasted for 8
weeks. The pretest was administered prior to
the intervention and the post test was
administered immediately after.
Large Scale Study Results
• Impact on learning
– ANCOVA analysis used posttest as dependent and
pretest, gender as covariates
– Main effect was significant F(1,141) = 11.04
p=.001
– Posttest (M=10.68, SD = 3.93, control) and (M =
11.09, SD= 3.93, experimental).
– No difference based on gender F(1, 141) = .833
p=.478
Boxplots Pretest to Posttest
NSF ITEST Program
• Expanded 4-H robotics program
– Includes the integration of robotics
with Geospatial technologies
(GIS, GPS, aerial photography)
– Looking at applications in precision
agriculture and natural resources
– Provide career exploration with visits from scientist
and engineers.
– New robotic kits
ITEST Program Components
• Open to middle school students (200)
• Year 1 - Pilot camps
• Year 2
– Start with 40-hour summer camp
– 1 day camp for educators and leaders
– Youth then complete 80 hours in clubs and afterschool programs
• Year 3
– 40-hour summer camp
– 80 hours in clubs and after school programs
Results of Pilot Camps
• Site 1 – Gretna, NE 6 day overnight camp.
– N = 12 ages, 11-14, median age 12.50
– 8 males, 4 females
– Paid to attend the camp
• Site 2 - Grand Island, NE 5 day camp.
– N = 26 ages 11-15, median age 12.00
– 18 males, 8 females
– Offered through CLCs at Barr M.S.
• Modified content examine and added new
questions, piloted an embedded assessment
Camp Results Academic
• Site 1 pretest m = 14.5, sd = 4.42 and posttest
m = 17.5 sd = 4.89
• Site 2 pretest m = 11.77, sd = 2.99 and posttest
m = 16.04, sd 3.68
• Overall significant increase in scores using
ANCOVA analysis using posttest as dependent
and pretest as covariate F(1,31) = 21.24, p =
.000.
• KR20 score = .81, Alpha score = .799
Boxplots Pretest to Posttest
Boxplots by Content Area
Camp Results Attitude
• Developed new attitude instrument focusing on
robotics
– Modeled after the Motivated Strategies for
Learning Questionnaire (Pintrick, Smith, Garcia, &
McKeachie, 1991)
– Multiple scales focusing on task value, motivation,
self-efficacy, problem solving, cooperative learning
• Conducted interviews with selected youth in
Grand Island
Camp results Attitude
• Goal - promote youths’ interest in STEM fields
3.2
3
2.8
Scientist
Engineer
2.6
Mathematician
Computer
2.4
2.2
2
1
4-point scale
2
Camp results Attitude
• Promote youth’s interest in STEM
– Six of the seven Grand Island youth interviewed
said the camp made them like math and science
more
– Four said it increased their interest in a career in
STEM
Camp results Attitude
• Goal – foster problem solving and inquiry
skills
4.6
Significant increase
for “make a plan”
4.4
I make a plan before I solve
a problem
4.2
I develop my own ways of
solving problems
4
3.8
I like to know how things
work
3.6
I can draw valid conclusions
based on evidence
3.4
I used a step by step
process to solve problems
3.2
3
Pre
Post
Camp results Attitude
• Interviews
– There was kind of a big task, so it made us break it
down into little parts and then get it done step by
step.
– For the robot, you have to start with nothing and
tell it exactly what to do.
– If your robot did something wrong, you have to
figure out after how many steps it was and then you
fix the particular problem.
Camp results Attitude
• Goal – Encourage teamwork
4.2
I like to w ork w ith
others to complete
projects
4
I like listening to
others w hen trying to
decide w hich
problem-solving
approach to use.
3.8
I ask my teammates
for help
3.6
3.4
Pre
Post
Significant increase for
“work with others” and
“listening to others”
Camp results Attitude
• Interviews
– I like the hands on aspect of it. I hate just being
stuck in a chair and writing stuff down.
– It was fun when we did challenges.
Future Directions: Attitude
• Revise the instrument
– Factor analyze the instrument and revise
• Use results to guide instructional improvement
• Initiate new study that measures attitude midway through the camp
Piloting Embedded Assessment
Pretest
Measures
Content Tests
Attitude Test
Robotics
Instruction
Posttest
Measures
Content Tests
Attitude Test
Envelope Activity
Think-Aloud Interview
Design Tasks
Task Rubrics
Control Groups: Partners and Partner Schools
Embedded Asssessment: Envelope Activity
(mathematical communication/reasoning, and also used to discuss programming instruction)
Steps to Mailing a Letter: Student Directions
Directions: In order to do something well, like playing a sport or a board game, you
usually need to know the "procedures" of how to do it. Thus, it is often helpful for you
to know the steps in doing some task, before you are able to explain that task or
procedure to someone else. For example, if you are given a letter that you need to mail,
with an envelope, a stamp, a ruler, and an address of where you want to send the letter,
can you describe how you would mail that letter? Or in other words, can you describe
the general procedure for mailing a letter?
Let's further pretend that this person who will be mailing the letter is a friendly
alien from Outer Space, and that they have absolutely no idea about how to mail a
letter here in the United States. List the steps that the alien should use to mail a
letter. Use the blank sheets of paper provided to make your list. You have 15
minutes to accomplish this task.
Sample Products
Student #5: Posttest
1. Front
Student #5: Pretest
1.
2.
3.
4.
5.
T urn the envolope, so the flap is down.
Make sure that the open flap is at the top.
P ut the stamp in the top right corner.
P ut your address, "the return address," in the t op left corner.
Your addes should look kind of like this:
First Las
111 Cave R
Omaha, NE 6811
6. P ut the address you are sending it t o in the middle.
7. It is supposed to look something like this: First Name Last Name
house # and road
city, state and zip code
Back How t o address and
send an envolope, or let ter.
1. First make sure the front is facing up.
2. Find a stamp & put it in the t op right corner.
3. P ut your address in the t op left corner.
4. P ut the person you're sending it t o in the middle of the
envolope.
Front
Sammi Bray
23 Merry lane
Omaha, NE 68114
Jack Black
222 Jumper lane
Oakland, NE 68045
5. P ut your let ter inside the envolope & close it.
6. Then put it inside the mail box, & put the flag up.
7. And you're good t o go.
By the way it should look something like this.
your address
Front
Back
How to address an envolope
Their address
First Last name
road address
city, state zip
First name Last name
road address
city, state, zip code
stamp
Scoring:
Holisitic Scoring Rubric for Envelope Activity
Student ID:____________________________
Student #5: Posttest
1. Front
En umerati on
of ste ps
Back How t o address and
send an envolope, or let ter.
3
Each step has a unique,
logical identifier
Score:_____
1. First make sure the front is facing up.
Pre ci sionof
ste ps
2. Find a stamp & put it in the t op right corner.
Steps are exact and
unambiguous
Pre:___ Post:___
2
Most steps have a
unique, logical
identifier, but some
steps have not been
enumerated or some
enumerated steps
contain multiple steps
Steps are mostly exact ,
but some are not as
precise as others
1
Few if any steps have
been enumerated
Steps are mostly inexact
and potentially
confusing
Score:_____
3. P ut your address in the t op left corner.
Loopi n g
Looping is used
appropriately in the
Score:_____ steps (e.g., Ņrepeat steps
3-6Ó)
4. P ut the person you're sending it t o in the middle of the
envolope.
Me asureme n t
Score:_____
5. P ut your let ter inside the envolope & close it.
Use of
Example s
Score:_____
6. Then put it inside the mail box, & put the flag up.
7. And you're good t o go.
By the way it should look something like this.
your address
Their address
stamp
C omple te n e ss
of i n stru ction s
Score:_____
First Last name
road address
city, state zip
Looping is used, but not
accurately, or looping is
implied indirectly (e.g.
put name, address, zip
under each other)
Exact measurements are Relative measurements
given in standard units
are given (e.g., Ņin the
(e.g., Ņ4 inches from the t op left cornerÓ)
leftÓ)
Extensive writ ten
Some writ ten and/or
and/or graphic
graphic examples are
examples are used t o
used t o illustrate steps
illustrate steps t o be
t o be taken, or a sample
taken, or a sample
envelope is created,
envelope is created with with both return and
addresses and referred
mailing address given.
t o in the narrative.
If followed, the steps
If followed, the steps
would lead t o the
would almost lead to
mailing of a correctly
the mailing of a
addressed envelope
correctly addressed
envelope
Total Rubric Score:______
First name Last name
road address
city, state, zip code
Looping is not used in
the steps.
Measurement s are not
given
No writ ten and/or
graphic examples are
used t o illustrate steps
t o be taken, or just a
stamp or partial address
is drawn on the sample
envelope.
If followed, the steps
would NOT lead to the
mailing of a correctly
addressed envelope
Envelope Activity Analysis:
Mean
2.1 1.9 1.0 1.6 1.3
1.7
9.6
0.6 0.5 0.2 0.5 0.5
0.7
2.1
Experimental Pretest vs. Experimental Posttest:
Stdev
2.4
0.7
S u m m ary of Pi l ot Data from Mai l in g a Le tte r Acti vity (Expe rim e n tal )
Pretest Data
Student 1
Student 2
Student 3
Student 4
Student 5
Student 6
Student 7
Student 8
Student 9
Student 10
Student 11
Student 12
Student 13
Student 14
Student 15
Student 16
Student 17
Student 18
Student 19
Student 20
Student 21
Student 22
Student 23
Mean
Stdev
Posttest Data
2
3
1
1
1
2
2
2
2
2
3
2
3
3
2
2
3
2
2
2
2
2
2
2
2
1
1
2
2
2
2
2
1
2
2
2
3
1
2
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
1
1
1
2
2
1
1
2
1
2
2
2
2
1
2
2
2
2
1
1
2
1
2
2
1
2
1
1
1
1
1
1
1
1
1
2
1
2
2
1
2
2
1
1
1
2
2
1
1
1
2
1
1
2
1
2
2
2
3
1
2
3
3
2
1
1
2
1
total
12
11
6
7
8
10
8
8
10
7
11
10
11
14
7
11
13
11
11
9
8
10
8
2.1
1.9
1.0
1.6
1.3
1.7
9.6
0.6
0.5
0.2
0.5
0.5
0.7
2.1
Experimental Pretest vs. Experimental Posttest:
Probability Value (p <):
sig at 0.05
sig at 0.01
Enumeratio Precision Looping Measurem Use of Completeness
n of steps of steps
ent
examples of instructions
3
1
1
2
2
2
2
3
2
3
3
3
3
3
2
3
3
3
2
2
2
3
2
2
1
2
1
2
2
2
2
1
3
3
3
2
3
2
3
3
2
2
2
2
3
2
2
1
1
1
2
1
1
2
1
1
1
2
3
1
1
2
1
3
2
1
1
2
1
1
1
2
1
2
2
1
1
1
1
2
3
2
1
2
2
3
2
1
1
1
2
2
3
3
2
2
3
1
1
1
1
1
1
1
2
1
2
2
2
3
2
1
2
1
2
1
1
1
1
2
2
1
2
1
2
2
3
3
2
2
3
3
3
2
2
2
3
2
2.4
0.7
2.2
0.7
1.5
0.7
1.6
0.7
1.7
0.8
2.0
0.7
EnumStep PrecStep Looping
Measure
1.6
0.7
1.7
0.8
2.0
0.7
11.4
2.6
Examples Completeness Total
0.02487 0.0249 0.0023 0.37346 0.0127 0.0347383 0.002
sig at 0.05 sig at 0.01
Enumerat Precision Looping Measure Use of Completeness
ion of of steps
ment examples of instructions
steps
1.5
0.7
EnumStep PrecStep Looping Measure
Probability Value (p <):
Student
2.2
0.7
total
12
8
9
8
13
10
8
11
7
11
12
15
15
11
11
15
15
16
11
9
10
14
11
11.4
2.6
Examples Completeness Total
0.02487 0.0249 0.0023 0.37346 0.0127 0.0347383 0.002
Student Interviews:
Are there any similarities
between the envelope activity
and programming?
Are there any differences
between the envelope
activity and programming?
“You have to do the steps
very carefully”
“It is easier to see the result
when you tell a robot”
“Both big things need to be
broken down into little steps”
“It is more fun to write steps on
the computer than with pencils”
“I had to tell a thing that
knows nothing how to do
something”
“I am better at thinking
than a robot…they just
do what they are told”
Summary
• Robotics seems to have a promising potential
impact on academic achievement
• Robotics impact on attitudes is difficult to
measure; current results suggest that the
impacts are limited to specific areas.
• More research is needed into long term effects
Summary
• Robotics seems to have a potential impact on
learning of robotics concepts and principles
based on pre to post test scores
• More research is needed into long term effects
• Intervention seems to increase interest in STEM
but it is difficult to measure directly and
mathematical interest is particularly challenging
• Embedded assessment is a promising area of
investigation for robotics activities
Questions?
ABack
final quote
following
questions…..
Page
“We have not succeeded in answering
all of your problems. The answers we
have found only serve to raise a whole
set of new questions. In some ways, we
feel we are as confused as ever, but we
believe we are confused on a higher
level and about more important things.”
Omni Magazine, 1992

similar documents