Interpreting IDEA reports Diagnostic Form Short Form www.lvc.edu/idea Page 1 of report Page 1 of report Guide says: “Report Responds to These Questions” 1. Overall, how effectively was this class taught? 2. How does this compare with the ratings of other teachers? 3. Were you more successful in facilitating progress on some class objectives than on others? 4. How can instruction be made more effective?* 5. Do some salient characteristics of this class and its students have implications for instruction? * Diagnostic Form Report only. Our Modest Ambitions Today • Understand summative scores – Raw averages – Converted scores – Adjusted scores – How used at LVC • Formative use of p.2 – relevant objectives – POD papers on objectives • Formative use of p.3 for Diagnostic only – POD papers on strategies Summative scores …. How “Summary Evaluation” Computed • 41. Overall, I rate this instructor an excellent teacher. (25%) • 42. Overall, I rate this course as excellent. (25%) • Objectives 1-12 (21.-32) that are ranked as Important or Essential (50%) “Converted averages” • Transform group’s mean to 50, s.d. to 10. • Examples: – Score 57 means .7 group s.d.’s above average – That’s at 75.8 %-tile of Normal Distribution – … but these groups of IDEA scores not Normal 6 5 4 Normal 3 IDEA 2 1 0 0 20 40 60 80 Adjusted Scores: Effects that are removed P. 2 of Diagnostic Form report: Adjusted Scores: Effects that are removed P. 2 of Diagnostic Form report: Only these on Short Form When to Use Adjusted Scores for Administrative Decisions: Where is there evidence of effective teaching? (old) Are adjusted scores lower or higher than unadjusted scores? Lower Higher Yes No Use adjusted scores *Expectations defined by your unit. Is unadjusted Score on Progress toward Objectives high? (≥4.2) Use unadjusted (raw) scores When to Use Adjusted Scores for Administrative Decisions: Where is there evidence of effective teaching? (new) Are adjusted scores lower or higher than unadjusted scores? Lower Higher Yes No Use adjusted scores *Expectations defined by your unit. Do unadjusted (raw) scores meet or exceed expectations?* Use unadjusted (raw) scores College Use for Evaluating Faculty • Current: – Summative: Pay attention to extremes – Assess formative use of IDEA by instructor • IDEA recommendation – Summative: Establish standards • Faculty set standards • Department level or College level • Assess progress in meeting them Page 2: Objectives P. 3 of Diagnostic Form Report: Directions…. Follow the links… www.theideacenter.org/podidea Teaching Methods Relevant to Target Objectives Teaching Methods Relevant to Target Objectives Try it 1. Identify objective… 2. Find related teaching method… 3. Follow link: www.theideacenter.org/podidea Last page: Raw response frequencies … especially additional questions We can retrieve the additional questions so you know which are which. Effects of Evals being On-Line? • Study of on-line vs paper by IDEA published in 2010: – http://www.theideacenter.org/sites/default/files/Tech nical Report 16.pdf • Online avg response rate 57% – LVC response rates 70-75% • No significant change in means or s.d.’s of responses – If loss of middle group, s.d. would go up. – On 12 objectives, s.d.’s slightly lower for on-line! Key questions for LVC (from IDEA) • How much emphasis will you give student ratings in the evaluation of teaching? • How many classes need to be rated? [LVC: all] • Performance categories: How many should you use? • Do you plan to authenticate objectives? • Will you weight the importance of items or groups of items in the decision-making process? If so, how? • Will you use averages on a five-point scale or converted averages? • Will you use adjusted or unadjusted scores? References • www.lvc.edu/idea – Campus IDEA page • POD notes: – The 20 methods: www.theideacenter.org/podidea – The 12 objectives: www.theideacenter.org/podidea/PODNotesLearning.html • IDEA: www.theideacenter.org • Technical: http://www.theideacenter.org/sites/default/files/ techreport-12.pdf - Data about adjusting, etc.