INSTITUTIONAL REPORTING LEONE NURBASARI, ANU PAMELA SARLY, ACU HIEDI WILKINSON, USC Institutional Reporting from the Australian Graduate Survey Leone Nurbasari Planning and Performance Measurement July 2014 Presentation Overview • • • • • • Key Performance Indicators Program (Course) Review reports Discipline reporting on CEQ/PREQ Graduate destination reporting Comments analyses Other reporting 3 Key Performance Indicators • • ANU Strategic Plan College Operational Plans and KPIs 4 KPI generation • • Raw unit record data at 6 digit FOE, assigned to College scores based on taught disciplines (rolling 5 year average, minimum 10% of load taught by that College) Benchmarking against Go8 data that is weighted according to ANU discipline weightings 5 Example of some underlying 6 digit FOE data and calculations for KPI generation 6 Program (Course) Review Reports • • • • Program reviews Data warehouse reports – CEQ, Employment & Further study Supplemented by additional detailed analyses (including item level results) Supplemented by anonymised open-ended comments 7 Discipline reporting – CEQ/PREQ • • • • Pseudo colleges (custom groupings of 2, 4 and 6 digit FOEs) Pseudo departments (also custom groupings of FOE) Benchmarking – Go8 and national, based on ANU taught FOE (excludes non ANU disciplines) Percentage of individual student-item response in disagreement/neutral/agreement categories, aggregated to scales 8 9 Graduate destination reporting • • • Data warehouse reports – employment rates, sector, industry, salary, employer, job search Cohort-specific analysis Ad-hoc data requests 10 Comment analyses • • • CEQuery with charting via Excel Various student cohorts and org. units Adopting Geoff Scott’s method of proxies for importance and quality 11 Other reporting • • Topic Papers – comprising data from various surveys and student feedback sources, eg. HDR Satisfaction Report Ad-hoc reporting from internal pivot tables – GDS, CEQ, PREQ 12 Questions? • Email [email protected] • Web http://unistats.anu.edu.au/surveys 13 AGS Data and Reporting at ACU Pamela Sarly | Acting Manager, Statistical Analysis & Surveys 2014 Survey Manager Information Forum 17 July 2014 A bit about ACU… ONE OF THE FIRST EDUCATION PROVIDERS IN AUSTRALIA 1857 Good Samaritan Sisters Teacher Training in NSW and VIC 1900 1963 Teacher Training in NSW, VIC, QLD and ACT Merger of education colleges, forming ACU 1991 2014 One of the fastest growing universities in Australia Australia's leading Catholic university which is supported by more than 2,000 years of Catholic intellectual tradition. 12 Schools across 4 Faculties 490 6,263 21,934 826 Higher Degree Research Students Postgraduate Students Undergraduate Students Non-Award Students 12,952 commencing students 16,561 continuing students 29,513 Enrolments 22,245 EFTSL 2,839 international students Staff FTE • 1,051 Academic • 982 Professional Data as at 11 July 2014 AUSTRALIAN GRADUATE SURVEY Data Governance at ACU • A central contact for surveys: o Office of Planning and Strategic Management • Publish aggregated data through internal sites: o Staff Site (www.acu.edu.au/opsm) o SharePoint site • Data updates communicated to relevant staff via email AUSTRALIAN GRADUATE SURVEY Institutional Reporting • Strategic Plan - Traffic Light Report • Quarterly Report • TEQSA Risk Indicators • New Course Application • Course Review and Renewal TRAFFIC LIGHT REPORT • Published in July and December • Progress against the University’s Strategic Goals and Key Result Areas STRATEGIC SCORECARD QUARTERLY REPORT Strategic Plan Goal 1: Student Experience • Published in January, April, July and October • High level quantitative data that is linked to Strategic Goal 1 (Student Experience) AUSTRALIAN GRADUATE SURVEY For the future… • Use of SPSS TAS for qualitative data analysis and reporting • Better linkage to Student Evaluation on Unit and Teaching surveys • Future of AGS – to inform our new Strategic Plan 2015+ Thank you Questions? Australian Graduate Survey COMMUNICATING RESULTS FOR EFFECTIVE DECISION MAKING Note that the figures contained within this presentation are fictitious. Why do we need to communicate results? AGS results are used at USC to: Inform strategic decision making Assess program and institution performance Influence improvements to programs Review learning and teaching Develop internal policies Why use visualisations? “Numbers have an important story to tell. They rely on you to give them a clear and convincing voice.” Stephen Few Choosing the right medium Textual Visual Oral Tapping into human nature Over 50% of the cerebral cortex is involved with the processing of visual inputs. -Lu & Dosher There are over 130 million sight receiving cells in the retina. - Vries 2011 Source: Atranik 2011 Tableau Knowing your audience USC’s Organisational Chart Know your goal Enable insight "The purpose of visualization is insight, not pictures" Shneiderman 1999 GDS Dashboards Flowchart CEQ Dashboards Flowchart Levels of Analysis Graduate Destination Survey Benchmark Group Dashboard Course Experience Questionnaire Longitudinal Dashboard Graduate Destination Survey Faculty Dashboard Graduate Destination Survey Program (Course) Dashboard Course Experience Questionnaire Program (Course) Dashboard How dashboards are used Are they being used? Who is using them? What are they using them for? Director - Quick reference point Program leaders - Program reviews Executive - Marketing and recruitment - Identifying strengths and weaknesses - Information sharing What next? Reports by Field of Education Beyond Graduation Survey University Experience Survey Internal Surveys Interactive Dashboards Qualitative Data Visualisations Atranik.org 2011, http://antranik.org/functional Bibliography areas-of-the-cerebral-cortex. Azzam, T & Evergreen, S 2013, Data Visualization, Part 2: New Directions for Evaluation, Number 140, Google eBook. Card, S, Mackinlay, J & Shneiderman, B 1999, “Readings in Information Visualization - Using Vision to Think”, Morgan Kaufmann, Massachusetts. De Vries, J 2011, The Five Senses, Random House, Victoria. Few, S 2006, Information Dashboard Design: The effective visual communication of data, O’Reilly, California. Lu, Z and Dosher, B 2013, Visual Psychophysics: From Laboratory to Theory, MIT Press, ISBN: 9780262019453. University of Leicester (Learning Development) 2012, Presenting numerical data, <http://www2.le.ac.uk/offices/ld/resources> If you have any questions after today: [email protected] Questions?