Site Visit Team Chairs Role and Responsibilities January 2013 ACCREDITATION COUNCIL FOR BUSINESS SCHOOLS AND PROGRAMS AGENDA ITEMS: Welcome/Introductions Role of the chair Resources available to site visit team Scheduling of Consensus Calls Travel arrangements Communication with institutional contact Site Visit Schedule WEBINAR AGENDA, CONT. Working with first-time evaluators Preparing feedback report – format, editing, timetable, etc. Submission of feedback report Submission of expense reports and other forms Any other topics ROLE OF THE CHAIR Receive self-study of the institution to be visited. Review self-study for completeness and content. Identify those areas in need of intensive scrutiny by the team while on site. Coordinate with team members, commissioner, and mentor to schedule consensus call. Determine how the self-study should be divided among the team members for on site review. Prepare tentative assignments for team members and communicate those assignments to each member. ROLE OF THE CHAIR Contact team members to obtain travel itineraries for arrival to and return from the site to be visited. Discuss tentative self-study assignments and areas of concern identified in the self-study. Contact institutional coordinator to confirm visit dates. Determine transportation and lodging accommodations. Provide the coordinator with each team member’s itinerary. ROLE OF THE CHAIR, CONT. Plan and finalize schedule of visit to include meetings with president, head of business unit, faculty, and other key personnel with the institutional coordinator. Provide institutional coordinator with a list of sites and units to be visited by the team. Provide institutional coordinator with the requirements for the team work room, including all materials to be provided, computer and clerical support, and telephones. Develop a schedule of activities, visits, and inspections for each day of the visit to be distributed to institutional representatives and to the visiting team. ROLE OF THE CHAIR, CONT. Schedule a meeting of the evaluators for the afternoon/evening prior to the initiation of the site visit to - finalize the schedule of activities, - discuss the self-study, - finalize the distribution/assignment of individual team member responsibilities, - establish the procedure to follow in summarizing observations, - determine the format of the exit conference, - discuss preparation of the team report, and finalize the time frame for completion of all activities. ROLE OF THE CHAIR, CONT. Include time frames in each day’s schedule when the team can convene to determine progress, identify problems or needs, and schedule additional interviews or data gathering as needed. Communicate to institutional coordinator the undesirability of scheduling social events unless it is agreed that such events will facilitate interaction with key personnel, i.e., board members, alumni, employers, community residents, etc., who would otherwise be unavailable. ROLE OF THE CHAIR, CONT. Utilize ACBSP staff or the Commissioners as resource persons to clarify any problems of interpretation, etc., which may confront the team during the course of the visit. Assure that each team member performs his/her tasks in such a manner as to validate the content of the self-study and not to function as consultants. Do not express personal comments as Chair nor allow team members to express such comments regarding their perceptions that the institution does/does not meet standards, or how the business unit should be structured, etc., in contrast to what is observed or documented. References to how business activities are conducted at team member’s institutions should not be used as the criterion for judging the quality of the business unit being examined. ROLE OF THE CHAIR, CONT. As team Chair, you will be expected to prepare a final report of the team visit that will reflect the consensus of the team as a whole. The Chair will be expected to prepare an Executive Summary of the visit to provide an overview of the entire visit and to express the extent to which the institution accurately described the business unit and the extent to which the team was able to validate the information contained in the self-study document. This Summary should be concise (about 3 pages) and cover all the major standards, e.g., Leadership, Strategic Planning, etc. RESOURCES AVAILABLE The CD enclosed with the site visit materials contains several folders with these files: Evaluation and Site visit handouts ACBSP Observation Guidelines Scoring Guidelines Practical Guide for Evaluators (which includes) Site Visit Issue Worksheet EVALUATION AND SITE VISIT HANDOUTS, CONT. Site Visit Questions Site Visit Issue Strategy Evaluator Workbook Executive Summary Feedback Report Forms to sign and return – evaluator agreement form, expense form, and evaluation form (electronically). EVALUATOR RESOURCES ON CD, CONT. Accreditation Process Manuals Standards and Criteria (most current version) Electronic copy of the self-study when provided CHANGES IN WORKBOOK FORMAT : ASSOCIATE DEGREE WORKBOOK EXAMPLE STANDARD 1: LEADERSHIP Business Unit administrators and faculty should lead and be involved in creating and sustaining values, business school or program directions, performance expectations, student focus, and a leadership system that promotes performance excellence. Values and expectations should be integrated into the business unit’s leadership system to enable the business unit to continuously learn, improve, and address its societal responsibilities and community involvement. Criterion 1.1 - Leadership Strategies Explain the business unit leadership strategies or systems to foster a high-performance work environment that results in meeting the mission of the business unit. Explain how the business unit maintains effective communications and develops participation throughout the business unit. Observations of Evaluator: Is there evidence that all areas in this Criterion have been addressed? Strengths: OFIs: Yes No CHANGES IN WORKBOOK FORMAT : BAC/GRAD DEGREE WORKBOOK EXAMPLE Criterion 1.1.c The business school or program must have processes in place for evaluating the performance of both administrators and faculty. Explain how the performances of administrators and faculty are evaluated. Observations of Evaluator: Is there evidence that all areas in this Criterion have been addressed? Yes No Strengths: OFIs: CHANGES IN FORMAT – NO MORE TABLE FORMAT TO ENTER INFORMATION – NO USE OF + AND – FOR COMMENTS. Advantage: After each team member has finalized their comments, it is a simple copy and paste into the feedback report. You do not have to remove the table formatting. FEEDBACK REPORT TEMPLATE NEW FOR ASSOCIATE DEGREE No changes for Baccalaureate/Graduate Degree Reports. The Executive Summary is part of the feedback report. EXECUTIVE SUMMARY Key summary findings from the feedback report: The most important strengths or outstanding practices are: The most significant opportunities for improvement are: FEEDBACK REPORT TEMPLATE, CONT. COPY AND PASTE FINAL COMMENTS FROM EVALUATOR WORKBOOK: STANDARD #1. Leadership Business Unit administrators and faculty should lead and be involved in creating and sustaining values, business school or program directions, performance expectations, student focus, and a leadership system that promotes performance excellence. Values and expectations should be integrated into the business unit’s leadership system to enable the business unit to continuously learn, improve, and address its societal responsibilities and community involvement. Strengths Example: The business unit administrators communicate program directions and performance expectations to their faculty as part of the faculty handbook and during the mandatory training provided at the beginning of the academic year. Opportunities for Improvement Example: While performance expectations are clearly communicated, there is no evidence that the business unit has a process in place to use results from administrator and faculty evaluations to make improvements. NEW EVALUATOR CRITIQUE FORM Excel Worksheet rather than Word Document Process can now be evaluated by Pre-site visit performance Site visit performance Post Site visit performance Individual team member performance The new format will allow ACBSP to provide each team member with feedback from the team as well as commissioners. NEW FORMAT FOR EVALUATOR CRITIQUE: SAMPLE CONSENSUS CALL AGENDA: Introduction of Participants / Self-introductions (5 min) Name, School, Location History with ACBSP / Experience as an evaluator Cell phone contact info Review of Planned Schedule (5 min) Plans for arrival in destination city Review of Divide and Conquer plans (attached) Travel Plans to Satellite Campuses (if applicable) Institution Accommodation forms Discussion of Scoring Summary Worksheet (30 min) Discussion led by Lead Reviewer for each Standard Goal = Preliminary score of BiC, VG/E, VG, G, IN, MIN for each examination item (worksheet attached) SCHEDULING CONSENSUS CALLS ACBSP requires that all site visit teams conduct a consensus call prior to the site visit. Once you have finalized the date and time, contact Maria Hallerud at [email protected] to schedule calls. SAMPLE CONSENSUS CALL AGENDA: Identification of Additional Info Needs from Institution (10 min) Roundtable – Open Discussion / Other Topics (5 min) Complete/submit Program Evaluator Agreements prior to Site Visit Goal: Complete Draft Feedback Report with Strengths and OFI’s before departure on Wednesday Wrap-Up / Adjourn (2 min) Divide & Conquer Plan Standard 1 - Chair Standard 2 - Team Member 2 Standard 3 - Team Member 3 Standard 4 - Chair Standard 5 - Team Member 2 Standard 6 - Team Member 3 Backup = Team Member 2 Backup = Team Member 3 Backup = Chair Backup = Team Member 2 Backup = Team Member 3 Backup = Chair SCORING BAND USED DURING CONSENSUS CALL: Individual Assessment Dimensions (See Qualitative Scoring Band Dimensions and Assessment Guidelines for definitions) Best in Class Very Good to Excellent Very Good Good Improvements Needed Major Improvements Needed SCORING BAND Qualitative Dimensions Approach/ Deployment Quantitative Dimensions Results Poor * no systematic approach evident; anecdotal information Poor * no results or poor results in areas reported Major Improvements Needed * beginning of a systematic approach to the primary purposes of the item * early stages of a transition from reacting to problems to a general improvement orientation * major gaps exist in deployment that would inhibit progress in achieving the primary purposes of the item * a sound, systematic approach, responsive to the primary purposes of the item * a fact-based improvement process in place in key areas; more emphasis is placed on improvement than on reaction to problems * no major gaps in deployment, though some areas or work units may be in very early stages of deployment Major Improvements Needed * early stages of developing trends; some improvements and/ or early good performance levels in a few areas * results not reported for many to most areas of importance to the applicant’s key business requirements Good * a sound, systematic approach, responsive to the overall purposes of the item * a fact-based improvement process is a key management tool; clear evidence of refinement and improved integration as a result of improvement cycles and analysis Good to Excellent * improvement trends and/ or good performance levels reported for many to most areas of importance to the applicant’s key business requirements * no pattern of adverse trends and/ or poor performance levels in areas of importance to the applicant’s key business requirements * some trends and/ or current performance levels -evaluated against relevant comparisons and/ or good to very good relative performance levels * current performance is good to excellent in most areas of importance to the applicant’s key business requirements * most improvement trends and/ or performance levels are sustained * many to most trends and/ or current performance levels -evaluated against relevant comparisons and/ or bench- Good Good to Excellent Best in Class * approach is well-deployed, with no major gaps; deployment may vary in some areas or work units * a sound, systematic approach, fully responsive to all the requirements of the item * a very strong, fact-based improvement process is a key management tool; strong refinement and integration -- backed by excellent analysis * approach is fully deployed without any significant weaknesses or gaps in any areas or work units Best in Class marks -- show areas of leadership and very good relative performance levels * current performance is excellent in most areas of importance to the applicant’s key business requirements * excellent improvement trends and/ or sustained excellent performance levels in most areas * strong evidence of industry and benchmark leadership demonstrated in many areas ASSOCIATE DEGREE SCORING SUMMARY WORKSHEET Evaluator Name: School Name: SUMMARY OF EXAMINATION ITEMS 1 LEADERSHIP 2 STRATEGIC PLANNING 3 STUDENT, STAKEHOLDER, AND MARKET FOCUS 4 MEASUREMENT, ANALYSIS, AND KNOWLEDGE MANAGEMENT 4.1 Student Learning Outcomes Assessment 4.2 Program Evaluation 4.3 Student Assessment 5 FACULTY AND STAFF FOCUS 5.1 Human Resource Planning 5.2 Faculty Qualifications 5.3 Faculty Composition 5.4 Faculty Deployment 5.5 Faculty Load 5.6 Faculty Evaluation 5.7 Faculty and Staff Professional Development & Scholarly Activities 5.8 Faculty Instructional Development 5.9 Faculty Operational Policies, Procedures, & Practices 6 PROCESS MANAGEMENT 6.1 Curriculum 6.2 Professional Component 6.3 General Education Component 6.4 Business Major Component 6.5 Off-Campus Operations and Unique Items 6.6 Minimum Grade Requirement 6.7 Learning and Academic Resources 6.8 Support Services 6.9 Educational Innovation 6.10 Articulation and Transfer Relationships Individual Assessment Dimension Baccalaureate/Graduate Degree Scoring Summary Worksheet Examiner Name: School's Name: SUMMARY OF EXAMINATION ITEMS 1 LEADERSHIP 2 STRATEGIC PLANNING 3 STUDENT AND STAKEHOLDER FOCUS 4 MEASUREMENT AND ANALYSIS OF STUDENT LEARNING AND PERFORMANCE 4.1 Selection and Use of Information and Data 4.2 Selection and Use of Information Results 4.3 Selection and Use of Comparative Information and Data 4.4 Continuous Process Improvement 5 FACULTY AND STAFF FOCUS 5.1 Human Resource Planning 5.2 Employment 5.3 Faculty Deployment 5.4 Faculty Size and Load 5.5 Faculty Evaluation 5.6 Faculty and Staff Development 5.7 Faculty Operational Procedures, Policies, and Practices 5.8 Scholarly and Professional Activities Individual Assessment Dimension TRAVEL ARRANGEMENTS: Airfare – ACBSP is trying to reduce expenses for our member institutions. Contact Steve or Diana prior to booking a flight if airfare is over $500. Location will determine airfare. Hotel arrangements – some schools will arrange for direct billing. Others will ask that each team member be responsible for payment and get reimbursed from ACBSP. Please ask your institutional contact how they want to handle this. TRAVEL ARRANGEMENTS: Meals – ACBSP appreciates your efforts to keep expenses down during your stay. Try to avoid the most expensive items on the menu and purchasing alcohol as some schools do not cover alcoholic beverages for meal reimbursements. Share transportation when possible. INSTITUTIONAL CONTACT The chair serves as the primary contact for the team. Any questions for the school should come through you so that the contact person is not overwhelmed with multiple messages and to avoid conflicting information. If you have difficulty getting responses from the institutional contact or other team members, please let ACBSP know. INSTITUTIONAL CONTACT As you know, some institutions are more prepared for a site visit than others. You can help by: providing a list of documents you will need in the resource room, asking where the resource room is located, where you should park, get lunch, etc. SITE VISIT SCHEDULE The Practical Guide for Evaluators - good outline of how the schedule should be arranged. Your schedule may vary depending on the availability of administrators, advisory groups, etc. Build in time for the team to regroup and discuss any additional items that have yet to be covered or need clarification. WORKING WITH FIRST-TIME EVALUATORS Make sure they know you are available for questions and assistance. Provide examples of how you conduct your review. Explain that institutions provide examples of how they meet criteria in various areas of the self-study, not necessarily where they are asked to address the specific criteria. WORKING WITH FIRST-TIME EVALUATORS Do they understand what has been provided or need guidance? Reinforce the importance of following the schedule and meeting deadlines. Suggest wording of questions if they are struggling with getting information. Review the comments early in the process to see if the comments sufficiently address the standard. PREPARING THE FEEDBACK REPORT Executive Summary – no more than three pages – cover the major strengths and OFI’s. The feedback report should Be written in complete sentences with correct grammar and punctuation. State that “there is no evidence” rather than “it is not clear”. Any questions regarding a standard or criteria should be clarified during the site visit. FEEDBACK REPORT, CONT. The feedback report should Use the same style and font throughout the document. After comments are copied and pasted, make sure format is the same. Be reviewed by the entire team with a final edit by the chair prior to submitting to ACBSP. Please complete the final review after returning home and submitting two to three days after the site visit, but no later than 10 days following the visit. FEEDBACK REPORT, CONT. The feedback report should not indicate an OFI if evidence of meeting a standard or criteria was provided on site even though not covered in the self-study. It is an OFI if the business unit does not provide evidence that the standard or criteria was met. Example: While the business unit did not provide information about Standard Four, Criteria 4.4 in the self-study, some evidence of improvements were provided on site for accounting, but no improvements were observed in the management discipline. FEEDBACK REPORT CONTINUED: THINGS TO REMEMBER: Comments are based on the requirements of the standards and criteria, not personal opinions or how processes are followed at other institutions. Comments should be clear and concise and should address the situation, how the business unit does (for strengths) or does not meet (for OFI’s) the standard, and explain the significance of the situation. Comments should not be prescriptive unless you are using language directly from the standards and criteria. Comments should be written in complete sentences and should use proper grammar and punctuation. Avoid the use of slang or local expressions others may not be familiar with, especially when on international visits. Example: “pig in a poke”. FEEDBACK REPORTS - STRENGTHS: If the institution has a strong strategic plan that includes both short and long-term goals and has a good approach, has been deployed or implemented, demonstrates learning, and identifies improvements, your comment could state: “The John Doe School of Business has a strategic plan that has identified both short and long-term goals. The plan is systematic with an approach that has been implemented for all disciplines, provides examples of learning and evidence of improvements the school has made. The strategic plan helps the school to plan for increasing enrollment and the replacement of faculty members as they retire.” WRITING A STRENGTH: If the institution has an assessment instrument and provides evidence of a systematic process and examples of how the results are used to make improvements, you could provide a comment such as: “The John Doe School of Business has administered the MFT test for all disciplines for five years. Each department documents their review of the pre and post results each semester during department meetings and makes improvements based on the results. For example, the accounting department added additional projects for preparing financial statements in the ACCT 3300 course.” FEEDBACK REPORTS - OPPORTUNITY FOR IMPROVEMENTS (OFI’S): Suppose the institution has identified: - increasing enrollment and a large number of faculty scheduled for retirement as strategic challenges. - strategic objectives in the strategic plan are for one to three years, and faculty are not consulted in the development of the objectives. The opportunity for improvement could read, “While the John Doe School of business has strategic objectives for up to three years, there was no evidence of planning beyond three years or involvement by faculty in the development of the objectives. Developing long term goals and involving more stakeholders such as faculty could provide resource planning to meet the increasing enrollment and replacement of faculty members scheduled to retire in the next five years. WRITING AN OPPORTUNITY FOR IMPROVEMENT: Suppose the institution administers the Major Field Test (MFT) to all students as a pre and post test. There is no evidence, either in the self-study or on-site, that the results are reviewed to indicate that there is learning or subsequent improvements are made. In addition, the school does not use the data for comparative purposes Your opportunity for improvement comment could read: “The John Doe School of Business provided three years of data for the Major Field Test (MFT). However, there is no evidence that the school is using this data to measure how their students are performing in each discipline and how the results compare to other institutions with a similar mission. Analyzing the results to identify program strengths and weaknesses could allow the school of business to make improvements where needed and provide an external source of comparative data.” FEEDBACK REPORT, CONT. Make sure your comments are written so that the business unit understands why an OFI was provided. There is a big difference (conditional accreditation v deferral) between providing an OFI because the business unit does not have results from an assessment and having results, but not using the results to make improvements. FEEDBACK REPORT, CONT. Once all team members have reviewed the feedback report in the entirety and it has been edited with a spell-check and checked for grammar, Email the report as a Word document to: [email protected] Please request a read receipt so you know it was received. We do not need a hard copy. SUBMISSION OF EXPENSES, ETC. Submit expense form and all receipts as soon as possible following the site visit. This can be sent to Mary Riley via email at [email protected] or fax at 913-339-6226. Payments are processed on the 15th & 30th each month. Submit evaluator agreement forms and evaluation forms via email at [email protected] or fax to 913-339-6226. BEST PRACTICES – WHAT DO YOU DO? Please share practices you have found to be helpful on site visits. WRAP-UP Final Questions Thank you for giving your time and helping ACBSP continuously improve our processes.