Changes in selection to the Foundation Programme Professor Paul O’Neill Chair, ISFP Project Group Member UKFPO Rules Group Lead for Research and Evaluation Selection Plan for Talk • Background to change – robust & numbers • Evidence around selection • SJTs • Evidence • Piloting • EPM • Algorithm • Academic FP Selection (appointment) Best prediction of the right person to do the best job Should be done fairly PON experience: • House jobs (Foundation) • CVs. application forms and letters • References & ‘putting a word in’ • Interview panels – numbers, questions • Presentations • Occupational pyschologists • Personality testing Now: • Mostly national • Application - anonymous, standardised and assessed against criteria • Use of standardised tests (e.g. SJT) • Selection centres – standardised, multimodal, some competency testing • Interview panels more & more standardised Evolution 2009 DH commissioned a review of selection methods. The Improving Selection to the Foundation Programme project was set up and overseen by the Medical Schools Council 2010/2011 New selection methods were piloted successfully FP 2012 Full-scale Parallel Recruitment Exercise (PRE) FP 2013 New selection methods implemented A collaborative venture between Scottish Foundation Board Foundation School Directors Educational Supervisors Selection in Medicine Undergraduate • Single School • UKCAT Postgraduate • Foundation • Postgraduate – Birmingham Review Substantive – Consultant & GP Literature Reviews 3 commissioned Warwick Newcastle Durham Overlapping Specify methods used Identify gaps and where judgement will be needed Durham: McLachlan & Turnbull • 236 References • Behaviour predictive of future behaviour • Conscientiousness significant component of concerns Warwick: Thistlethwaite et al • 197 papers 1990-2009, medical school and residency. • Most from USA • Mostly looked at face validity and reliability NOT predictive validity • Lack of consensus • Non-medical literature – focus on employability skills (job analysis) Newcastle: Illing et al • 202 references • Foundation programme – narrow range of applicants, very few cannot do the job Selection methods must satisfy their stakeholders (employers, students etc) Cognitive tests are moderately predictive of later cognitive tests Non-cognitive elements need to be considered to ensure that a doctor is able to perform well Assessment centres allow for a range of methods to be used • • • • Selection Methods Combination of: • Cognitive ability (academic) • Non-cognitive Academic range very narrow • High ‘justice’ Incorporate measures of big 5 personality applied to medicine • Not IQ or general aptitude tests Selection Methods Big 5 Model of Personality is predictive of job and academic performance • • • • Extraversion (outgoing, sociable, impulsive) Emotional stability (calm, relaxed) Agreeableness (trusting, co-operative, helpful) Conscientiousness (hardworking, dutiful, organised) • Openess to experience (artistic, cultured, creative) Assessment Criteria Reliability Validity Consistency Granularity Longevity Will the process pick the ‘best’ applicants across the UK? Will the process be good for many years? Assessment Criteria Educational Impact Applicant Burden Clinician Time Will the process support or undermine educational objectives? How onerous is the process for applicants ? To what extent does the process distract from service delivery? Assessment Criteria Compliance Will the process pick minimise cheating or malpractice? Transparency Can the applicants see exactly where the goal posts are? Fairness (Justice) Public Opinion Is there a level playing field? Person in the street thinks that it is fair Summary of Results Lower 5-year cost Better score Selection to FP 2013 Situational Judgement Test (SJT) • SJTs will replace the ‘white space’ application form questions • This is an invigilated, machine markable test in exam conditions • The SJT will consist of 70 questions in 2 hours 20 mins Educational Performance Measure (EPM) • The EPM will replace the academic quartile scores. The EPM score is comprised of three elements: 1. medical school performance in deciles 2. additional degrees 3. academic achievements The EPM and SJT will each be worth 50 points from a 100 point application score The case for change Concern SJT and EPM • ‘White space’ questions not sustainable as a selection tool • Will become steadily less discriminatory between eligible applicants (limited range of new questions that can be generated) • SJTs draw upon bank of items to be available for each application round •Situations experienced in the Foundation Programme varied and complex • New items built incrementally and continuously against Job Analysis • ‘White space’ questions non-invigilated conditions , model answers concerns about risk of plagiarism and coaching • SJTs in invigilated conditions in the UK (3 national dates) • Not possible to revise for the SJT (scenarios complex, answers relate to judgement rather than knowledge) The case for change Concern • Long-term technical reliability and validity could be improved SJT and EPM • 30 year evidence for reliability of SJTs • SJT pilots demonstrate the technical reliability, internal reliability, and validity for use for FP selection •Academic quartile system - difficult to compare fairly between applicants from different medical schools (not standardised or subject to quality assurance across medical schools) • EPM - standardised framework for deciles • Medical schools and students decide ‘basket of assessments’ • Schools will be required to publish their locally agreed deciles framework, which will facilitate transparency and quality assurance from the wider community. • Deciles fairer to applicants at margins Situational Judgement Test What is a Situational Judgement Test? SJTs are: • a test of aptitude • designed to assess the professional attributes expected of a Foundation doctor • based on a detailed job analysis of an FY1 doctor SJT questions assess your judgement by presenting you with challenging situations you are likely encounter at work during the first year of an integrated Foundation Programme Example SJT Questions There are two question formats: • Rank the five responses in the most appropriate order • Choose the three most appropriate responses from eight You should answer what you ‘should’ do in the scenario described, not what you ‘would’ do Example Question 1 – ranking Mr Reese has end-stage respiratory failure and needs continuous oxygen therapy. While you are taking an arterial blood gas sample, he confides in you that he knows he is dying and he really wants to die at home. He has not told anyone else about this as he thinks it will upset his family, and the nursing staff who are looking after him so well. Rank in order the following actions in response to this situation (1= most appropriate; 5= least appropriate). A. B. C. D. E. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital Reassure Mr Reese that the team will take account of his wishes Discuss his case with the multi-disciplinary team* Discuss with Mr Reese's family his wish to die at home Discuss Mr Reese's home circumstances with his General Practitioner Answer to Question 1 B. Reassure Mr Reese that the team will take account of his wishes C. Discuss his case with the multi-disciplinary team* E. Discuss Mr Reese's home circumstances with his General Practitioner D. Discuss with Mr Reese's family his wish to die at home A. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital This question is focusing on effective communication with patients. Ensuring that patients’ informed wishes are met in relation to their care is central to your approach to patient care and this needs to be communicated to the patient in a reassuring manner even in situations relating to end of life care (B). These wishes should have been sought when addressing the management plan for Mr Reese and once identified the multidisciplinary team needs to be made aware of them in order to ensure that as far as possible Mr Reese’s views in relation to his end of life care are implemented (C). The management of Mr Reese will require the active involvement of his GP and communication with the GP is therefore of importance (E). Any decision to discuss Mr Reese’s wishes in relation to his end of life care with his family can only be made with the full agreement of Mr Reese (D). It would not be appropriate to give the patient inaccurate information in order to engineer a different medical pathway (A). SJTs (Literature Review 77 papers) • Management, university, police, engineers • Large scale selection – short-listing • Construct validity not clearly identified • Single construct (e.g. ‘practical intelligence’) • Can be designed to measure differing constructs • Predictive validity will depend on what criterion is targeted • SJT designed to test interpersonal skills will more likely predict inter-personal orientated performance SJTs • Used nationally to select GP registrars and other ‘high stakes’ occupations • significant validity in predicting job performance • incremental validity over methods such as ability tests and personality questionnaires • typically relate to general experience and ability, rather than job-specific knowledge or experience • tend to show smaller differences between candidate groups (e.g. based on race) than cognitive ability tests Job analysis of FY1 doctor • • • • • • • • • Commitment to professionalism Coping with pressure Effective communication Learning and professional development Organisation and planning Patient focus Problem solving and decision-making Self-awareness and insight Working effectively as part of a team Job analysis of FY1 doctor Item development Item writing workshops Previously piloted & refined (43 items) CIT interviews (78 items) (89 items) Review, concordance, piloting Example Question 2 – multiple choice You have been prescribed codeine for persistent back pain which has become worse in the last few weeks. You have noticed that during shifts you are becoming increasingly tired, finding it difficult to concentrate and your performance, as a result, has been less effective. Choose the THREE most appropriate actions to take in this situation A. Ask a colleague to assist with your workload until you finish your codeine prescription B. Make an effort to increase the number of breaks during your next shift C. Stop taking the codeine immediately D. Make an appointment to see your General Practitioner E. Seek advice from a specialist consultant about your back pain F. Arrange to speak with your specialty trainee (registrar)* before your next shift and make them aware of your situation G. Seek advice from your clinical supervisor* regarding further support H. Consider taking some annual leave Answer to Question 2 D. Make an appointment to see your General Practitioner F. Arrange to speak with your specialty trainee (registrar) before your next shift and make them aware of the situation G. Seek advice from your clinical supervisor regarding further support This question looks at how you demonstrate commitment to professionalism and selfawareness. The essential problem is that as an FY1 doctor the level of your clinical performance is dropping. This constitutes a risk to the patients you are caring for and will impose a greater workload on your colleagues. In this circumstance you should inform and seek the advice of the senior clinician responsible for your work (G) and alert your colleagues (F). This matter is most likely to be related to your prescribed medicine and you should therefore consult with your GP (D) rather than any other specialist (E). It is not your place to re-allocate workload (A). Increasing the number of breaks is unlikely to improve a situation that is likely to be due to an adverse effect of a drug (B). You should not make any unilateral decisions about your medical treatment (C) and should seek the advice of others (D). You should not be seeking to use your annual leave (H) to compensate for a medical problem. Scoring of the SJT Scoring key, determined through: • Consensus at item review stage (item writers, SMEs) • Expert judgement in concordance panel review • Review and analysis of the pilot data Scoring not “all or nothing”, but based on how close to scoring key Scoring of the SJT – ranking • Up to 20 marks available • Up to 4 marks available for each response (points for “near misses”) • No negative marking Scoring of the SJT – multiple choice A B C D E F G H 0 4 0 0 0 0 4 4 • Points for each correct answer • No negative marking • 4 points for each correct answer • Up to 12 points per item Parallel Recruitment Exercise (PRE) • New selection methods trialled alongside the normal selection methods during 2012 FP application round • Aims: logistics, awareness, pilot new SJT content • All 31 medical schools involved • SJT – 1 hr, 30 questions • EPM – each medical school consulted study body on ‘basket of assessments’ to be used Parallel Recruitment Exercise (PRE) • 90+% of FP applicants participated in the SJT • Valuable learning experience ahead of live implementation • Feedback to inform live implementation: • Applicants • Medical schools • Was the final step in ensuring the selection methods can be consistently and robustly applied for implementation for FP2013 PRE - SJT 30 item, one hour SJT 6,842 medical students took part in the PRE Participants included: Sheffield SJT pilot • Final year medical students • Students who had been pre-allocated to the Defence Deanery • Students who had chosen to take a year out post-graduate • International students returning overseas after graduation 30 medical schools (plus two centres for Eligibility Office applicants) delivered the SJT in 72 venues Psychometric analysis shows that a 60 item SJT is a reliable measurement Descriptive Statistics Internal Reliability: • Adjusted for a 60 - item test that included only robust items (such as would be used in an operational paper), all papers had an estimated reliability of α = 0.80 or above (α = 0.80 to α = 0.87) • Demonstrates that the SJT is a reliable test in this context with items testing different things Histograms showing the distribution of scores for Paper One (left) and Paper Two (right) Descriptive Statistics Mean: • Overall Mean = 79.5%; Range from 78.0% to 80.6%. So not too easy – differentiating appropriately Standard deviations: • Mean SD = 18.6; Range from 17.3 to 20.0. Distributions: • 305 to 468 (out of a maximum of 512) – as expected given length of paper • Appears to be slightly negatively skewed, (more people towards top end) although results do show a close to normal distribution Item Facility (difficulty) Ranked Items Multiple Choice Items Maximum score 20 Maximum score 12 Score 18 = ‘very easy’; Score 10.8 = ‘very easy’; Score 11.6 = ‘very hard’ Score 3.6 = ‘very hard’ Mean facility similar across all papers (approx 16) Mean facility across papers ranged from Range of facility values differed across papers 7.9-9.0 SD range similar for all papers, except for Paper 1 where one item had a very high SD Range of facility values differed across papers SD range was similar for all papers Student views The content of the assessment seemed relevant to the Foundation Programme The scenario content seemed appropriate for my training level Educational Performance Measure Why change to EPM? • A clear framework with agreed principles used to calculate the EPM, ensuring that it is fair, transparent and consistent across the schools of the UK • Splitting cohorts into deciles rather than quartiles provides a wider spread of scores, which makes it easier to differentiate between applicants, and will be more fair for applicants at the margins • It makes more sense for all the academic components of the application to be one part of the application How is the EPM calculated? Score produced by applicant’s medical school to reflect achievement and performance compared to rest of cohort EPM = 3 parts (maximum 50 points): 1. Medical school performance in deciles (34 – 43 points) E.g. Top 10% = 43; Top 20% = 42; etc 2. Additional degrees (max 5 points) 3. Educational achievements (presentations, prizes and publications (max 2 points) Schools have consulted with students about how the decile points for the EPM will be calculated. PRE - EPM • 27 of 30 medical school initiated a new consultation and review of framework • (3 schools consult annually and framework aligns) • Other schools do consult annually – but undertook new consultation with students for the PRE • Majority of schools pleased with student engagement, especially amongst later years • Benefit of raising awareness with students & staff • Students felt sense of ownership Consultation on frameworks 20 18 16 14 12 10 8 6 4 2 0 Meeting student reps Committee Email to all students Online survey Open meeting Student reps Convened consulted working group PRE EPM – Decile Points Decile/ Quartile 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 4th 1 1 2 10 13 37 337 680 671 3rd 2nd 1st 3 5 13 52 625 629 348 30 7 2 18 341 649 606 50 19 10 8 2 604 646 334 27 8 2 3 PRE EPM – successes • All medical schools have agreed a ‘basket of assessments’ in consultation with students • All medical schools aligned with EPM framework • All medical schools calculated EPM deciles, with around 10% in each decile (some ties) • All medical schools confident they can calculate EPM deciles in line with common principles Academic Foundation Programme 2012 Same timetable as for standard application Apply to 2 UoA Have to sit SJT Appointed – application, EPM, interview (+/-) If not appointed, then revert to standard process Algorithms • Was initially triggered by student preferences • Unstable, unfairness • Worse if increasing number applicants/school • Changed for FP2012 • Now triggered by application score • Has changed patterns of applications Selection to the Foundation Programme – improving and evolving FP 2005 Foundation Programme introduced FP 2006 National timetable and application process FP 2007 Online application – white space & quartiles 2009-2011 Improving Selection to Foundation Programme FP 2012 Full-scale Parallel Recruitment Exercise (PRE) FP 2013 New selection methods implemented The greatest forward step in the baking industry since bread was wrapped – Missouri, 1928 ‘ What’s Wrong with SJTs Lawrence Clinical Pharmacology Wonderful Useful in some situations Not fit for my dog More information UKFPO - www.foundationprogramme.nhs.uk • • • • • FP 2013 Applicant Handbook Introductory videos SJT monograph SJT practice paper FAQs (FP 2013, SJT, EPM) Archived ISFP website – www.isfp.org.uk Any questions?