Developing Multiple Choice Questions David K. Coats, MD OKAP Committee Chair Texas Children’s Hospital in Houston The Ophthalmic Knowledge Assessment Program (OKAP) is an in-training ophthalmology examination offered annually by the American Academy of Ophthalmology for ophthalmology residency programs in the United States and Canada. Dr. Coats has disclosed the following financial interests: David J. Wilson, MD WQE Chairman Stacey Samuels Program Manager, Assessment and Evaluation Clinical Education Division of the AAO The intention of this tutorial is to educate item writers on acceptable formats, good technique, and what to avoid when writing questions. STEM Correct Response Which of the following phenomena increases as a pinhole aperture gets smaller? A. Diffraction B. Chromic aberration C. Polarization D. Spherical aberration Distractors Category 1: Memory or Knowledge Recall Category 2: Interpretive Category 3: Decision Making and Clinical Management Examples of each type of question are provided in the training copy of this presentation Memory or Knowledge Recall A Hudson-Stahli line represents iron deposition in which of the following corneal layers? A. Bowman’s membrane B. Descemet’s membrane C. Epithelium D. Stroma INTERPRETIVE The lesion shown in Figure 1 is most consistent with a diagnosis of: A. chronic viral infection. B. hemangioma C. Kaposi sarcoma. D. recurrent conjunctival dysplasia. INTERPRETIVE Three months after sustaining blunt ocular trauma and associated hypotony, a patient experiences sudden onset of pain with corneal edema. Which of the following is the most likely cause of this condition? A. Corneal decompensation from acute endothelial dysfunction B. Late bleeding and hyphema from injured iris vessels C. Neovascular glaucoma D. Spontaneous closure of an occult cyclodialysis cleft Decision Making and Clinical Management Three weeks ago, the parents of a 2½-year-old girl noticed intermittent esotropia in the child's right eye. Examination reveals 20 prism diopters of esotropia at 6 ⅓ m. Which of the following is the most appropriate initial therapy? A. Alternate occlusion B. Base-out, 10 PD Fresnel prisms on plano spectacles C. Recession of the medial rectus muscles D. Spectacles for correction of hyperopia Decision Making and Clinical Management (can include visuals) A 24-year-old woman experiences sudden loss of vision in her right eye with a visual acuity of 20/400. She has a right afferent pupillary defect, optic disc swelling, and a central scotoma. Her left eye is normal. A complete neurological history and physical examination reveal no abnormalities. Which of the following is the most appropriate next step? A. Administer intravenous methylprednisolone alone B. Administer intravenous methylprednisolone, followed by oral prednisone C. Perform CT scan of the orbit D. Perform MRI of the brain The ABO’s secure website contains a visual library of images available to its item writers You may upload images that will become part of the question directly onto the website • Be sure to include a diagnosis and a description for uploaded visuals • Once submitted, visuals are copyrighted and should not be used or have been previously used in a publication or lecture materials. You will be asked to complete a copyright. This allows the AAO unlimited use of the images in its educational programs. If you have concerns about unlimited copyright, work with your staff liaison to explore alternatives. You can post your images directly to a secure area of the AAO’s ftp site. Staff will then catalog the image as appropriate. Due to the new “patient privacy” and HIPAA regulations, it is essential that you have a signed release form (see Release Form) if you are submitting published material or photographs in which patients may be identifiable. This applies to photographs taken after April 15, 2003. Release forms from your particular institution also are acceptable. Images such as photographs, diagrams, or visual field results can enhance the quality of your questions The ABO and AAO encourage use of images when they contribute required clinical information necessary to answer the question Images should clearly illustrate only the condition or finding described Two examples with images are provided in the training copy of this presentation. The 60-year-old woman with the findings shown has pericentral scotoma in the right eye 2 months after combined cataract and glaucoma surgery. Which of the following is the most likely diagnosis? A. Age-related macular degeneration B. Cytomegalovirus retinitis C. Histoplasmosis D. Photic maculopathy The visual provided is an accurate representation of the entity helps to improve the validity of the examination. The 60-year-old woman with the findings shown has pericentral scotoma in the right eye 2 months after combined cataract and glaucoma surgery. Which of the following is the most likely diagnosis? A. B. C. D. Age-related macular degeneration Cytomegalovirus retinitis Histoplasmosis Photic maculopathy While the stem is well written and the distractors are plausible, the photograph is not a good representation of photic maculopathy. The stem or question must provide a sufficiently clear and focused description of the problem/case scenario for a candidate to make a decision. The stem should include all clinical information necessary for the candidate to make an unequivocal choice about the correct response. The information should assure that there is only one, clear correct response. No excess verbiage irrelevant to the question. a knowledgeable candidate should be able to answer the question before reading the responses Category I questions should be clearly stated and have clinical significance Category II and III questions should consist of: A description of the clinical findings, patient symptoms, or other specific data that provides background for the question (clinical scenario), and A question that details specifically what is being asked As you develop the question, you have a clear idea of exactly what the correct response will be. A clear question should be presented at the end of a clinical scenario Good lead-in questions include: • Which of the following is the most likely diagnosis, cause, complication? • Which of the following procedures, tests, treatments is most appropriate? Good Category 1, 2, and 3 stems are provided in the training copy of this presentation Which of the following is an early feature of choroideremia (progressive tapetochoroidal dystrophy)? Which of the following treatments is most likely to minimize the risk of epithelial keratitis in a patient with herpes simplex infection? The histopathology report on a specimen of posterior retina indicates thinning of the nerve fiber layer and loss of ganglion cells, but the inner nuclear layer, outer nuclear layer, and photoreceptors are intact. What is the most likely diagnosis? Eosinophils are found in conjunctival scrapings from a patient who has symblepharon. What is the most likely diagnosis? After 1% atropine drops are instilled into the eyes of a blue-eyed, 3-year-old boy, he becomes hyperpyretic, disoriented, and hyperactive. Which treatment options are most appropriate? A patient who undergoes complicated intraocular surgery with an open lens capsule develops secondary open-angle glaucoma that requires a combination of a beta blocking agents, topical carbonic anhydrase inhibitors, and latanoprost to control. Two months after surgery, vision has decreased to 20/200. Which of the following is the most likely cause of the decrease in vision? A 70-year old with open angle glaucoma in both eyes presents with pain, redness, and decreased vision in his right eye for 2 days. He is using no medication in that eye. Two years ago a trabeculectomy was performend on the right eye, with mitomycin C. His best corrected visual acuity is 20/200, and slit lamp exam shows 3+ injection with a seidel positive bleb and yellowish discharge. There is a 1+ cell and flare in the interior chamber. What is the most important condition to rule out? Trivia, esoteric information clinically irrelevant information Poor gauge of knowledge or clinical competence The stem is an open-ended statement Unfocused (True/False or Global) Negatively worded Examples of Poor Question Stems are provided in the training copy of this presentation Which of the following medications can cause palinopsia? A. Amitriptyline B. Doxycycline C. Gabapentin D. Trazodone Problem: TOO ESOTERIC Extremely difficult for candidates to answer correctly. A 5-year-old boy has acute and chronic polyarthritis, recurrent fever, pericarditis, rash, and hepatosplenomegaly. What is the probability that iridocyclitis will develop? A. Negligible B. 25% C. 50% D. 90% Problem: NOT CLINICALLY RELEVANT The chances that the child will develop iridocyclitis are not clinically relevant. Would knowing the % change the candidate’s treatment of this patient? Which of the following best describes the Landolt ring? A. A device for stabilizing the lens capsule during cataract surgery B. A measuring tool for determining the corneal diameter C. An optical phenomenon that results from diffraction D. An optotype for measuring visual acuity Problem: POOR GAUGE OF KNOWLEDGE OR CLINICAL COMPETENCE The question would be better if it focused on the use of the Landolt ring Instead of asking for the description of the Landolt ring. This question could be improved by asking: The Landolt ring is best used in the measurement of which of the following visual functions? 1. 2. 3. 4. 5. 6. 7. The interference property of light: Statin use: Viruses: Lung cancer is: Which of the following statements is likely regarding XXX? Which best describes a common occurrence in a patient who XXX? What would be a common practice regarding XXX? Problem: THESE STEMS ARE OPEN-ENDED Whether open ended or in question format , unfocused stems do not provide candidates with enough information to formulate a response before reading the choices. A well-constructed stem allows knowledgeable candidates to look for the correct response. Which of the following statements regarding lattice degeneration is true? A. It is equally common in emmetropes and myopes. B. It is found in 20-30 % of all eyes that present with a rhegmatogenous retinal detachment. C. It frequently is associated with peripheral retinoschisis in children. D. It is nearly always bilateral. Problem: True/False The answers are a list of statements following an unfocussed stem. A, C, and D may be related to frequency, so rewriting the stem to ask about frequency might help. 1. 2. Which of the statements about the reflex of the retinoscope is true? Which of the statements about a patient with mixed astigmatism is correct? 3. Identify which statement is accurate with regard to closed-angel glaucoma: 4. Compared to essential blepharospasm, hemifacial spasm usually: Problem: UNFOCUSED (TRUE/FALSE OR GLOBAL) If the correct response and distractors are a series of true/false statements, the stem most likely is unfocused. Epithelial keratitis is less likely to occur after a herpes simplex infection if the active infection is treated with: A. minimal wiping debridement and topical trifluridine drops. B. topical trifluridine drops. C. topical trifluridine drops and vidarabine ointment. D. none of the above. Problem: Negatively-worded “Less likely” is an example of negative question writing. Additionally, there is NO correct response. In reality, A, B, and C all are correct treatments. This type of question is confusing for candidates. Each answer choice should rarely exceed more than 1 line Listed in logical, alphabetical, or numerical order unless doing so gives away the correct answer Have both the correct response and the distractors balanced in their plausibility, length and amount of detail All responses are parallel, e.g., all tests, all treatments Wrong answers should reflect faulty reasoning or misconceptions. There should only be 1 defensible answer. Content that is controversial or that varies regionally should be avoided in order to eliminate the possibility of more than one correct response. Incorrect answers should be attractive to at least some candidates Question Stem: What is the most appropriate initial management of intraoperative bleeding at the site of a neodymium: yttrium aluminum garnet (Nd:YAG) laser iridotomy? A. compression of the eye with the laser iridotomy lens B. pharmacologic pupillary dilation and cycloplegia C. cauterization of the vessel with the Nd:YAG laser D. bed rest and topical corticosteroid eyedrops “All of the Above” /“None of the Above” are unacceptable to the ABO and the AAO attempt to test too many areas at once, are confusing, and when a candidates gets the question wrong, it’s difficult to determine which aspect of the question they did not know. Examples of poor distractors are provided in training copy of this presentation Ocular conditions associated with hypertension included which of the following? A. Retinal vein occlusion B. Ischemic optic neuropathy C. Glaucoma D. All of the above Problem: In addition to having an unfocused stem, “All of the above” and “None of the Above” are unacceptable answer choices. Similarly, “Part I only” and “Part II and III only” types of answers are unacceptable. Which of the following characteristics distinguishes congenital fibrosis syndrome from double elevator palsy? A. Convergence on attempted elevation B. Hypotropia in primary gaze increasing on upgaze C. Positive forced ductions to elevation D. Ptosis or pseudoptosis Problem: There is more than 1 defensible answer. Although choice A is marked as the preferred response, answer D is arguable (ptosis or pseudoptosis is also correct.) What should treatment of acute dacryocystitis ALWAYS involve? A. Systemic antibiotics B. Orbital imaging C. Immediate nasolacrimal duct probing D. Lacrimal sac massage Problem: Answers choices are not homogenous or parallel. All the answer choices should be tests, symptoms, or differential diagnoses, for example. Unparallel answer choices are a clue that the stem needs to be focused. Threshold retinopathy of prematurity (ROP) disease is defined as which one of the following? A. Five contiguous clock hours of Stage 3 or 8 total clock hours of Stage 3 in Zone I or II with plus disease B. Five contiguous clock hours of Stage 2 or 8 total clock hours of Stage 2 in Zone I or II with no plus disease C. Five contiguous clock hours of Stage 3 or 8 total clock hours of Stage 3 in Zone II or III with plus disease D. Five contiguous clock hours of Stage 3 or 8 contiguous clock hours of Stage 3 in Zone I or II Problem: Answer choices are too long. Answer choices should rarely exceed 1 line. If the answers are too long, it takes a disproportionate amount of time for the candidate to answer a question even in areas which they are knowledgeable. Which of the following pairs best describes the outflow of aqueous humor? Problem: Attempts to test too many areas The question is complex in format and difficult to answer. Moreover, if candidates get the question wrong, it’s difficult to determine which aspect of the question they did not know. Implausible or silly distractors can be easily dismissed, increasing ability to guess the correct response the same words appear in both the stem and one answer choice. Incorrect answers can be too obvious (e.g. ‘smoke more cigarettes’) One answer is longer than the others One answer is grammatically different Mutually exclusive responses help the candidate, because if you know one response is false, the other must be true. Ambiguous or indefinite terms like might, may, sometimes and absolute terms like never, always Scenarios almost always “might” or “may” be true. Especially in the absence of qualifiers in other answers, vague terms are a clue to the correct response. The reverse is true of absolute terms, where scenarios can be ruled out as incorrect since they almost always may be true. Examples of questions that have clues to the correct response are provided in the training copy of this presentation. Which of the following lesions is most likely to cause a retinal hole or tear leading to retinal detachment? A. Cyst in the pars plana B. Cystic retinal tuft C. Enclosed oral bay D. Peripheral cystoid degeneration Problem: the same words appear in both the stem and one answer choice. Which of the following is an appropriate alternative to the use of cyanoacrylate glues in the management of corneal perforations? A. Aminocaproic acid B. Autologous plasmin C. Erythropoietin D. Fibrinogen and thrombin glue Problem: The incorrect answer choices are too obvious. While D is not a proven “appropriate alternative”, the other distractors are so obviously incorrect that they lead the candidate to the correct response. Pupillary block and iris bombé: A. occur in reverse in pseudoexfoliation syndrome. B. are absolute in primary angle closure glaucoma. C. are common in neovascular glaucoma. D. may be compartmentalized in pseudophakic eyes due to localized adhesions between the intraocular lens surface and iris pigment epithelium, not all of which will respond to a single iridectomy. Problems: longer, grammatically different, vague terms. In addition to an unfocussed stem, the correct answer is much longer and grammatically different than the other answers. Further, it contains the vague term “may be”. Which one of the following is true about acquired limbal stem cell deficiency? A. It can be caused by preservatives in topical medications. B. It is always diffuse and not focal. C. It can never be treated by limbal stem cell transplantation. D. It is never seen in graft-versus-host disease. Problem: absolute terms “always” and “never” In addition to a true/false stem, the incorrect answer choices all contain absolute terms. This allows the candidate to eliminate answers B, C, and D. In the treatment of herpes simplex stromal keratitis without epithelial keratitis, oral acyclovir: A. alone is more effective than topical corticosteroids alone. B. provides no beneficial effect when added to topical corticosteroids. C. speeds resolution of keratitis when added to topical corticosteroids. D. is less effective than topical trifluridine. Problems: Mutually excusive answer choices B and C. In addition to an unfocused stem, B and C are mutually exclusive. If a candidate knows one is false, the other must be true. The globe shown was enucleated 6 weeks after cataract extraction, which was followed by rapid growth of a subconjunctival mass at one end of the cataract incision. Higher magnification of the cataract incision is shown. Which of the following is the most likely diagnosis? A. Endophthalmitis B. Metastatic tumor from an extraocular site C. A primary conjunctival tumor invading the eye D. Previously unrecognized iris and ciliary body melanoma extending through the cataract wound The globe shown was enucleated 6 weeks after cataract …. A. Endophthalmitis B. Metastatic tumor from an extraocular site C. A primary conjunctival tumor invading the eye D. Previously unrecognized iris and ciliary body melanoma extending through the cataract wound Problem: Correct response and distractors should be approximately the same length and contain the same detail have repeated phrases in the distractors contain unnecessary information use unnecessary medical jargon, slang, eponyms, and/or abbreviations Examples and exercises to help you improve questions are provided in the training copy of this presentation Which of the following is most likely to be associated with crystalline lens dislocation and/or subluxation? A. Aniridia B. Essential iris atrophy C. Chandler’s syndrome D. Cogan-Reese syndrome Problem: C and D are eponyms. In order to include common eponyms, reformat to : • iridocorneal endothelial syndrome (Chandler’s) • iris nevus syndrome (Cogan-Reese) A patient undergoes a difficult phacoemulsification. An opening is made in the posterior capsule, necessitating a limited anterior vitrectomy and placement of the intraocular lens implant in the sulcus. The postoperative vision is 20/40, but a secondary open-angle glaucoma develops that requires a combination of a beta blocking agent, topical carbonic anhydrase inhibitors, and latanoprost to control the intraocular pressure. Two months after surgery, vision has decreased to 20/200. What is the most appropriate next step? Problem: This stem contains unnecessary information. Rewrite this stem and compare your effort to the up-side down re-write provided on the next slide. Which of the following therapies is most likely to be associated with pseudopemphigoid? A. B. C. D. Topical administration of cromolyn sodium Topical administration of gentamicin Topical administration of timolol Topical administration of trifluridine Problem: This question contains repeated phrases. Rewrite this question. Compare your effort to the up-side down re-write provided there. Negative question stems ask what is false, not, least, or the exception to the rule. do not perform well. Statistics show that knowledgeable test takers perform poorly on negatively worded questions are not useful for assessment and act much like “trick questions” Negative The question stems ABO does not use negative questions The AAO is slowly eliminating negative questions from its programs and will not accept new, negative items “Contraindicated” is an acceptable use of a negative only when testing clearly contraindicated medications or procedures. Questions should not be intentionally misleading “Trick” questions are not good. They tend to test savvy or reading skills instead of clinical competence Examples of unacceptable questions follow this slide in the training copy of this presentation. Which of the following agents is LEAST effective in the treatment of ocular disease? A. Thiabendazole B. Clindamycin C. Ganciclovir D. Acyclovir When planning injection of an eyelid hemangioma in an infant, which of the following systemic consequences must be considered? A. Adrenal insufficiency B. Central retinal artery occlusion C. Fat necrosis D. Skin necrosis This is a trick question because B also is correct, although not “systemic” as stated in the stem. In the following examples, determine whether the format shown is acceptable or unacceptable for ABO examinations Which of the following is true regarding the oculomotor nucleus? □ Acceptable □ Unacceptable This question does not give candidates any idea what the correct response might be until all of the choices are read. Unacceptable. A persistent, clear fluid encountered when removing bone superior to the medial canthal tendon during a dacryocystorhinostomy most likely would result from inadvertent penetration of which of the following structures? □ Acceptable □ Unacceptable This question provides sufficient information for a candidate to formulate a response and quickly find it among the choices. Acceptable. Which of the following histocompatability leukocyte antigens is least likely to be associated with retinal vasculitis of noninfectious origin? □ Acceptable □ Unacceptable This question asks candidates to identify what is NOT correct. Unacceptable. Which of the following findings is most likely to be associated with meibomian gland dysfunction? □ Acceptable □ Unacceptable This question directs candidates to look for the MOST likely finding. Acceptable. As compared to a normal rigid contact lens, a translating bifocal rigid contact lens does not have: □ Acceptable □ Unacceptable This question is both unfocused AND negatively worded. Unacceptable. Examples of items that are well-written. e.g., have good question stems and answer choices, are provided in the hard-copy of this hand-out after this slide. Prophylactic laser photocoagulation of asymptomatic lattice degeneration is most indicated for a patient with which of the following findings? A. High myopia B. Preoperative cataract C. Pseudophakia D. Retinal detachment in fellow eye Focused question that asks candidates to recognize the significance of retinal detachment in lattice degeneration Which of the following is the most likely diagnosis for the condition shown? A. Chloroquine retinopathy B. Cone dystrophy C. Dominant drusen D. Fundus flavimaculatus Interpretive question that tests candidate’s ability to recognize and diagnose conditions Which of the following is the most appropriate management of a corneal perforation that remains unsealed after 3 days of aqueous suppression therapy? A. Aminocaproic acid B. Autologous plasmin C. Cyanoacrylate glue D. Topical antibiotic This question tests whether candidates know the most appropriate accepted management for the problem. References are required for every American Board of Ophthalmology (ABO) exam item and for every OKAP exam item. References must be complete in order to claim CME credit for high-stakes examination item-writing √ BCSC section 9, 2006-07, p. 52 √ Mamalis n et al, journal of cataract and refractive surgery 2006; 32: 324-333. Unacceptable: BCSC Unacceptable: Shields, The Glaucomas Please speak to your staff representative from the ABO or AAO as appropriate if you write items for a high-stakes examination. They will provide you appropriate information .