Error Rates and Random Match Probabilities (RMP) Based on the RUGER 10-Barrel Test and the GLOCK Cartridge Case Tests James E. Hamby, Ph.D., David J. Brundage, M.S., Steven A. Norris, B.S., Nicholas D.K. Petraco, Ph.D., and James W. Thorpe, Ph.D. James E. Hamby, Ph.D., International Forensic Science Laboratory & Training Centre, Indianapolis, Indiana David J. Brundage, M.S., Forensic Science Consultant, Nashville, Tennessee James W. Thorpe, Ph.D., University of Strathclyde, Glasgow, Scotland (Retired) A total of 240 test sets were produced by Dave Brundage, Mickey French and Jim Hamby – all then employed at the Indianapolis-Marion County Forensic Laboratory, Indianapolis, IN Recently 20 additional test sets were produced by Hamby and 10 sets sent to Europe and other countries for examination. The various test sets have used 4 different types of 9mm ammunition. Of the 626 participants in this research project, nine have used some type of ‘ballistics’ imaging equipment. Of the 626 participants in this worldwide project,9,382 of the possible 9,390 unknown bullets were identified. (15 unknowns x 626 participants = 9,390) Three bullets were mutilated during test firing (tank rash) and five ‘unknown’ bullets could not be associated to the known bullets by two young examiners-in-training - listed as inconclusive. This worldwide research project and numerous other research projects reported in the scientific literature – the AFTE Journal, the Journal of Forensic Sciences, and others - since the early 1930’s should provide data that will obviate some of the concerns within the legal community especially in the United States. • We request that those members who haven’t participated ‘take’ the test in their laboratory & submit their answers! • We also request that you return the set if you are no longer using it as there are labs that have requested test sets. Some labs may have two sets in their possession. 13 Modified SWGGUN Power Point Using the excellent SWGGUN power point, I have made a few changes that work well when testifying especially in Frye or Daubert Hearings. The following are a few of the changes. Our beloved AV man–Mike–has the power point if you are interested. Definition: Subjective Examination The results of a subjective examination are based on an individual’s opinion based on his/her training and experience. This does not mean that this type of examination is unreliable or unscientific. There is subjectivity in every science and in every test, whether it is a doctor diagnosing a head cold, a physician examining an x-ray or a cytotechnologist examining slides for cancer cells (pap smears). • National Institute of Justice, Washington, D.C. • NIJ - in conjunction with the National Forensic Science Technology Center (NFSTC) – funded a distance learning program titled ‘Firearms Examiner Training Program’ to augment training for new examiners, • Federal Bureau of Investigation (FBI) • Since 1986, the FBI has offered a one week course titled ‘Specialized Techniques in Firearms Identification’. The course has been attended by several hundred firearms examiners. • Bureau of Alcohol, Tobacco and Firearms (ATF) • Since 1999, the ATF has offered a one-year long course titled the ‘National Firearms Examiner Academy. To date, 140+ firearms examiners have attended the NFEA. (Funded by ATF and NIJ) • California Criminalistics Institute (CCI), CA DOJ • For several years, CCI has offered a variety of specialized courses in firearms and toolmark related fields. • United Nations Office on Drugs and Crime (UNODC) • • • • In 2013, the UNODC – with funding from the Canadian Government – is developing a limited service forensic laboratory for the Palestinian Civil Police (PCP) Temporary Forensic Science Laboratory (TFSL) in Ramallah, Palestine.. The TFSL will have the following forensic specialties: Forensic Chemistry – Drug and Arson Analysis Questioned Documents Analysis Firearms and Tool Mark Identification • United States Department of State • In 2014, the US Department of State – through the Bureau of International Narcotics and Law Enforcement Affairs, US Embassy, Belize, funded a program to strengthen the Belize National Forensic Science Laboratory by providing technical assistance related to the handling, processing and analysis of ballistics (firearms / tool marks) related evidence. • Other programs such as ICITAP teach firearms and tool mark Identification around the world. Firearm & Toolmark Identification meets the reliability standard put forth by both Daubert or Frye requirements. The science has been accepted in every state and Federal court that uses Daubert or Frye. Examples: A recent (9-11) Indiana Supreme Court Decision (Frye) upheld the reliability of Firearms and Tool Mark Identification; and a recent (12-11) Kansas District Court Decision (Frye) upheld ‘general acceptance’ in the field. A recent (7-13) Florida Court Issued a Daubert decision upholding the science. Cases are available for review on the SWGGUN website (under ARK) at www.swggun.org. • The Good News: o 10-Barrel Test: # examiner errors committed = 0 o GLOCK Cartridge Case Test: # of falsely matched cartridge cases = 0 THIS SHOULD BE GREAT! We WANT error rates of 0% for good classification systems!! • The Bad News: o You have to deal with the U.S. court system o 0% error rates present an opportunity to muddy waters with the “unrealistic study” criticisms o Little court interest in understanding the technicalities of estimating small error rates. So what can we do? • 0% error rate is the “frequentist” estimate o “Bayesian” statistics provide complementary methods o Can work much better in estimating small probabilities o We looked to sports statistics for low scoring games • For 10-Barrel we need to estimate a small error rate • For GLOCK we need to estimate a small random match probability (RMP) • Use Bayesian “Beta-binomial” method when no “failures” are observed (Schuckers) • Basic idea of the reverend Bayes: Prior Knowledge × Data = Updated Knowledge a Error Rate/RMP = a+b U(a,b) × Beta-Binomial(data | a,b) Posterior(a,b | data) Get updated estimates of Error rate/RMP • So given the observed data and assuming “prior ignorance” o Posterior error rate/RMP distributions: Posterior Distribution of RMP, 1632 Cartridge Cas Distribution of Average Examiner Error Rate for 626 Participants Posterior Dist. GLOCK 100 300 RMP 0.000086% [0.0000020%, 0.00031%] 0 Frequency 100 200 300 Average Examiner Error Rate 0.011% [0.00023%, 0.040%] 0 Frequency 400 Posterior Dist. 10-Barrel 0.00 0.02 0.04 0.06 0.08 Error Rate (%) 0.10 0e+00 2e−04 4e−04 RMP (%) 6e−04 8e−04 # of # ofParticipants Participants 2e−04 4e−04 GLOCK 0e+00 300 400 500 600 700 800 900 Estimated AverageRMP Examiner (%) Error Rate (%) 0.020 Actual: 626 participants 0.010 Error Rate Examiner Estimated Examiner Error(%) Rate (%) 10-Barrel Actual: 1632 cartridge cases 1000 2000 3000 4000 5000 6000 Cartridge Cases # Cartrige#Cases Examined with NoExamined Errors in Identifica With No Errors in I.D.