Report

Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization Tien-Duy B. Le, Ferdian Thung, and David Lo School of Information Systems Singapore Management University 1 Spectrum-Based Fault Localization • Locating buggy program elements by – Analyzing two sets of execution traces • Normal traces and faulty traces – Assigning suspiciousness scores to program elements • Two well-known SBFL formulas – Tarantula – Ochiai 2 Spectrum-Based Fault Localization • Xie et al. “A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization” (TOSEM, 2013) – Two families of SBFL formulas • ER1 and ER5 (5 formulas in total) • Theoretically proven to outperform Ochiai and Tarantula – Under the assumption: test coverage is 100% 3 Our Goal Theoretically Best SBFL Formula by Xie et. al. vs. Benchmark Programs ? Popular SBFL Formula 4 Notations Notion ns (e) n f (e) ns (e ) n f (e ) nf ns Description Number of successful test cases that execute e Number of failing test cases that execute e Number of successful test cases that do not execute e Number of failing test cases that do not execute e n f n f (e) n f (e ) ns ns (e) ns (e ) 5 Popular SBFL Formulas • Tarantula n f ( e) Tarantula(e) nf ns ( e ) n f ( e ) ns nf • Ochiai Ochiai (e) n f (e) n f ( n f (e) ns (e)) 6 Theoretically Best SBFL Formulas 7 Dataset • 10 programs, 199 faulty versions – Siemens test suite – Space, NanoXML, XML-Security • Evaluation Metric Rank of thefirst faultyprogramelement EXAM score T otalnumber of executableprogramelements – The lower the EXAM score, the better the performance 8 Results Technique Tarantula Ochiai ER1a ER1b ER5a ER5b ER5c Average % Inspected 23.37% 21.02% 33.34% 21.09% 43.04% 43.04% 54.95% Standard Deviation 23.44% 21.96% 35.22% 19.48% 19.63% 19.63% 26.83% • Ochiai has the lowest EXAM score (21.02%) 9 Results Technique Tarantula Ochiai Average % Inspected 23.37% 21.02% Standard Deviation 23.44% 21.96% ER1a ER1b ER5a 33.34% 21.09% 43.04% 35.22% 19.48% 19.63% ER5b ER5c 43.04% 54.95% 19.63% 26.83% • Tarantula‘s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas 10 Results Technique Tarantula Ochiai Average % Inspected 23.37% 21.02% Standard Deviation 23.44% 21.96% ER1a ER1b ER5a 33.34% 21.09% 43.04% 35.22% 19.48% 19.63% ER5b ER5c 43.04% 54.95% 19.63% 26.83% • Wilcoxon signed rank test (significance level of 0.05) Ochiai is statistically better than ER5a, ER5b, ER5c 11 100% Test Coverage Assumption • For 135 out of the 199 faulty versions – Test coverage < 100% • Average test coverage of the 199 versions – 84.97% Theoretically best SBFL formulas cannot outperform popular SBFL formulas 12 Conclusion • We conduct an empirical study on 10 programs with 199 versions – Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai • We find that: – Ochiai outperforms all theoretically best formulas – Tarantula outperforms 4 out of the 5 formulas – Assumption of 100% test coverage is not valid in many cases 13 Future work • In-depth study how test coverage and other factors affect effectiveness of SBFL formulas • Theoretically analyze performance of SBFL formulas – Assumption: test coverage < 100% 14 Thank you! Questions? Comments? Advice? {btdle.2012, ferdiant.2013, [email protected] 15