An Analysis of Machine Learning Algorithms for Condensing Reverse Engineered Class Diagrams Presenter: Hafeez Osman Hafeez Osman ([email protected]) Michel R.V. Chaudron ([email protected]) Peter v.d Putten ([email protected]) Leiden University. The university to discover. OVERVIEW 1. Introduction 2. Research Question 3. Approach 4. Results 5. Discussion 6. Future Work 7. Conclusion Leiden University. The university to discover. Introduction Why ? Reverse engineered class diagrams are typically too detailed a representation What ? Simplifying UML Class Diagram: Based on Software Design Metrics using Machine Learning Who ? Software Engineer, Software Maintainer, Software Designer Leiden University. The university to discover. Leiden University. The university to discover. Introduction Aim: analyze performance of classification algorithms that decide which classes should be included in a class diagram This paper focusses on using design metrics as predictors (input variables used by the classification algorithm) Omit Leiden University. The university to discover. Introduction Explore Structural Properties of Classes • Software design metrics from the following categories : • Size : NumAttr, NumOps, NumPubOps, Getters, Setters • Coupling : Dep_Out, Dep_In, EC_Attr, IC_Attr, EC_Par, IC_Par Machine Learning Classification Algorithms • Supervised classification algorithms • J48 Decision Tree, Decision Tables, Decision Stumps, Random Forests and Random Trees • k-Nearest Neighbor, Radial Basis Function Networks • Logistic Regression, Naive Bayes, Leiden University. The university to discover. Research Questions RQ1: Which individual predictors are influential for the classification? For each case study, the predictive power of individual predictors are explored RQ2: How robust is the classification to the inclusion of particular sets of predictors? Explore how the performance of the classification algorithm is influenced by partitioning the predictor-variables in different sets. RQ3: What are suitable algorithms for classifying classes? The candidate classification algorithms are evaluated w.r.t. how well they perform in classifying the key classes in a class diagram. Leiden University. The university to discover. Approach Evaluation Method RQ1: Predictors Univariate Analysis – Information Gain Attribute Evaluator To measure predictive power of predictors RQ2, 3: Machine Learning Classification Algorithm Area Under ROC Curve (AUC) The AUC shows the ability of the classification algorithms to correctly rank classes as included in the class diagram or not Leiden University. The university to discover. Approach Case Study Characteristics Project Total Classes (a)/(b) = % Source code (a) Design (b) ArgoUML 903 44 4.87 Mars 840 29 3.45 JavaClient 214 57 26.64 JGAP 171 18 10.52 Neuroph 2.3 161 24 14.9 JPMC 127 11 8.66 Wro4J 87 11 12.64 xUML-Compiler 84 37 44.05 Maze 59 28 47.45 Leiden University. The university to discover. Approach Grouping Predictors in Sets No Predictor 1 2 3 4 5 6 7 8 9 10 11 NumAttr NumOps NumPubOps Setters Getters Dep_out Dep_In EC_Attr IC_Attr EC_Par IC_Par Predictor Set A Predictor Set Predictor Set B C x x x x x x x x Leiden University. The university to discover. Approach 1. Reverse engineer the source code to UML design. i. Eliminate library classes 2. Calculate design metrics i. Eliminate unused metrics 3. Merge the information “In Design” with the software design metrics data 4. Prepare set of predictors 5. Run all set of predictors with machine learning tool Leiden University. The university to discover. Result RQ1 : Predictor Evaluation Influential Predictors 7 6 No of Projects 5 4 3 2 1 0 EC_Par NumOps Dep_In Predictor 6 5 5 NumPub Dep_out NumAttr Setters Ops 4 4 3 3 Getters EC_Attr 3 3 IC_Attr IC_Par 3 2 ** Out of 9 projects Leiden University. The university to discover. Result RQ2 : Dataset Evaluation No. of Projects for which a Classification Algorithm scores AUC > 0.60 10 9 No of projects 8 7 6 5 4 3 2 1 0 Decision Table Prediction Set A 3 Prediction Set B 3 Prediciton Set C 3 J48 5 5 5 Decision Stump 6 7 6 RBF Network 7 7 7 Naïve Bayes 8 7 7 Random Tree 8 7 7 Function Logistic 7 7 8 k-NN(1) k-NN(5) 8 7 9 8 8 9 Random Forest 9 9 9 ** Out of 9 projects Leiden University. The university to discover. Result RQ3 : Evaluation on Classification Algorithms Average AUC Score 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 Decision Table Predictor Set A 0.60 Predictor Set B 0.59 Predictor Set C 0.58 J48 0.63 0.61 0.61 Random Tree 0.66 0.66 0.66 RBF Network 0.66 0.67 0.66 Decision Stump 0.66 0.64 0.65 Function Logistic 0.69 0.70 0.68 Naïve Bayes 0.70 0.70 0.69 k-NN(1) k-NN (5) 0.70 0.70 0.71 0.73 0.72 0.72 Random Forest 0.75 0.75 0.74 ** Out of 9 projects Leiden University. The university to discover. Discussion A. Predictor Three class diagram metrics should be considered as influential predictors: • Export Coupling Parameter(EC Par), • Dependency In (Dep In) • Number of Operation (NumOps) ** Means, a higher value of these metrics for a class indicates that this class can be a candidate for inclusion in the CD. B. Classification Algorithm k-NN(5) and Random Forest are suitable classification algorithms in this study. • Their AUC score is at least 0.64 • The classifiers are robust for all projects and predictor sets Leiden University. The university to discover. Discussion C. Threat to Validity i. Assumption of ground truth: Exactly all classes that should be in the forward designs are in the forward design. There is a possibility that : • some of these classes were not the key classes of the system. • there is a possibility that the forward design used is too ‘old’. i. Input is dependent on Reverse Engineering tool (MagicDraw) ii. Cover only 9 open-source projects Leiden University. The university to discover. Future Work 1. Alternative predictor variables • use of other type of design metrics ex. (semantics of) the names of classes, methods and attributes. • use source code metrics such as Line of Code (LOC) and Lines of Comments. • Change History of a class 2. Learning models (classification algorithm) • testing out an ensemble approach (combines classification algorithms) 3. Semi supervised or interactive approach 4. Compare this study result with other approaches • Other works that apply different algorithm such as HITS web mining, network analysis on dependency graphs and PageRank. 5. Validate understandability of abstract Class Diagrams Leiden University. The university to discover. Conclusion 1. The most influential predictors • Export Coupling Parameter • Dependency In • Number of Operation 2. Most suitable Classification Algorithms • Random Forest • k-Nearest Neighbor 3. Classification algorithms are able to produce a predictor that can be used to rank classes by relative importance. 4. Based on this class-ranking information, a tool can be developed that provides views of reverse engineered class diagrams at different levels of abstraction. 5. Developers may generate multiple levels of class diagram abstractions. Questions………….. Leiden University. The university to discover.