Report

Chapter 5 Overfitting and Its Avoidance 指 導 教 授 ： 徐 立 群 學 生 教 授 ： R16014101 陳怡齊 R16011234 吳年鑫 Overfitting 即「過適」、「超適」或稱「過度擬合」 意指在調適一個model時，使用過多參數。對比於可取得的資料總量 來說，一個荒謬的model只要足夠複雜，是可以完美地適應(fit)資料。 不合乎一般化 (Generalization) 違反奧卡姆剃刀（Occam’s Razor ) 原則 Overfitting & Generalization A extreme example – Customer churn or non-churn Training data & Holdout data Overfitting Examined • Holdout Data and Fitting Graphs A fitting graph shows the accuracy of a model as a function of complexity . Figure 1. A typical fitting graph. Overfitting Examined Base rate What would b be ? Figure 2. A fitting graph for the customer churn (table) model. Overfitting in Tree Induction Decision tree induction overfitting starts to the “sweet spot” in the graph . Figure 3. A typical fitting graph for tree induction. Overfitting in Mathematical Functions We add more Xi, the function becomes more and more complicated. Each Xi has a corresponding Wi, which is a learned parameter of the model . Two dimensions you can fit a line to any two points and in three dimensions you can fit a plane to any three points . This concept generalizes: as you increase the dimensionality, you can perfectly fit larger and larger sets of arbitrary points . Example: Overfitting Linear Functions Data：sepal width, petal width Types：Iris Setosa, Iris Versicolor Two different separation lines： a. Logistic regression b. Support vector machine Figure 4 Example: Overfitting Linear Functions Figure 4 Figure 5 Example: Overfitting Linear Functions Figure 6 Figure 7 From Holdout Evaluation to Cross-Validation Holdout Evaluation Splits the data into only one training and one holdout set. Cross-validation computes its estimates over all the data by performing multiple splits and systematically swapping out samples for testing. ( k folds, typically k would be 5 or 10. ) The Churn Dataset Revisited “Example: Addressing the Churn Problem with Tree Induction” in Chapter 3. The logistic regression models show slightly lower average accuracy (64.1%) and with higher variation ( standard deviation of 1.3 ) Average accuracy of the folds with classification trees is 68.6%—significantly lower than our previous measurement of 73%. ( the standard deviation of the fold accuracies is 1.1 ) Classification trees may be preferable to logistic regression because of their greater stability and performance. Learning Curves The generalization performance of data-driven modeling generally improves as more training data become available. Overfitting Avoidance & Complexity Control Concept in Tree Induction : Tree induction commonly uses two techniques to avoid overfitting. These strategies are : (i) to stop growing the tree before it gets too complex, and (ii) to grow the tree until it is too large, then “prune” it back, reducing its size (and thereby its complexity). Methods in Tree Induction : To limit tree size is to specify a minimum number of instances that must be present in a leaf. Hypothesis test ( P-value ) Overfitting Avoidance & Complexity Control General Method for Avoiding Overfitting Compare the best model we can build from one family (say, classification trees) against the best model from another family (say, logistic regression). Nested holdout testing Select the best model by assess by having a complexity of 122 nodes ( the sweet spot). Training subset Training set Induce a new tree with 122 nodes from the whole, original training data. Validation set Test set ( hold out ) Final hold out Overfitting Avoidance & Complexity Control Nested Cross-Validation Sequential Forward Selection Training set Test set Original data