Report

Neural network architectures and learning algorithms Author : Bogdan M. Wilamowski Source : IEEE INDUSTRIAL ELECTRONICS MAGAZINE Date : 2011/11/22 Presenter : 林哲緯 1 Outline • • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 2 Neural Architectures Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 3 Neural Architectures Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 4 Neural Architectures Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 5 error back propagation(EBP) algorithm • error back propagation(EBP) algorithm – multilayer perceptron (MLP) Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 6 multilayer perceptron (MLP) MLP-type architecture 3-3-4-1(without connections across layers) Neural network architectures and learning algorithms, Wilamowski, B.M. 7 neuron by neuron(NBN) algorithm • neuron by neuron(NBN) algorithm – bridged multilayer perceptron (BMLP) – fully connected cascade (FCC) arbitrarily connected network Neural network architectures and learning algorithms, Wilamowski, B.M. 8 neuron by neuron(NBN) algorithm • Levenberg–Marquardt(LM) algorithm – Improve nonlinear function of least square – Forward & Backward Computation • Jacobian Matrix – Forward-Only Computation 9 bridged multilayer perceptron (BMLP) BMLP architecture 3=3=4=1(with connections across layers marked by dotted lines) Neural network architectures and learning algorithms, Wilamowski, B.M. 10 fully connected cascade (FCC) Bipolar neural network for parity-8 problem in a FCC architecture Neural network architectures and learning algorithms, Wilamowski, B.M. 11 Outline • • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 12 parity-8 problem MLP 8*9 + 9 = 81 weights BMLP 4*9 + 8 + 4 + 1 = 49 weights Neural network architectures and learning algorithms, Wilamowski, B.M. 13 parity-8 problem 9 + 10 + 11 + 12 = 42 weights Neural network architectures and learning algorithms, Wilamowski, B.M. 14 parity-17 problem • MLP architecture needs 18 neurons • BMLP architecture with connections across hidden layers needs 9 neurons • FCC architecture needs only 5 neurons 15 parity-N problem • MLP architectures nn = neurons nw = weights • BMLP architectures • FCC architectures Neural network architectures and learning algorithms, Wilamowski, B.M. 16 Outline • • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 17 suitable architectures • For a limited number of neurons, FCC neural networks are the most powerful architectures, but this does not mean that they are the only suitable architectures 18 suitable architectures • if the two weights marked by red dotted lines – signal has to be propagated by fewer layers Neural network architectures and learning algorithms, Wilamowski, B.M. 19 Outline • • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 20 Use Minimum Network Size • receive a close-to-optimum answer for all patterns that were never used in training • generalization abilities 21 Case Study TSK fuzzy controller: (a) Required control surface (b) 8*6 = 48 defuzzification rules TSK fuzzy controller: (a) Trapezoidal membership functions (b) Triangular membership functions Neural network architectures and learning algorithms, Wilamowski, B.M. 22 Case Study (a) 3 neurons in cascade (12 weights), training error = 0.21049 (b) 4 neurons in cascade (18 weights), training error = 0.049061 (a) 5 neurons in cascade (25 weights), training error = 0.023973 (b) 8 neurons in cascade (52 weights), training error = 1.118E-005 Neural network architectures and learning algorithms, Wilamowski, B.M. 23 time complexity NBN algorithm can train neural networks 1,000 times faster than the EBP algorithm. (a) EBP algorithm, average solution time of 4.2s, and average 4188.3 iterations (b) NBN algorithm, average solution time of 2.4ms , and average 5.73 iterations Neural network architectures and learning algorithms, Wilamowski, B.M. 24 two-spiral problem NBN algorithm using FCC architecture 244 iterations and 0.913s EBP algorithm using FCC architecture 30,8225 iterations and 342.7s Neural network architectures and learning algorithms, Wilamowski, B.M. 25 Outline • • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 26 Conclusions • FCC or BMLP architectures are not only more powerful but also easier to train • use networks with a minimum number of neurons • NBN have to invert a nw*nw matrix, but 500 weights are limit now. 27