About This Book This book is about training methods - in particular, fast second-order training methods - for multi-layer perceptrons (MLPs). MLPs (also known as feed-forward neural networks) are the most widely-used class of neural network. Over the past decade MLPs have achieved increasing popularity among scientists, engineers and other professionals as tools for tackling a wide variety of information processing tasks. In common with all neural networks, MLPsare trained (rather than programmed) to carryout the chosen information processing function. Unfortunately, the (traditional' method for trainingMLPs- the well-knownbackpropagation method - is notoriously slow and unreliable when applied to many prac­ tical tasks. The development of fast and reliable training algorithms for MLPsis one of the most important areas ofresearch within the entire field of neural computing. The main purpose of this book is to bring to a wider audience a range of alternative methods for training MLPs, methods which have proved orders of magnitude faster than backpropagation when applied to many training tasks. The book also addresses the well-known (local minima' problem, and explains ways in which fast training methods can be com­ bined with strategies for avoiding (or escaping from) local minima. All the methods described in this book have a strong theoretical foundation, drawing on such diverse mathematical fields as classical optimisation theory, homotopic theory and stochastic approximation theory.

Inhalt

1 Multi-Layer Perceptron Training.- 1.1 Introduction to MLPs.- 1.1.1 The MLP Architecture.- 1.1.2 MLP Training.- 1.2 Error Surfaces and Local Minima.- 1.2.1 Error Surface Fundamentals.- 1.2.2 MLP Error Surfaces.- 1.3 Backpropagation.- 1.3.1 An Introduction to Backpropagation.- 1.3.2 The Bold Driver Method.- 1.3.3 Backpropagation with Momentum.- 1.3.4 On-Line Backpropagation.- 1.3.5 The Delta-Bar-Delta Method.- 2 Classical Optimisation.- 2.1 Introduction to Classical Methods.- 2.1.1 The Linear Model and Steepest Descent.- 2.1.2 The Quadratic Model and Newton's Method.- 2.1.3 Line-Search Methods vs. Model-Trust Region Methods.- 2.2 General Numerical Considerations.- 2.2.1 Finite-Precision Arithmetic and Computational Errors.- 2.2.2 Positive Definiteness and the Model Hessian.- 2.2.3 Scaling and Preconditioning.- 3 Second-Order Optimisation Methods.- 3.1 Line-Search Strategies.- 3.1.1 Line Minimisation Fundamentals.- 3.1.2 Inaccurate Line Searches.- 3.1.3 Backtracking Line Searches.- 3.2 Model-Trust Region Strategies.- 3.2.1 A Simple Model-Trust Region Algorithm.- 3.2.2 Fletcher's Method.- 3.2.3 Modern Model-Trust Region Algorithms.- 3.2.4 Møller's `Scaled' Model-Trust Region Strategy.- 3.3 Multivariate Methods for General Nonlinear Optimisation.- 3.3.1 Finite-Difference Newton's Method.- 3.3.2 Quasi-Newton Methods.- 3.3.3 The 'Memoryless' Quasi-Newton Method.- 3.3.4 Conjugate Gradient Methods.- 3.4 Special Methods for Nonlinear Least Squares.- 3.4.1 The Gauss-Newton Method.- 3.4.2 The Levenberg-Marquardt Method.- 3.5 Comparison of Methods.- 4 Second-Order Training Methods for MLPs.- 4.1 The Calculation of Second Derivatives.- 4.1.1 Exact Evaluation of the Hessian Matrix.- 4.1.2 Exact Evaluation of the Hessian Times a Vector.- 4.1.3 Exact Evaluation of the Jacobian Matrix.- 4.2 Reducing Storage and Computational Costs.- 4.2.1 Diagonal Approximations of the Hessian Matrix.- 4.2.2 Reduced Function and Gradient Evaluations.- 4.3 Second-Order On-Line Training.- 4.3.1 An Introduction to Second-Order On-Line Training Strategies.- 4.3.2 'Noise-Free' On-Line Training Schemes.- 4.4 Conclusion.- 5 An Experimental Comparison of MLP Training Methods.- 5.1 Benchmark Training Tasks.- 5.1.1 N-Parity.- 5.1.2 The sin(x) Problem.- 5.1.3 The sin(x)cos(2x) Problem.- 5.1.4 The tan(x) Problem.- 5.2 Initial Training Conditions.- 5.2.1 MLP Architectures.- 5.2.2 Training Algorithms.- 5.2.3 Termination Conditions.- 5.3 Experimental Results.- 5.3.1 'Global Reliability'.- 5.3.2 Training Speed Metrics.- 5.3.3 Training Speed Results.- 5.3.4 Conclusions.- 6 Global Optimisation.- 6.1 Introduction to Global Methods.- 6.1.1 Stochastic Methods.- 6.1.2 Deterministic Methods.- 6.2 Expanded Range Approximation (ERA).- 6.2.1 An Introduction to ERA.- 6.2.2 The ERA Method in Practice.- 6.3 The TRUST Method.- 6.3.1 The Tunnelling Function.- 6.3.2 The Tunnelling Phase.- 6.3.3 An Evaluation of the TRUST Method.