Neural Network Toolbox

Bibliography

[Batt92] Battiti, R., "First and second order methods for learning: Between steepest descent and Newton's method," Neural Computation, vol. 4, no. 2, pp. 141-166, 1992.

[Beal72] Beale, E. M. L., "A derivation of conjugate gradients," in F. A. Lootsma, ed., Numerical methods for nonlinear optimization, London: Academic Press, 1972.

[Bren73] Brent, R. P., Algorithms for Minimization Without Derivatives, Englewood Cliffs, NJ: Prentice-Hall, 1973.

[Caud89] Caudill, M., Neural Networks Primer, San Francisco, CA: Miller Freeman Publications, 1989.

This collection of papers from the AI Expert Magazine gives an excellent introduction to the field of neural networks. The papers use a minimum of mathematics to explain the main results clearly. Several good suggestions for further reading are included.

[CaBu92] Caudill, M., and C. Butler, Understanding Neural Networks: Computer Explorations, Vols. 1 and 2, Cambridge, MA: the MIT Press, 1992.

This is a two volume workbook designed to give students "hands on" experience with neural networks. It is written for a laboratory course at the senior or first-year graduate level. Software for IBM PC and Apple Macintosh computers is included. The material is well written, clear and helpful in understanding a field that traditionally has been buried in mathematics.

[Char92] Charalambous, C.,"Conjugate gradient algorithm for efficient training of artificial neural networks," IEEE Proceedings, vol. 139, no. 3, pp. 301-310, 1992.

[ChCo91] Chen, S., C. F. N. Cowan, and P. M. Grant, "Orthogonal least squares learning algorithm for radial basis function networks," IEEE Transactions on Neural Networks, vol. 2, no. 2, pp. 302-309, 1991.

This paper gives an excellent introduction to the field of radial basis functions. The papers use a minimum of mathematics to explain the main results clearly. Several good suggestions for further reading are included.

[DARP88] DARPA Neural Network Study, Lexington, MA: M.I.T. Lincoln Laboratory, 1988.

This book is a compendium of knowledge of neural networks as they were known to 1988. It presents the theoretical foundations of neural networks and discusses their current applications. It contains sections on associative memories, recurrent networks, vision, speech recognition, and robotics. Finally, it discusses simulation tools and implementation technology.

[DeSc83] Dennis, J. E., and R. B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Englewood Cliffs, NJ: Prentice-Hall, 1983.

[Elma90] Elman, J. L.,"Finding structure in time," Cognitive Science, vol. 14, pp. 179-211, 1990.

This paper is a superb introduction to the Elman networks described in Chapter 10, "Recurrent Networks."

[FlRe64] Fletcher, R., and C. M. Reeves, "Function minimization by conjugate gradients," Computer Journal, vol. 7, pp. 149-154, 1964.

[FoHa97] Foresee, F. D., and M. T. Hagan, "Gauss-Newton approximation to Bayesian regularization," Proceedings of the 1997 International Joint Conference on Neural Networks, pages 1930-1935, 1997.

[GiMu81] Gill, P. E., W. Murray, and M. H. Wright, Practical Optimization, New York: Academic Press, 1981.

[Gros82] Grossberg, S., Studies of the Mind and Brain, Drodrecht, Holland: Reidel Press, 1982.

This book contains articles summarizing Grossberg's theoretical psychophysiology work up to 1980. Each article contains a preface explaining the main points.

[HaDe99] Hagan, M. T., and H.B. Demuth, "Neural Networks for Control," Proceedings of the 1999 American Control Conference, San Diego, CA, 1999, pp. 1642-1656.

[HaJe99] Hagan, M. T., O. De Jesus, and R. Schultz, "Training Recurrent Networks for Filtering and Control," Chapter 12 in Recurrent Neural Networks: Design and Applications, L. Medsker and L.C. Jain, Eds., CRC Press, 1999, pp. 311-340.

[HaMe94] Hagan, M. T., and M. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Transactions on Neural Networks, vol. 5, no. 6, pp. 989-993, 1994.

This paper reports the first development of the Levenberg-Marquardt algorithm for neural networks. It describes the theory and application of the algorithm, which trains neural networks at a rate 10 to 100 times faster than the usual gradient descent backpropagation method.

[HDB96] Hagan, M. T., H. B. Demuth, and M. H. Beale, Neural Network Design, Boston, MA: PWS Publishing, 1996.

This book provides a clear and detailed survey of basic neural network architectures and learning rules. It emphasizes mathematical analysis of networks, methods of training networks, and application of networks to practical engineering problems. It has demonstration programs, an instructor's guide and transparency overheads for teaching.

[Hebb49] Hebb, D. O., The Organization of Behavior, New York: Wiley, 1949.

This book proposed neural network architectures and the first learning rule. The learning rule is used to form a theory of how collections of cells might form a concept.

[Himm72] Himmelblau, D. M., Applied Nonlinear Programming, New York: McGraw-Hill, 1972.

[HuSb92] Hunt, K. J., D. Sbarbaro, R. Zbikowski, and P. J. Gawthrop, Neural Networks for Control System - A Survey," Automatica, Vol. 28, 1992, pp. 1083-1112.

[Joll86] Jolliffe, I. T., Principal Component Analysis, New York: Springer-Verlag, 1986.

[Koho87] Kohonen, T., Self-Organization and Associative Memory, 2nd Edition, Berlin: Springer-Verlag, 1987.

This book analyzes several learning rules. The Kohonen learning rule is then introduced and embedded in self-organizing feature maps. Associative networks are also studied.

[Koho97] Kohonen, T., Self-Organizing Maps, Second Edition, Berlin: Springer-Verlag, 1997.

This book discusses the history, fundamentals, theory, applications and hardware of self-organizing maps. It also includes a comprehensive literature survey.

[LiMi89] Li, J., A. N. Michel, and W. Porod, "Analysis and synthesis of a class of neural networks: linear systems operating on a closed hypercube," IEEE Transactions on Circuits and Systems, vol. 36, no. 11, pp. 1405-1422, 1989.

This paper discusses a class of neural networks described by first order linear differential equations that are defined on a closed hypercube. The systems considered retain the basic structure of the Hopfield model but are easier to analyze and implement. The paper presents an efficient method for determining the set of asymptotically stable equilibrium points and the set of unstable equilibrium points. Examples are presented. The method of Li et al. is implemented in Chapter 9 of this User's Guide.

[Lipp87] Lippman, R. P., "An introduction to computing with neural nets," IEEE ASSP Magazine, pp. 4-22, 1987.

This paper gives an introduction to the field of neural nets by reviewing six neural net models that can be used for pattern classification. The paper shows how existing classification and clustering algorithms can be performed using simple components that are like neurons. This is a highly readable paper.

[MacK92] MacKay, D. J. C., "Bayesian interpolation," Neural Computation, vol. 4, no. 3, pp. 415-447, 1992.

[McPi43] McCulloch, W. S., and W. H. Pitts, "A logical calculus of ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5, pp. 115-133, 1943.

A classic paper that describes a model of a neuron that is binary and has a fixed threshold. A network of such neurons can perform logical operations.

[Moll93] Moller, M. F., "A scaled conjugate gradient algorithm for fast supervised learning," Neural Networks, vol. 6, pp. 525-533, 1993.

[MuNe92] Murray, R., D. Neumerkel, and D. Sbarbaro, "Neural Networks for Modeling and Control of a Non-linear Dynamic System," Proceedings of the 1992 IEEE International Symposium on Intelligent Control, 1992, pp. 404-409.

[NaMu97] Narendra, K.S., and S. Mukhopadhyay, "Adaptive Control Using Neural Networks and Approximate Models," IEEE Transactions on Neural Networks Vol. 8, 1997, pp. 475-485.

[NgWi89] Nguyen, D., and B. Widrow, "The truck backer-upper: An example of self-learning in neural networks," Proceedings of the International Joint Conference on Neural Networks, vol 2, pp. 357-363, 1989.

This paper describes a two-layer network that first learned the truck dynamics and then learned how to back the truck to a specified position at a loading dock. To do this, the neural network had to solve a highly nonlinear control systems problem.

[NgWi90] Nguyen, D., and B. Widrow, "Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights," Proceedings of the International Joint Conference on Neural Networks, vol 3, pp. 21-26, 1990.

Nguyen and Widrow demonstrate that a two-layer sigmoid/linear network can be viewed as performing a piecewise linear approximation of any learned function. It is shown that weights and biases generated with certain constraints result in an initial network better able to form a function approximation of an arbitrary function. Use of the Nguyen-Widrow (instead of purely random) initial conditions often shortens training time by more than an order of magnitude.

[Powe77] Powell, M. J. D., "Restart procedures for the conjugate gradient method," Mathematical Programming, vol. 12, pp. 241-254, 1977.

[Pulu92] Purdie, N., E. A. Lucas, and M. B. Talley, "Direct measure of total cholesterol and its distribution among major serum lipoproteins," Clinical Chemistry, vol. 38, no. 9, pp. 1645-1647, 1992.

[RiBr93] Riedmiller, M., and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," Proceedings of the IEEE International Conference on Neural Networks, 1993.

[Rose61] Rosenblatt, F., Principles of Neurodynamics, Washington D.C.: Spartan Press, 1961.

This book presents all of Rosenblatt's results on perceptrons. In particular, it presents his most important result, the perceptron learning theorem.

[RuHi86a] Rumelhart, D. E., G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation,", in D. E. Rumelhart and J. L. McClelland, eds. Parallel Data Processing, vol.1, Cambridge, MA: The M.I.T. Press, pp. 318-362, 1986.

This is a basic reference on backpropagation.

[RuHi86b] Rumelhart, D. E., G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533-536, 1986.

[RuMc86] Rumelhart, D. E., J. L. McClelland, and the PDP Research Group, eds., Parallel Distributed Processing, Vols. 1 and 2, Cambridge, MA: The M.I.T. Press, 1986.

These two volumes contain a set of monographs that present a technical introduction to the field of neural networks. Each section is written by different authors. These works present a summary of most of the research in neural networks to the date of publication.

[Scal85] Scales, L.E., Introduction to Non-Linear Optimization, New York: Springer-Verlag, 1985.

[SoHa96] Soloway, D., and P. J. Haley, "Neural Generalized Predictive Control," Proceedings of the 1996 IEEE International Symposium on Intelligent Control, 1996, pp. 277-281.

[VoMa88] Vogl, T. P., J. K. Mangis, A. K. Rigler, W. T. Zink, and D. L. Alkon, "Accelerating the convergence of the backpropagation method," Biological Cybernetics, vol. 59, pp. 256-264, 1988.

Backpropagation learning can be speeded up and made less sensitive to small features in the error surface such as shallow local minima by combining techniques such as batching, adaptive learning rate, and momentum.

[Wass93] Wasserman, P. D., Advanced Methods in Neural Computing, New York: Van Nostrand Reinhold, 1993.

[WiHo60] Widrow, B., and M. E. Hoff, "Adaptive switching circuits," 1960 IRE WESCON Convention Record, New York IRE, pp. 96-104, 1960.

[WiSt85] Widrow, B., and S. D. Sterns, Adaptive Signal Processing, New York: Prentice-Hall, 1985.

This is a basic paper on adaptive signal processing.

Glossary Demonstrations and Applications