interest in nonlinear dynamical systems,statistical mechanics and automata theory.1 It is the job of applied mathematicians to discover and formalize the properties of new systems using tools previously employed in other areas of science.For example,there are strong links between a certain type of net(the Hopfield net-see Ch.7)and magnetic systems known as spin glasses.The full mathematical apparatus for exploring these links is developed (alongside a series of concise summaries)by Amit(1989). All these groups are asking different questions:neuroscientists want to know how animal brains work,engineers and computer scientists want to build intelligent machines and mathematicians want to understand the fundamental properties of networks as complex systems.Another(perhaps the largest)group of people are to be found in a variety of industrial and commercial areas and use neural networks to model and analyze large,poorly understood datasets that arise naturally in their workplace.It is therefore important to understand an author's perspective when reading the literature.Their common focal point is,however,neural networks and is potentially the basis for close collaboration.For example,biologists can usefully learn from computer scientists what computations are necessary to enable animals to solve particular problems,while engineers can make use of the solutions nature has devised so that they may be applied in an act of"reverse engineering". In the next chapter we look more closely at real neurons and how they may be modelled by their artificial counterparts.This approach allows subsequent development to be viewed from both the biological and engineering-oriented viewpoints. 18
interest in nonlinear dynamical systems, statistical mechanics and automata theory. 1 It is the job of applied mathematicians to discover and formalize the properties of new systems using tools previously employed in other areas of science. For example, there are strong links between a certain type of net (the Hopfield net—see Ch. 7) and magnetic systems known as spin glasses. The full mathematical apparatus for exploring these links is developed (alongside a series of concise summaries) by Amit (1989). All these groups are asking different questions: neuroscientists want to know how animal brains work, engineers and computer scientists want to build intelligent machines and mathematicians want to understand the fundamental properties of networks as complex systems. Another (perhaps the largest) group of people are to be found in a variety of industrial and commercial areas and use neural networks to model and analyze large, poorly understood datasets that arise naturally in their workplace. It is therefore important to understand an author's perspective when reading the literature. Their common focal point is, however, neural networks and is potentially the basis for close collaboration. For example, biologists can usefully learn from computer scientists what computations are necessary to enable animals to solve particular problems, while engineers can make use of the solutions nature has devised so that they may be applied in an act of "reverse engineering". In the next chapter we look more closely at real neurons and how they may be modelled by their artificial counterparts. This approach allows subsequent development to be viewed from both the biological and engineering-oriented viewpoints. 18
1.3 Summary Artificial neural networks may be thought of as simplified models of the networks of neurons that occur naturally in the animal brain.From the biological viewpoint the essential requirement for a neural network is that it should attempt to capture what we believe are the essential information processing features of the corresponding "real"network.For an engineer,this correspondence is not so important and the network offers an alternative form of parallel computing that might be more appropriate for solving the task in hand. The simplest artificial neuron is the threshold logic unit or TLU.Its basic operation is to perform a weighted sum of its inputs and then output a "1"if this sum exceeds a threshold,and a "O"otherwise.The TLU is supposed to model the basic "integrate-and-fire"mechanism of real neurons. 19
1.3 Summary Artificial neural networks may be thought of as simplified models of the networks of neurons that occur naturally in the animal brain. From the biological viewpoint the essential requirement for a neural network is that it should attempt to capture what we believe are the essential information processing features of the corresponding "real" network. For an engineer, this correspondence is not so important and the network offers an alternative form of parallel computing that might be more appropriate for solving the task in hand. The simplest artificial neuron is the threshold logic unit or TLU. Its basic operation is to perform a weighted sum of its inputs and then output a "1" if this sum exceeds a threshold, and a "0" otherwise. The TLU is supposed to model the basic "integrate-and-fire" mechanism of real neurons. 19
1.4 Notes 1.It is not important that the reader be familiar with these areas.It suffices to understand that neural networks can be placed in relation to other areas studied by workers in these fields. 20
1.4 Notes 1. It is not important that the reader be familiar with these areas. It suffices to understand that neural networks can be placed in relation to other areas studied by workers in these fields. 20
Chapter Two Real and artificial neurons The building blocks of artificial neural nets are artificial neurons.In this chapter we introduce some simple models for these,motivated by an attempt to capture the essential information processing ability of real,biological neurons.A description of this is therefore our starting point and,although our excursion into neurophysiology will be limited,some of the next section may appear factually rather dense on first contact.The reader is encouraged to review it several times to become familiar with the biological"jargon"and may benefit by first re-reading the precis of neuron function that was given in the previous chapter.In addition,it will help to refer to Figure 2.1 and the glossary at the end of the next section. 21
Chapter Two Real and artificial neurons The building blocks of artificial neural nets are artificial neurons. In this chapter we introduce some simple models for these, motivated by an attempt to capture the essential information processing ability of real, biological neurons. A description of this is therefore our starting point and, although our excursion into neurophysiology will be limited, some of the next section may appear factually rather dense on first contact. The reader is encouraged to review it several times to become familiar with the biological "jargon" and may benefit by first re-reading the précis of neuron function that was given in the previous chapter. In addition, it will help to refer to Figure 2.1 and the glossary at the end of the next section. 21
2.1 Real neurons:a review Neurons are not only enormously complex but also vary considerably in the details of their structure and function.We will therefore describe typical properties enjoyed by a majority of neurons and make the usual working assumption of connectionism that these provide for the bulk of their computational ability. Readers interested in finding out more may consult one of the many texts in neurophysiology;Thompson(1993)provides a good introductory text,while more comprehensive accounts are given by Kandel et al.(1991)and Kuffler et al. (1984). A stereotypical neuron is shown in Figure 2.1,which should be compared with the simplified diagram in Figure 1.1.The cell body or soma contains the usual subcellular components or organelles to be found in most cells throughout the body (nucleus,mitochondria,Golgi body,etc.)but these are not shown in the diagram. Instead we focus on what differentiates neurons from other cells allowing the neuron to function as a signal processing device.This ability stems largely from the properties of the neuron's surface covering or membrane,which supports a wide variety of electrochemical processes.Morphologically the main difference lies in the set of fibres that emanate from the cell body.One of these fibres-the axonis responsible for transmitting signals to other neurons and may therefore be considered the neuron output.All other fibres are dendrites,which carry signals from other neurons to the cell body,thereby acting as neural Figure 2.1 Biological neuron. inputs.Each neuron has only one axon but can have many dendrites.The latter often appear to have a highly branched structure and so we talk of dendritic arbors.The 22
2.1 Real neurons: a review Neurons are not only enormously complex but also vary considerably in the details of their structure and function. We will therefore describe typical properties enjoyed by a majority of neurons and make the usual working assumption of connectionism that these provide for the bulk of their computational ability. Readers interested in finding out more may consult one of the many texts in neurophysiology; Thompson (1993) provides a good introductory text, while more comprehensive accounts are given by Kandel et al. (1991) and Kuffler et al. (1984). A stereotypical neuron is shown in Figure 2.1, which should be compared with the simplified diagram in Figure 1.1. The cell body or soma contains the usual subcellular components or organelles to be found in most cells throughout the body (nucleus, mitochondria, Golgi body, etc.) but these are not shown in the diagram. Instead we focus on what differentiates neurons from other cells allowing the neuron to function as a signal processing device. This ability stems largely from the properties of the neuron's surface covering or membrane, which supports a wide variety of electrochemical processes. Morphologically the main difference lies in the set of fibres that emanate from the cell body. One of these fibres—the axon—is responsible for transmitting signals to other neurons and may therefore be considered the neuron output. All other fibres are dendrites, which carry signals from other neurons to the cell body, thereby acting as neural Figure 2.1 Biological neuron. inputs. Each neuron has only one axon but can have many dendrites. The latter often appear to have a highly branched structure and so we talk of dendritic arbors. The 22