terminal. PSP Postsynaptic Potential.The change in membrane potential brought about by activity at a synapse. receptor sites The sites on the postsynaptic membrane to which molecules of neurotransmitter bind.This binding initiates the generation of a PSP. refractory period The shortest time interval between two action potentials. soma The cell body. synapse The site of physical and signal contact between neurons.On receipt of an action potential at the axon terminal of a synapse,neurotransmitter is released into the synaptic cleft and propagates to the postsynaptic membrane.There it undergoes chemical binding with receptors,which,in turn,initiates the production of a postsynaptic potential PSP). synaptic cleft The gap between the pre-and postsynaptic membranes across which chemical neurotransmitter is propagated during synaptic action.vesicles The spherical containers in the axon terminal that contain neurotransmitter.On receipt of an action potential at the axon terminal,the vesicles release their neurotransmitter into the synaptic cleft. 28
terminal. PSP Postsynaptic Potential. The change in membrane potential brought about by activity at a synapse. receptor sites The sites on the postsynaptic membrane to which molecules of neurotransmitter bind. This binding initiates the generation of a PSP. refractory period The shortest time interval between two action potentials. soma The cell body. synapse The site of physical and signal contact between neurons. On receipt of an action potential at the axon terminal of a synapse, neurotransmitter is released into the synaptic cleft and propagates to the postsynaptic membrane. There it undergoes chemical binding with receptors, which, in turn, initiates the production of a postsynaptic potential (PSP). synaptic cleft The gap between the pre- and postsynaptic membranes across which chemical neurotransmitter is propagated during synaptic action. vesicles The spherical containers in the axon terminal that contain neurotransmitter. On receipt of an action potential at the axon terminal, the vesicles release their neurotransmitter into the synaptic cleft. 28
2.2 Artificial neurons:the TLU Our task is to try and model some of the ingredients in the list above.Our first attempt will result in the structure described informally in Section 1.1. The "all-or-nothing"character of the action potential may be characterized by using a two-valued signal.Such signals are often referred to as binary or Boolean2 and conventionally take the values "O"and "1".Thus,if we have a node receiving n input signalsx1x2....,xthen these may only take on the values "O"or "1".In line with the remarks of the previous chapter,the modulatory effect of each synapse is encapsulated by simply multiplying the incoming signal with a weight value,where excitatory and inhibitory actions are modelled using positive and negative values respectively.We therefore have n weights w1,w2....,w and form the n products w,w2x2...w Each product is now the analogue of a PSP and may be negative or positive,depending on the sign of the weight.They should now be combined in a process which is supposed to emulate that taking place at the axon hillock.This will be done by simply adding them together to produce the activation a(corresponding to the axon-hillock membrane potential)so that a=D1x1+2r2+···+wnrn (2.1) As an example,consider a five-input unit with weights (0.5,1.0,-1.0,-0.5,1.2), that is w=0.5,w2=1.0,...,w5=1.2,and suppose this is presented with inputs (1,1, 1,0,0)so thatx=1,x2=1,...,x5-0.Using (2.1)the activation is given by a=(0.5×1)+(1.0×1)+(-1.0×1)+(-0.5×0)+(1.2×0) =0.5 To emulate the generation of action potentials we need a threshold value 0(Greek theta)such that,if the activation exceeds(or is equal to)0 then the node outputs a "1"(action potential),and if it is less than0 then it emits a "O".This may be represented graphically as shown in Figure 2.3 where the output has been designated the symbol y.This relation is sometimes called a step function or hard- limiter for obvious reasons.In our example,suppose that 0-0.2;then,since a0.2 (recall a=0.5)the node's outputy is 1.The entire node structure is shown in Figure 2.4 where the weights have been depicted by encircled multiplication signs.Unlike Figure 1.1,however,no effort has been made to show the size of the weights or 29
2.2 Artificial neurons: the TLU Our task is to try and model some of the ingredients in the list above. Our first attempt will result in the structure described informally in Section 1.1. The "all-or-nothing" character of the action potential may be characterized by using a two-valued signal. Such signals are often referred to as binary or Boolean 2 and conventionally take the values "0" and "1". Thus, if we have a node receiving n input signals x1 , x2 ,…, xn , then these may only take on the values "0" or "1". In line with the remarks of the previous chapter, the modulatory effect of each synapse is encapsulated by simply multiplying the incoming signal with a weight value, where excitatory and inhibitory actions are modelled using positive and negative values respectively. We therefore have n weights w1 , w2 ,…, wn and form the n products w1x1 , w2x2 ,…, wnxn . Each product is now the analogue of a PSP and may be negative or positive, depending on the sign of the weight. They should now be combined in a process which is supposed to emulate that taking place at the axon hillock. This will be done by simply adding them together to produce the activation a (corresponding to the axon-hillock membrane potential) so that (2.1) As an example, consider a five-input unit with weights (0.5, 1.0, -1.0, -0.5, 1.2), that is w1=0.5, w2=1.0,…, w5=1.2, and suppose this is presented with inputs (1, 1, 1, 0, 0) so that x1=1, x2=1,…, x5=0. Using (2.1) the activation is given by To emulate the generation of action potentials we need a threshold value (Greek theta) such that, if the activation exceeds (or is equal to) then the node outputs a "1" (action potential), and if it is less than then it emits a "0". This may be represented graphically as shown in Figure 2.3 where the output has been designated the symbol y. This relation is sometimes called a step function or hardlimiter for obvious reasons. In our example, suppose that =0.2; then, since a>0.2 (recall a=0.5) the node's output y is 1. The entire node structure is shown in Figure 2.4 where the weights have been depicted by encircled multiplication signs. Unlike Figure 1.1, however, no effort has been made to show the size of the weights or 29
signals.This type of artificial neuron is known as a threshold logic unit(TLU)and was originally proposed by McCulloch and Pitts(McCulloch Pitts 1943). It is more convenient to represent the TLU functionality in a symbolic rather than a graphical form.We already have one form for the activation as supplied by(2.1). However,this may be written more compactly using a notation that makes use of the way we have written the weights and inputs.First,a word on 0 Figure 2.3 Activation-output threshold relation in graphical form. SUM X 的州 Figure 2.4 TLU. the notation is relevant here.The small numbers used in denoting the inputs and weights are referred to as subscripts.If we had written the numbers near the top (e.g.x)they would have been superscripts and,quite generally,they are called indices irrespective of their position.By writing the index symbolically (rather than numerically)we can refer to quantities generically so that x;for example, denotes the generic or ith input where it is assumed that i can be any integer between 1 and n.Similar remarks apply to the weights wi Using these ideas it is possible to represent(2.1)in a more compact form 30
signals. This type of artificial neuron is known as a threshold logic unit (TLU) and was originally proposed by McCulloch and Pitts (McCulloch & Pitts 1943). It is more convenient to represent the TLU functionality in a symbolic rather than a graphical form. We already have one form for the activation as supplied by (2.1). However, this may be written more compactly using a notation that makes use of the way we have written the weights and inputs. First, a word on Figure 2.3 Activation-output threshold relation in graphical form. Figure 2.4 TLU. the notation is relevant here. The small numbers used in denoting the inputs and weights are referred to as subscripts. If we had written the numbers near the top (e.g. x 1 ) they would have been superscripts and, quite generally, they are called indices irrespective of their position. By writing the index symbolically (rather than numerically) we can refer to quantities generically so that xi , for example, denotes the generic or ith input where it is assumed that i can be any integer between 1 and n. Similar remarks apply to the weights wi . Using these ideas it is possible to represent (2.1) in a more compact form 30
(2.2) where E(upper case Greek sigma)denotes summation.The expressions above and below E denote the upper and lower limits of the summation and tell us that the index i runs from 1 to n.Sometimes the limits are omitted because they have been defined elsewhere and we simply indicate the summation index(in this case i)by writing it below the E. The threshold relation for obtaining the output y may be written 1 ifa≥8 0 if a< (2.3) Notice that there is no mention of time in the TLU;the unit responds instantaneously to its input whereas real neurons integrate over time as well as space.The dendrites are represented (if one can call it a representation)by the passive connecting links between the weights and the summing operation.Action-potential generation is simply represented by the threshold function. 31
(2.2) where E (upper case Greek sigma) denotes summation. The expressions above and below E denote the upper and lower limits of the summation and tell us that the index i runs from 1 to n. Sometimes the limits are omitted because they have been defined elsewhere and we simply indicate the summation index (in this case i) by writing it below the E. The threshold relation for obtaining the output y may be written (2.3) Notice that there is no mention of time in the TLU; the unit responds instantaneously to its input whereas real neurons integrate over time as well as space. The dendrites are represented (if one can call it a representation) by the passive connecting links between the weights and the summing operation. Action-potential generation is simply represented by the threshold function. 31
2.3 Resilience to noise and hardware failure Even with this simple neuron model we can illustrate two of the general properties of neural networks.Consider a two-input TLU with weights (0,1)and threshold 0.5.Its response to all four possible input sets is shown in Table 2.1. Table 2.1 TLU with weights (0,1)and threshold 0.5. T2 Activation Output 0 0 0 0 1 1 0 0 1 1 Now suppose that our hardware which implements the TLU is faulty so that the weights are not held at their true values and are encoded instead as (0.2,0.8).The revised TLU functionality is given in Table 2.2.Notice that,although the activation has changed,the output is the same as that for the original TLU.This is because changes in the activation,as long as they don't cross the threshold,produce no change in output.Thus,the threshold function doesn't care whether the activation is just below 0 or is very much less than it still outputs a 0.Similarly,it doesn't matter by how much the activation exceeds 0,the TLU always supplies a 1 as output. Table 2.2 TLU with weights (0.2,0.8)and threshold 0.5. 工1 工2 Activation Output 0 0 0.8 1 02 0 1 This behaviour is characteristic of nonlinear systems.In a linear system,the output is proportionally related to the input:small/large changes in the input always produce corresponding small/large changes in the output.On the other hand, nonlinear relations do not obey a proportionality restraint so the magnitude of the change in output does not necessarily reflect that of the input.Thus,in our TLU example,the activation can change from 0 to 0.2(a difference of 0.2)and make no difference to the output.If,however,it were to change from 0.49 to 0.51(a difference of 0.02)the output would suddenly alter from 0 to 1. We conclude from all this that TLUs are robust in the presence of hardware failure; 32
2.3 Resilience to noise and hardware failure Even with this simple neuron model we can illustrate two of the general properties of neural networks. Consider a two-input TLU with weights (0, 1) and threshold 0.5. Its response to all four possible input sets is shown in Table 2.1. Now suppose that our hardware which implements the TLU is faulty so that the weights are not held at their true values and are encoded instead as (0.2, 0.8). The revised TLU functionality is given in Table 2.2. Notice that, although the activation has changed, the output is the same as that for the original TLU. This is because changes in the activation, as long as they don't cross the threshold, produce no change in output. Thus, the threshold function doesn't care whether the activation is just below or is very much less than ; it still outputs a 0. Similarly, it doesn't matter by how much the activation exceeds , the TLU always supplies a 1 as output. This behaviour is characteristic of nonlinear systems. In a linear system, the output is proportionally related to the input: small/large changes in the input always produce corresponding small/large changes in the output. On the other hand, nonlinear relations do not obey a proportionality restraint so the magnitude of the change in output does not necessarily reflect that of the input. Thus, in our TLU example, the activation can change from 0 to 0.2 (a difference of 0.2) and make no difference to the output. If, however, it were to change from 0.49 to 0.51 (a difference of 0.02) the output would suddenly alter from 0 to 1. We conclude from all this that TLUs are robust in the presence of hardware failure; 32