是it f(z Figure 5.1 Single neuron model Network of neurons The basic structure for the multilayer perceptron is shown in Figure 5.2. There, the circles represent the neurons(weights bias, and activation function) and the lines represent the connections between the inputs and neurons, and between the neurons in one layer and those in the next layer. This is a three-layer perceptron since there are three stages of neural processing between the inputs and outputs. More layers can be added by concatenating additional"hidden"layers of neurons The multilayer perceptron has inputs, i=1, 2,,n, and outputs, j=1, 2,, m. The number of neurons in the first hidden layer(see Figure 5.2)is. In the second hidden layer there are neurons, and in the output layer there are m neure Hence, in an N layer perceptron there are "i neurons in the / hidden layer, i=1,2,N-I First Shead Output aidian nEde lasee Figure 5.2 multilayer perceptron models The neurons in the first layer of the multilayer perceptron perform computations, and the outputs of these neurons are x=f(∑vg"x)-6 with=1,2.n/. The neurons in the second layer of the multilayer perceptron perform computations, and the outputs of these neurons are given by x=°(∑x2)-02) withj=1,2,., nz. The neurons in the third layer of the multilayer perceptron perform computations, and the outputs of these neurons are given by
Figure 5.1 Single neuron model Network of Neurons The basic structure for the multilayer perceptron is shown in Figure 5.2. There, the circles represent the neurons (weights, bias, and activation function) and the lines represent the connections between the inputs and neurons, and between the neurons in one layer and those in the next layer. This is a three-layer perceptron since there are three stages of neural processing between the inputs and outputs. More layers can be added by concatenating additional "hidden" layers of neurons. The multilayer perceptron has inputs, i = 1,2,..., n, and outputs , j =1,2,..., m. The number of neurons in the first hidden layer (see Figure 5.2) is . In the second hidden layer there are neurons, and in the output layer there are m neurons. Hence, in an N layer perceptron there are i neurons in the i n th hidden layer, i = 1,2,..., N- 1. Figure 5.2 multilayer perceptron models. The neurons in the first layer of the multilayer perceptron perform computations, and the outputs of these neurons are given by (1) (1) (1) (1) 1 (( ) ) n j j ij i j i x f wx θ = = − ∑ with j = 1,2.....n1. The neurons in the second layer of the multilayer perceptron perform computations, and the outputs of these neurons are given by 1 (2) (2) (2) (1) (2) 1 (( ) ) n j j ij i j i x f wx θ = = − ∑ with j = 1,2,..., n2. The neurons in the third layer of the multilayer perceptron perform computations, and the outputs of these neurons are given by
y=/(∑x2) with i=1, 2,...,m The parameters(scalar real numbers)w) are called the weights of the first hidden layer. The w@2)are called the weights of the second hidden layer. The w are called the weights of the output layer. The parameters 0, are called the biases of the first hidden layer. The parameters 0(are called the biases of the second hidden layer, and the 0, are the biases of the output layer. The functions f (for the output layer), f (2)(for the second hidden layer), and f (for the first hidden layer) represent the activation functions. The activation functions can be different for each neuron in the multilayer perception(e.g, the first layer could have one type of sigmoid, while the next two layers could have different sigmo id functions or threshold functions) This completes the definition of the multilayer perception. Next, we will introduce the radial basis function neural network. After that we explain how both of these neural networks relate to the other topics covered in this book 5.3.2 Radial Basis Function Neural Networks A locally tuned, overlapping receptive field is found in parts of the cerebral cortex, in the visual cortex, and in other parts of the brain. The radial basis function neural network model is based on these biological systems A radial basis function neural network is shown in Figure 5.3. There, the inputs are xi, i=1, 2,, n, and the output is ,f(x)where f represents the processing by the entire radial basis function neural network. Letx=[-1, r2 x, .The input to the i receptive field unit is x, and its output is denoted with R;(x). It has what is called a"strength"which denote by y,. Assume that there are M receptive field units. Hence, from Figure 5.3 y=f(x)=∑R(x) is the output of the radial basis function neural network Figure 5.3 Radial basis function neural network model There are several possible choices for the"receptive field units"Ri(x) 'e could choose R()=ex Wheres=[ci c.c]. o, is a scalar, and if: is a vector thenI=I=V==
2 (2) 1 (( ) ) n j j ij i j i y f wx θ = = − ∑ with j = 1,2, ...,m. The parameters (scalar real numbers) (1) ij w are called the weights of the first hidden layer. The (2) ij w are called the weights of the second hidden layer. The are called the weights of the output layer. The parameters wij (1) θ j are called the biases of the first hidden layer. The parameters (2) θ j are called the biases of the second hidden layer, and theθ j are the biases of the output layer. The functions fj (for the output layer), (2) j f (for the second hidden layer), and (1) j f (for the first hidden layer) represent the activation functions. The activation functions can be different for each neuron in the multilayer perception (e.g., the first layer could have one type of sigmoid, while the next two layers could have different sigmoid functions or threshold functions). This completes the definition of the multilayer perception. Next, we will introduce the radial basis function neural network. After that we explain how both of these neural networks relate to the other topics covered in this book. 5.3.2 Radial Basis Function Neural Networks A locally tuned, overlapping receptive field is found in parts of the cerebral cortex, in the visual cortex, and in other parts of the brain. The radial basis function neural network model is based on these biological systems. A radial basis function neural network is shown in Figure 5.3. There, the inputs are xi, i = 1,2,..., n, and the output is y =f(x) where f represents the processing by the entire radial basis function neural network. Let [ 1 2 , , ] T n x = xx x " . The input to the i th receptive field unit is x, and its output is denoted with Ri (x). lt has what is called a "strength" which we denote by i y . Assume that there are M receptive field units. Hence, from Figure 5.3, ( ) ( ) 1 M i i i y f x yR x = = = ∑ (5.3) is the output of the radial basis function neural network. Figure 5.3 Radial basis function neural network model. There are several possible choices for the "receptive field units" R i (x): 1. We could choose ( ) 2 2 exp i i i x c R x σ ⎛ ⎞ − = ⎜− ⎟ ⎜ ⎟ ⎝ ⎠ Where 1 2 , , T ii i i n ⎤ ⎦ c cc c = ⎡ ⎣ … i ,σ is a scalar, and if z is a vector then T z = z z
2. We could choose R(x)= where c. and o. are defined in choice 1 There are also alternatives to how to compute the output of the radial basis function neural network. For instance rather than computing the simple sum as in Equation(5.3), you could compute a weighted average . R( y=f(x)=iel (54) ∑R(x) It is also possible to define multilayer radial basis function neural networks This completes the definition of the radial basis function neural network. Next, we explain the relationshil between multilayer perceptions and radial basis function neural networks and fuzzy systems 5.3.3 Relationships Between Fuzzy Systems and Neural Networks There are two ways in which there are relationships between fuzzy systems and neural networks. First, techniques from one area can be used in the other. Second, in some cases the functionality (i.e, the nonlinear function that they implement) is identical. Some label the intersection between fuzzy systems and neural networks with the term fuzzy-neural"or"neuro-fuzzy"to highlight that techniques from both fields are being used. Here, we avoid this terminology and simply highlight the basic relationships between the two field Multilayer Perceptrons The multilayer perceptron should be viewed as a nonlinear network whose nonlinearity can be tuned by changing the weights, biases, and parameters of the activation functions. The fuzzy system is also a tunable nonlinearity whose shape can be changed by tuning, for example, the membership functions. Since both are tunable nonlinearities, the following approaches are possible a Gradient methods can be used for training neural networks to perform system identification or to act as estimators or predictors in the same way as fuzzy systems were trained Indeed, the gradient training of neural networks, called"back-propagation training, "was introduced well before the gradient training of fuzzy systems, and the idea for training fuzzy systems this way came from the field of neural networks Hybrid methods for training can also be used for neural networks. For instance, gradient methods may be used in conjunction with clustering methods applied to neural networks a Indirect adaptive control can also be achieved with a multilayer perceptron. To do this we use two multilayer perceptrons as the tunable nonlinearities in the certainty equivalence control law and the gradient method for tuning a Gain scheduled control may be achieved by training a multilayer perceptron to map the associations between operating conditions and controller parameters This list is by no means exhaustive. It simply shows that multilayer pere networks can take on a similar role to that of a fuzzy system in performing the function of being a tunable nonlinearity. An advantage that the fuzzy system may have, however, is that it often facilitates the incorporation of heuristic knowledge into the solution to the problem, which
2. We could choose ( ) 2 2 1 1 exp i i i R x x c σ = ⎛ ⎞ − + −⎜ ⎟ ⎜ ⎟ ⎝ ⎠ where i c and σ i are defined in choice 1. There are also alternatives to how to compute the output of the radial basis function neural network. For instance, rather than computing the simple sum as in Equation (5.3), you could compute a weighted average ( ) ( ) ( ) 1 1 M i i i M i i y R x y fx R x = = = = ∑ ∑ （5.4） It is also possible to define multilayer radial basis function neural networks. This completes the definition of the radial basis function neural network. Next, we explain the relationships between multilayer perceptions and radial basis function neural networks and fuzzy systems. 5.3.3 Relationships Between Fuzzy Systems and Neural Networks There are two ways in which there are relationships between fuzzy systems and neural networks. First, techniques from one area can be used in the other. Second, in some cases the functionality (i.e., the nonlinear function that they implement) is identical. Some label the intersection between fuzzy systems and neural networks with the term "fuzzy-neural" or "neuro-fuzzy" to highlight that techniques from both fields are being used. Here, we avoid this terminology and simply highlight the basic relationships between the two fields. Multilayer Perceptrons The multilayer perceptron should be viewed as a nonlinear network whose nonlinearity can be tuned by changing the weights, biases, and parameters of the activation functions. The fuzzy system is also a tunable nonlinearity whose shape can be changed by tuning, for example, the membership functions. Since both are tunable nonlinearities, the following approaches are possible: Gradient methods can be used for training neural networks to perform system identification or to act as estimators or predictors in the same way as fuzzy systems were trained. Indeed, the gradient training of neural networks, called "back-propagation training," was introduced well before the gradient training of fuzzy systems, and the idea for training fuzzy systems this way came from the field of neural networks. Hybrid methods for training can also be used for neural networks. For instance, gradient methods may be used in conjunction with clustering methods applied to neural networks. Indirect adaptive control can also be achieved with a multilayer perceptron. To do this we use two multilayer perceptrons as the tunable nonlinearities in the certainty equivalence control law and the gradient method for tuning. Gain scheduled control may be achieved by training a multilayer perceptron to map the associations between operating conditions and controller parameters. This list is by no means exhaustive. It simply shows that multilayer perceptron networks can take on a similar role to that of a fuzzy system in performing the function of being a tunable nonlinearity. An advantage that the fuzzy system may have, however, is that it often facilitates the incorporation of heuristic knowledge into the solution to the problem, which
can,at times, have a significant impact on the quality of the solutio Radial basis Function Neural Networks Some radial basis function neural networks are equivalent to some standard fuzzy systems in the sense that they are functionally equivalent (ie, given the same inputs, they will produce the same outputs). To see this, suppose that in Equation(5.4)we let M=R(i.e, the number of receptive field units equal to the number of rules ), y,=b, (i.., the receptive field unit strengths equal to the output membership function centers ), and choose the receptive field units R(x)=(x) (.e, choose the receptive field units to be the same as the premise membership functions). In this case we see that the radial basis function neural network is identical to a certain fuzzy system that uses center-average defuzzification. This fuzzy system is then given by y=f(x) bu (x) (x) It is also interesting to note that the functional fuzzy system(the more general version of the Takagi-Sugeno fuzzy stem)is equivalent to a class of two-layer neural networks (2001 The equivalence between this type of fuzzy system and a radial basis function neural network shows that all the echniques in this book for the above type of fuzzy system work in the same way for the above type of radial basis function neural network (or, using [200, the techniques for the Takagi fuzzy system can be used for a type of multilayer radial basis function neural network) Due to the above relationships between fuzzy systems and neural networks, some would like to view fuzzy sys d neural networks as identical areas This is however not the case for the following reasons There are classes of neural networks(e.g, dynamic neural networks) that may have a fuzzy system analog, but if so it would have to include not only standard fuzzy components but some form of a differential equation component There are certain fuzzy systems that have no clear neural analog. Consider, for example, certain"fuzzy dynamic systems".We can, however, envision how you could go about "designing a neural analog to such fuzzy systems using gradient methods like back-propagation)using data, often without using extra heuristic knowledge w? ip The neural network has traditionally been a"black box"approach where the weights and biases are trained(e often have. In fuzzy systems you can incorporate heuristic information and use data to train them. This last difference is often quoted as being one of the advantages of fuzzy systems over neural networks, at least for some applications Regardless of the differences, it is important to note that many methods in neural control (i.e, when we use a neural network for the control of a system) are quite similar to those in adaptive fuzzy control. For instance, since the fuzzy system and radial basis function neural network can be linearly parameterized, we can use them as the identifier structures in direct or indirect adaptive control schemes and use gradient or least squares methods to update the parameters. Indeed, we could have used neural networks as the structure that we trained for all of the identification methods. In this sense we can use neural networks in system identification, estimation, and prediction, and as a direct (fixed) controller that is trained with input-output data. Basically, to be fluent with the methods of adaptive fuzzy systems and control, you must know the methods of neural control-and vice versa
can, at times, have a significant impact on the quality of the solution. Radial Basis Function Neural Networks Some radial basis function neural networks are equivalent to some standard fuzzy systems in the sense that they are functionally equivalent (i.e., given the same inputs, they will produce the same outputs). To see this, suppose that in Equation (5.4) we let M =R (i.e., the number of receptive field units equal to the number of rules), i y b = i (i.e., the receptive field unit strengths equal to the output membership function centers), and choose the receptive field units as () () Ri i x x = μ (i.e., choose the receptive field units to be the same as the premise membership functions). In this case we see that the radial basis function neural network is identical to a certain fuzzy system that uses center-average defuzzification. This fuzzy system is then given by 1 1 ( ) ( ) ( ) R i i i R i i b x y fx x μ μ = = = = ∑ ∑ It is also interesting to note that the functional fuzzy system (the more general version of the Takagi-Sugeno fuzzy system) is equivalent to a class of two-layer neural networks [200]. The equivalence between this type of fuzzy system and a radial basis function neural network shows that all the techniques in this book for the above type of fuzzy system work in the same way for the above type of radial basis function neural network (or, using [200], the techniques for the Takagi-Sugeno fuzzy system can be used for a type of multilayer radial basis function neural network). Due to the above relationships between fuzzy systems and neural networks, some would like to view fuzzy systems and neural networks as identical areas. This is, however, not the case for the following reasons: There are classes of neural networks (e.g., dynamic neural networks) that may have a fuzzy system analog, but if so it would have to include not only standard fuzzy components but some form of a differential equation component. There are certain fuzzy systems that have no clear neural analog. Consider, for example, certain "fuzzy dynamic systems". We can, however, envision how you could go about "designing a neural analog to such fuzzy systems. The neural network has traditionally been a "black box" approach where the weights and biases are trained (e.g., using gradient methods like back-propagation) using data, often without using extra heuristic knowledge we often have. In fuzzy systems you can incorporate heuristic information and use data to train them. This last difference is often quoted as being one of the advantages of fuzzy systems over neural networks, at least for some applications. Regardless of the differences, it is important to note that many methods in neural control (i.e., when we use a neural network for the control of a system) are quite similar to those in adaptive fuzzy control. For instance, since the fuzzy system and radial basis function neural network can be linearly parameterized, we can use them as the identifier structures in direct or indirect adaptive control schemes and use gradient or least squares methods to update the parameters. Indeed, we could have used neural networks as the structure that we trained for all of the identification methods. In this sense we can use neural networks in system identification, estimation, and prediction, and as a direct (fixed) controller that is trained with input-output data. Basically, to be fluent with the methods of adaptive fuzzy systems and control, you must know the methods of neural control—and vice versa