Computational Intelligence（Concepts to Implementations）Part 1

Definitions strategies,and particle swarm optimization.All of these paradigms use populations of individuals(potential solutions),rather than single data points or vectors. Genetic algorithms are search algorithms that incorporate natural evolution mechanisms,including crossover,mutation,and survival of the fittest.They are more often used for optimization,but also are used for classification.Evolutionary programming algorithms are similar to genetic algorithms,but do not incorporate crossover.Rather,they rely on survival of the fittest and mutation.Evolution strate- gies are similar to genetic algorithms but use recombination to exchange information between population members instead of crossover,and often use a different type of mutation as well.Genetic programming is a methodology used to evolve computer programs.The structures being manipulated are usually hierarchical tree structures. Particle swarm optimization flies potential solutions,called particles,through the problem space.The particles are accelerated toward selected points in the problem space where previous fitness values have been high. Computational intelligence is a methodology involving computing that provides a system with an ability to learn and/or to deal with new situations,such that the system is perceived to possess one or more attributes of reason,such as general- ization,discovery,association,and abstraction.Computational intelligence systems usually incorporate hybrids of paradigms such as artificial neural networks,fuzzy systems,and evolutionary computation systems,augmented with knowledge ele- ments.They are often designed to mimic one or more aspects of biologiacal intelli- gence.Computational intelligence is also closely related to adaptation.In fact, another definition of CI is that it comprises practical adaptation concepts, paradigms,algorithms,and implementations that enable or facilitate appropriate actions(intelligent behavior)by systems in complex and changing environments. We discuss adaptation in more detail in the next chapter. A paradigm is a particular example of computational intelligence attributes-in the case of a neural network,the architecture,activation and learning rules,update procedure,and so on-that exhibits a certain type of behavior.Put another way,it is a clear and specific example of a concept.Back-propagation is one example of a neural network paradigm because it implies a certain set of attributes,for example, the architecture and the learning rule.A paradigm is a particular set of choices for all attributes.Development of a new paradigm involves assembling a set of attributes that define the intended behavior of the CI tool. An implementation is a computer program written and compiled for a specific computer or class of computers that implements a paradigm.The back-propagation neural network application on the book's web site (described in Chapter 4)is an implementation of the back-propagation paradigm. The discussion in this book deals with semantics,as well as with concepts.To an extent,we are the prisoners of our terminology.For example,consider the term arti- ficial intelligence.It is the authors'opinion that labeling some subset of intelligence artificial is somewhat analogous to calling what an airplane does "artificial flight

Definitions (~ ~~ strategies, and particle swarm optimization. All of these paradigms use populations of individuals (potential solutions), rather than single data points or vectors. Genetic algorithms are search algorithms that incorporate natural evolution mechanisms, including crossover, mutation, and survival of the fittest. They are more often used for optimization, but also are used for classification. Evolutionary programming algorithms are similar to genetic algorithms, but do not incorporate crossover. Rather, they rely on survival of the fittest and mutation. Evolution strate￾gies are similar to genetic algorithms but use recombination to exchange information between population members instead of crossover, and often use a different type of mutation as well. Genetic programming is a methodology used to evolve computer programs. The structures being manipulated are usually hierarchical tree structures. Particle swarm optimization flies potential solutions, called particles, through the problem space. The particles are accelerated toward selected points in the problem space where previous fitness values have been high. Computational intelligence is a methodology involving computing that provides a system with an ability to learn and/or to deal with new situations, such that the system is perceived to possess one or more attributes of reason, such as general￾ization, discovery, association, and abstraction. Computational intelligence systems usually incorporate hybrids of paradigms such as artificial neural networks, fuzzy systems, and evolutionary computation systems, augmented with knowledge ele￾ments. They are often designed to mimic one or more aspects ofbiologiacal intelli￾gence. Computational intelligence is also closely related to adaptation. In fact, another definition of CI is that it comprises practical adaptation concepts, paradigms, algorithms, and implementations that enable or facilitate appropriate actions (intelligent behavior) by systems in complex and changing environments. We discuss adaptation in more detail in the next chapter. A paradigm is a particular example of computational intelligence attributes--in the case of a neural network, the architecture, activation and learning rules, update procedure, and so on--that exhibits a certain type of behavior. Put another way, it is a clear and specific example of a concept. Back-propagation is one example of a neural network paradigm because it implies a certain set of attributes, for example, the architecture and the learning rule. A paradigm is a particular set of choices for all attributes. Development of a new paradigm involves assembling a set of attributes that define the intended behavior of the CI tool. An implementation is a computer program written and compiled for a specific computer or class of computers that implements a paradigm. The back-propagation neural network application on the book's web site (described in Chapter 4) is an implementation of the back-propagation paradigm. The discussion in this book deals with semantics, as well as with concepts. To an extent, we are the prisoners of our terminology. For example, consider the term arti￾ficial intelligence. It is the authors' opinion that labeling some subset of intelligence artificial is somewhat analogous to calling what an airplane does "artificial flight

Chapter One-Foundations There are also terms that require careful usage.One example is neural networks,for which it is necessary to specify whether we are referring to biological wetware or artificial neural network analytical tools.We must also be aware of what Bezdek (1994)calls"seductive semantics,"which are words and phrases that are often inter- preted too literally,resulting in meanings being inferred that are more profound and important than are warranted.Examples are cognitive and genetic.With that caveat,and having presented the basic definitions we use,let us now review the theory and technology foundations of computational intelligence tools and com- ponent methodologies. Biological Basis for Neural Networks Every day of our lives,each of us carries out thousands of tasks that require us to keep track of many things at once and to process and act on these things.Relatively simple actions,such as picking up a glass of water or dialing a telephone number,involve many individual components requiring memory,learning,and physical coordina- tion.The complexity of such "simple"tasks,which most of us do all the time with- out consciously"thinking"about them,is underscored by the difficulty involved in teaching robots to perform them.Performance of these tasks is facilitated by our complex adaptive biological structure. Neurons Studies in fields such as biology and biophysics over the past few decades have shed some light on the construction and operation of our brains and nervous systems, which helps us understand how these tasks are performed.Living organisms are made up of cells,and the basic building blocks of the nervous system are nerve cells called neurons.The major components of a neuron include a central cell body, dendrites,and an axon. Figure 1.1 is a conceptual diagram of a neuron.The signal flow goes from left to right,from the dendrites,through the cell body,and out through the axon.The signal from one neuron is passed on to others by means of connections between the axon of the first and dendrites of the others.These connections are called synapses. Axons often synapse onto the trunk of a dendrite,but they can also synapse directly onto the cell body. The human brain has a large number of neurons,or processing elements(PEs). Typical estimates of the total number are on the order of 10 to 500 billion(Rumelhart and McClelland 1986).According to one estimate by Stubbs(1988),neurons are 1There are many kinds of neuron;for detailed information on their configuration and functioning, refer to a book on neuroanatomy or neurology,such as Kandel,Schwartz,and Jessell(2000)

Chapter One--Foundations There are also terms that require careful usage. One example is neural networks, for which it is necessary to specify whether we are referring to biological wetware or artificial neural network analytical tools. We must also be aware of what Bezdek (1994) calls "seductive semantics," which are words and phrases that are often inter￾preted tOO literally, resulting in meanings being inferred that are more profound and important than are warranted. Examples are cognitive and genetic. With that caveat, and having presented the basic definitions we use, let us now review the theory and technology foundations of computational intelligence tools and com￾ponent methodologies. Biological Basis for Neural Networks Every day of our lives, each of us carries out thousands of tasks that require us to keep track of many things at once and to process and act on these things. Relatively simple actions, such as picking up a glass of water or dialing a telephone number, involve many individual components requiring memory, learning, and physical coordina￾tion. The complexity of such "simple" tasks, which most of us do all the time with￾out consciously "thinking" about them, is underscored by the difficulty involved in teaching robots to perform them. Performance of these tasks is facilitated by our complex adaptive biological structure. Neurons Studies in fields such as biology and biophysics over the past few decades have shed some light on the construction and operation of our brains and nervous systems, which helps us understand how these tasks are performed. Living organisms are made up of cells, and the basic building blocks of the nervous system are nerve cells called neurons. The major components of a neuron include a central cell body, dendrites, and an axon. Figure 1.1 is a conceptual diagram of a neuron. 1 The signal flow goes from left to right, from the dendrites, through the cell body, and out through the axon. The signal from one neuron is passed on to others by means of connections between the axon of the first and dendrites of the others. These connections are called synapses. Axons often synapse onto the trunk of a dendrite, but they can also synapse directly onto the cell body. The human brain has a large number of neurons, or processing elements (PEs). Typical estimates ofthe total number are on the order of 10 to 500 billion (Rumelhart and McClelland 1986). According to one estimate by Stubbs (1988), neurons are 1 There are many kinds of neuron; for detailed information on their configuration and functioning, refer to a book on neuroanatomy or neurology, such as Kandel, Schwartz, and ]essell (2000)

Biological Basis for Neural Networks Axons Axons from Other Cell Body Neurons Information Flow Dendrites Figure 1.1 Conceptual diagram of a neuron. arranged into about 1,000 main modules,each with about 500 neural networks.Each network has on the order of 100,000 neurons.The axon of each neuron connects to anywhere from hundreds to thousands of other neurons;the value varies greatly from neuron to neuron and from neuron type to neuron type.According to a rule called Eccles's law,each neuron either excites or inhibits all neurons to which it is connected. Biological versus Artificial Neural Networks While the processing element in an artificial neural network(ANN)is generally con- sidered to be very roughly analogous to a biological neuron,there are significant dif- ferences between a neural biological structure(as it is currently understood)and the implementation or representation of this structure in artificial neural networks.We summarize the most important differences here,recognizing there are many others. Eccles's Law In a typical implementation of an ANN,connections among PEs can have either positive or negative weights.These weights correspond to excitatory and inhibitory neural connections,so Eccles'law is not usually implemented in ANNs. AC versus DC Information about the state of activation,or excitation,of a PE generally is passed to other PEs to which it is connected as a value that roughly corresponds to a direct current (DC)level.In biological neural networks(BNNs),a train of pulses across a synapse carries the information,and higher absolute values of activation result in higher pulse rates,so that something analogous to alternating current(AC)fre- quency,or pulse repetition rate,generally corresponds to activation level.There are exceptions to the pulse rate carrying information in biological networks,but they are relatively unimportant for our discussion

Biological Basis for Neural Networks Axons from Other Neurons Figure 1.1 Conceptual diagram of a neuron. / Cell Body Dendrites Axons v Information Flow arranged into about 1,000 main modules, each with about 500 neural networks. Each network has on the order of 100,000 neurons. The axon of each neuron connects to anywhere from hundreds to thousands of other neurons; the value varies greatly from neuron to neuron and from neuron type to neuron type. According to a rule called Eccles's law, each neuron either excites or inhibits all neurons to which it is connected. Biological versus Artificial Neural Networks While the processing element in an artificial neural network (ANN) is generally con￾sidered to be very roughly analogous to a biological neuron, there are significant dif￾ferences between a neural biological structure (as it is currently understood) and the implementation or representation of this structure in artificial neural networks. We summarize the most important differences here, recognizing there are many others. Eccles's Law In a typical implementation of an ANN, connections among PEs can have either positive or negative weights. These weights correspond to excitatory and inhibitory neural connections, so Eccles' law is not usually implemented in ANNs. AC versus DC Information about the state of activation, or excitation, of a PE generally is passed to other PEs to which it is connected as a value that roughly corresponds to a direct current (DC) level. In biological neural networks (BNNs), a train of pulses across a synapse carries the information, and higher absolute values of activation result in higher pulse rates, so that something analogous to alternating current (AC) fre￾quency, or pulse repetition rate, generally corresponds to activation level. There are exceptions to the pulse rate carrying information in biological networks, but they are relatively unimportant for our discussion

Chapter One-Foundations PE Types While there are many kinds of neuron in biological systems,an artificial neural network is typically implemented with only one type of PE.Occasionally,two or three types of PE are used,and as the technology of ANNs develops,more sophisticated tools may make use of several PE types in each implementation.On the other hand,some studies indicate that any required implementation can be carried out with as few as two types of PE(Rumelhart and McClelland 1986). Speed It is reported widely in the literature that neurons in BNNs typically operate on individual cycle times of about 10 to 100 milliseconds.The basic clock frequency in a personal computer is a few gigahertz,which results in a basic cycle time for the computer of less than a nanosecond.Even taking into account the number of multiply-accumulate operations needed to calculate and propagate a new value for a PE(typically 10-100),the basic cycle time for an individual PE is still only about 10 to 100 nanoseconds.In some ways,however,speed is deceptive.Despite its slower cycle,the brain is still able to perform some tasks orders of magnitude faster than today's fastest digital computer.This,most likely,is because of the brain's massively parallel architecture.(Recent research related to neural processing in echo-locating bats,however,indicates that these creatures are physiologically processing signals in a time span of a few hundred nanoseconds,so it seems obvious that we still have much to learn about how the brain functions. Quantity of PEs There is a significant difference between the number of PEs in the typical ANN and the number of biological neurons involved in any task in a BNN.Typical ANNs are implemented with something like a few dozen to several hundred PEs.Each of the 1,000 main modules in the human brain described by Stubbs(1988)contains about 500 million neurons,and it is almost certain that several(perhaps many)of these main modules are involved in any simple task.Of course,for any practical appli- cation,most engineers and computer scientists might be hard pressed to figure out how to effectively use a neural network tool(NNT)with 500 million PEs! Some biologically oriented scientists have criticized artificial neural networks because they don't model all the activities of the brain sufficiently well.Our primary goal as engineers and computer scientists,however,is to solve complex problems, not to model the brain.Our interest,then,is in adapting relevant concepts to solve difficult problems.As an oft-quoted saying (oft-quoted in engineering circles,any- way)puts it,"Scientists study what is.Engineers create what has never been."This statement is not meant to be antagonistic toward scientists.What scientists do is just as noble and worthwhile as what engineers do;they just have a different mission and a different perspective

Chapter OnemFoundations PE Types While there are many kinds of neuron in biological systems, an artificial neural network is typically implemented with only one type of PE. Occasionally, two or three types of PE are used, and as the technology of ANNs develops, more sophisticated tools may make use of several PE types in each implementation. On the other hand, some studies indicate that any required implementation can be carried out with as few as two types of PE (Rumelhart and McClelland 1986). Speed It is reported widely in the literature that neurons in BNNs typically operate on individual cycle times of about 10 to 100 milliseconds. The basic clock frequency in a personal computer is a few gigahertz, which results in a basic cycle time for the computer of less than a nanosecond. Even taking into account the number of multiply-accumulate operations needed to calculate and propagate a new value for a PE (typically 10-100), the basic cycle time for an individual PE is still only about 10 to 100 nanoseconds. In some ways, however, speed is deceptive. Despite its slower cycle, the brain is still able to perform some tasks orders of magnitude faster than today's fastest digital computer. This, most likely, is because of the brain's massively parallel architecture. (Recent research related to neural processing in echo-locating bats, however, indicates that these creatures are physiologically processing signals in a time span of a few hundred nanoseconds, so it seems obvious that we still have much to learn about how the brain functions.) Quantity of PEs There is a significant difference between the number of PEs in the typical ANN and the number of biological neurons involved in any task in a BNN. Typical ANNs are implemented with something like a few dozen to several hundred PEs. Each of the 1,000 main modules in the human brain described by Stubbs (1988) contains about 500 million neurons, and it is almost certain that several (perhaps many) of these main modules are involved in any simple task. Of course, for any practical appli￾cation, most engineers and computer scientists might be hard pressed to figure out how to effectively use a neural network tool (NNT) with 500 million PEs! Some biologically oriented scientists have criticized artificial neural networks because they don't model all the activities of the brain sufficiently well. Our primary goal as engineers and computer scientists, however, is to solve complex problems, not to model the brain. Our interest, then, is in adapting relevant concepts to solve difficult problems. As an oft-quoted saying (oft-quoted in engineering circles, any￾way) puts it, "Scientists study what is. Engineers create what has never been." This statement is not meant to be antagonistic toward scientists. What scientists do is just as noble and worthwhile as what engineers do; they just have a different mission and a different perspective

Biological Basis for Neural Networks Biological Basis for Evolutionary Computation Whereas individuals adapt and learn over their lifetimes using their neural networks to accomplish tasks,species survive by reproducing and evolving over time by pass- ing on new information through their genes.In a manner somewhat analogous to neural networks'ties with biology,the field of evolutionary computation has roots in biological genetics.The concept of chromosomes is central to both genetics and evolutionary computation.2 Chromosomes All living organisms are made up of cells such as neurons,as described earlier.Chro- mosomes are structures in cell nuclei (cell bodies)that transmit genetic information. Each representative of a given species has a characteristic number of chromosomes. Humans normally have 46,occurring as 23 homologous(corresponding)pairs in the female and 22 homologous pairs and one nonidentical pair in the male.One of each pair is derived from the father,one from the mother.A sketch of three pairs of human chromosomes appears in Figure 1.2. Individual patterns,or strings,in evolutionary computation systems are basi- cally analogous to chromosomes in biological systems.In fact,the term chromo- some is commonly used in most genetic algorithm and evolutionary programming systems.In genetics,the collection of chromosomes required to completely specify Figure 1.2 Sketch of three pairs of human chromosomes.The patterns of bands along the chromosomes are the result of a staining technique and allow identification of the individuals of chromosome pairs.Source:Drawing by Mark C.Eberhart. 2 In this text,the term genetics refers to biological genetics,which isa branch of biology that deals with the heredity and variation of organisms"(Webster's New Collegiate Dictionary,1975)

Biological Basis for Neural Networks Biological Basis for Evolutionary Computation Whereas individuals adapt and learn over their lifetimes using their neural networks to accomplish tasks, species survive by reproducing and evolving over time by pass￾ing on new information through their genes. In a manner somewhat analogous to neural networks' ties with biology, the field of evolutionary computation has roots in biological genetics. The concept of chromosomes is central to both genetics and evolutionary computation. 2 Chromosomes All living organisms are made up of cells such as neurons, as described earlier. Chro￾mosomes are structures in cell nuclei (cell bodies) that transmit genetic information. Each representative of a given species has a characteristic number of chromosomes. Humans normally have 46, occurring as 23 homologous (corresponding) pairs in the female and 22 homologous pairs and one nonidentical pair in the male. One of each pair is derived from the father, one from the mother. A sketch of three pairs of human chromosomes appears in Figure 1.2. Individual patterns, or strings, in evolutionary computation systems are basi￾cally analogous to chromosomes in biological systems. In fact, the term chromo￾some is commonly used in most genetic algorithm and evolutionary programming systems. In genetics, the collection of chromosomes required to completely specify Figure 1.2 Sketch of three pairs of human chromosomes. The patterns of bands along the chromosomes are the result of a staining technique and allow identification of the individuals of chromosome pairs. Source: :Drawing by Mark C. Eberhart. 2 In this text, the term genetics refers to biological genetics, which is "a branch of biology that deals with the heredity and variation of organisms" (Webster's New Collegiate Dictionary, 1975)