Recent advances in Pattern Classification Marek R. ogiela' and Lakhmi C Jain AGH University of Science and Technology, Al. Mickiewicz 30, 30-059 Krakow. Poland e-mail: mogielaeagh. edu. pl 2 University of South Australia, School of Electrical and Information Engineering, Adelaide, Mawson Lakes Campus, South Australia SA 5095, Australia e-mail: Lakhmi. jain@unisa. edu.au Abstract. This chapter describes some advances in modern pattern classification chniques, and new classes of information systems dedicated for image analysis, interpretation and semantic classification. In this book we present tions for the development of modern pattern recognition techniques for processin and analysis of several classes of visual patterns, as well as some theoretical foun- dations for modern pattern interpretation approaches. In particular this monograph resents selected areas of application of pattern recognition and classification approaches including handwriting recognition, medical image analysis and inter retation, development of cognitive systems for image computer understanding, moving object detection, advanced image filtration and intelligent multi-object labeling and classification. 1 New Directions in pattern Classification In the field of advanced pattern recognition and computational intelligence meth ds, new directions in the field referred to advanced visual patterns analysis, recognition and interpretation, strongly connected with computational cognitive science or cognitive informatics has recently been distinguished. Computational cognitive science is a new branch of computer science and pattern classification originating mainly from neurobiology and psychology, but is currently also devel- oped by science(e.g. descriptive mathematics)and technical disciplines(informat- ics). In this science, models of the cognitive process taking place in the human brain [2], which is studied by neurophysiologists (at the level of biological mechanisms), psychologists(at the level of analysing specific human behaviours) ind philosophers(at the level of a general reflection on the nature of cognitive processes and their conditions), have become the basis for designing various types of intelligent computer systems M.R. Ogiela and L.C. Jain(Eds. Computational Intelligence Paradigms, SCI 386, Pp 1-4. springerlink.com Springer-Verlag Berlin Heidelberg 2012
M.R. Ogiela and L.C. Jain (Eds.): Computational Intelligence Paradigms, SCI 386, pp. 1–4. springerlink.com © Springer-Verlag Berlin Heidelberg 2012 Chapter 1 Recent Advances in Pattern Classification Marek R. Ogiela1 and Lakhmi C. Jain2 1 AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Kraków, Poland e-mail: mogiela@agh.edu.pl 2 University of South Australia, School of Electrical and Information Engineering, Adelaide, Mawson Lakes Campus, South Australia SA 5095, Australia e-mail: Lakhmi.jain@unisa.edu.au Abstract. This chapter describes some advances in modern pattern classification techniques, and new classes of information systems dedicated for image analysis, interpretation and semantic classification. In this book we present some new solutions for the development of modern pattern recognition techniques for processing and analysis of several classes of visual patterns, as well as some theoretical foundations for modern pattern interpretation approaches. In particular this monograph presents selected areas of application of pattern recognition and classification approaches including handwriting recognition, medical image analysis and interpretation, development of cognitive systems for image computer understanding, moving object detection, advanced image filtration and intelligent multi-object labeling and classification. 1 New Directions in Pattern Classification In the field of advanced pattern recognition and computational intelligence methods, new directions in the field referred to advanced visual patterns analysis, recognition and interpretation, strongly connected with computational cognitive science or cognitive informatics has recently been distinguished. Computational cognitive science is a new branch of computer science and pattern classification originating mainly from neurobiology and psychology, but is currently also developed by science (e.g. descriptive mathematics) and technical disciplines (informatics). In this science, models of the cognitive process taking place in the human brain [2], which is studied by neurophysiologists (at the level of biological mechanisms), psychologists (at the level of analysing specific human behaviours) and philosophers (at the level of a general reflection on the nature of cognitive processes and their conditions), have become the basis for designing various types of intelligent computer systems
2 M.R. Ogiela and L C. jain The requirements of an advanced user are not limited to just collecting, proc- essing and analysing information in computer systems. Today, users expect IT systems to offer capabilities of automatically penetrating the semantic layer as well, as this is the source for developing knowledge, and not just only collecting messages.This is particularly true of information systems or decision support sys tems. Consequently, IT systems based on cognition will certainly be developed in tensively, as they meet the growing demands of the Information Society, in which the ability to reach the contents of information collected in computer systems will be gaining increasing importance. In particular, due to the development of system which, apart from numerical data and text, also collect multimedia information, and particularly images or movies, there is a growing need to develop scientific cornerstones for designing IT systems allowing one to easily find the requisite multimedia information, which conveys a specific meaning in its structure, but which requires the semantic contents on an image to be understood, and not just the objects visible in it to be analysed and possibly classified according to their form. Such systems, capable of not only analysing but also interpreting the mean- ing of the data they process(scenes, real-life contexts, movies etc. ) can also play the role of advisory systems supporting human decision-making, whereas the ef- fectiveness of this support can be significantly enhanced by the system automati cally acquiring knowledge adequate for the problem in question. Information Knowledge perception actions Environment Environmen General Philosophy Fig. 1 Taxonomy of issues explored by cognitive science It is thus obvious that contemporary solutions should aim at the development of new classes of information systems which can be assigned the new name of Cog- nitive Information Systems. We are talking about systems which can process data at a very high level of abstraction and make semantic evaluations of such data Such systems should also have autonomous learning capabilities, which will allow them to improve along with the extension of the knowledge available to them, pre- sented in the form of various patterns and data. Such systems are significantly more complex in terms of the functions they perform than solutions currently em- ployed in practice, so they have to be designed with the use of advanced
2 M.R. Ogiela and L.C. Jain The requirements of an advanced user are not limited to just collecting, processing and analysing information in computer systems. Today, users expect IT systems to offer capabilities of automatically penetrating the semantic layer as well, as this is the source for developing knowledge, and not just only collecting messages. This is particularly true of information systems or decision support systems. Consequently, IT systems based on cognition will certainly be developed intensively, as they meet the growing demands of the Information Society, in which the ability to reach the contents of information collected in computer systems will be gaining increasing importance. In particular, due to the development of system which, apart from numerical data and text, also collect multimedia information, and particularly images or movies, there is a growing need to develop scientific cornerstones for designing IT systems allowing one to easily find the requisite multimedia information, which conveys a specific meaning in its structure, but which requires the semantic contents on an image to be understood, and not just the objects visible in it to be analysed and possibly classified according to their form. Such systems, capable of not only analysing but also interpreting the meaning of the data they process (scenes, real-life contexts, movies etc.), can also play the role of advisory systems supporting human decision-making, whereas the effectiveness of this support can be significantly enhanced by the system automatically acquiring knowledge adequate for the problem in question. Fig. 1 Taxonomy of issues expplored by cognitive science It is thus obvious that contemporary solutions should aim at the development of new classes of information systems which can be assigned the new name of Cognitive Information Systems. We are talking about systems which can process data at a very high level of abstraction and make semantic evaluations of such data. Such systems should also have autonomous learning capabilities, which will allow them to improve along with the extension of the knowledge available to them, presented in the form of various patterns and data. Such systems are significantly more complex in terms of the functions they perform than solutions currently employed in practice, so they have to be designed with the use of advanced
Recent Advances in Pattern Classification achievements of computer technologies. What is more, such systems do not fit the heoretical frameworks of todays information collection and searching systems, so when undertaking the development and practical implementation of Cognitive In- formation Systems, the first task is to find, develop and research new theoretical formalisms adequate for the jobs given to these systems. They will use the theo- retical basis and conceptual formalisms developed for cognitive science by physi ology, psychology and philosophy(see Fig. 1), but they have to adjusted to the new situation, namely the intentional initiation of cognitive processes in techno- logical systems. Informatics has already attempted to create formalisms for sim- pler information systems on this basis [2, 5]. In addition, elements of a cognitive approach are increasingly frequently cropping up in the structure of new generation pattern classification systems [ 3, 6], although the adequate terminology is not always used. On the other hand, some researchers believe that the cognitive domain can be conquered by IT systems just as the researchers of simple percep- tion and classification mechanisms have managed to transplant selected biological observations into the technological domain, namely into artificial neural networks [ 4]. However, the authors have major doubts whether this route will be productive nd efficient, as there is a huge difference in scale between neurobiological proc- esses which are mapped by neural networks and mental processes which should be deployed in cognitive information systems or cognitive pattern recognition ap- proaches. The reason is that whereas neural networks are based on the action of neurons numbering from several to several thousand (at the most), mental proc- esses involve hundreds of millions of neurons in the brain, which is a significant hindrance in any attempt to imitate them with computers. This is why it seems tems on attempts at the behavioural modelling of psychological phenomena and not on the structural imitation of neurophysiological processes. The general foundations for the design of such systems have been the subject of earlier publications [6, 7, 8]. However, it must be said that the methodology of de signing universal systems of cognitive interpretation has yet to be developed fully This applies in particular to systems oriented towards the cognitive analysis of multimedia information. Overcoming the barrier between the form of multimedia information(e.g. the shape of objects in the picture or the tones of sounds) and the sense implicitly contained in this information requires more research initially ori- ented towards detailed goals. Possibly, after some time, it will be possible to ag- regate the experience gained while executing these individual, detailed jobs into a comprehensive, consistent methodology. However, for the time being, we have to satisfy ourselves with achieving individual goals one after another. These goals are mainly about moving away from the analysis of data describing single objects to a more general and semantically deepened analysis of data presenting or de- scribing various components of images or different images from the same video sequence. Some good examples of such visual data analysis will be presented in following chapters
Recent Advances in Pattern Classification 3 achievements of computer technologies. What is more, such systems do not fit the theoretical frameworks of today's information collection and searching systems, so when undertaking the development and practical implementation of Cognitive Information Systems, the first task is to find, develop and research new theoretical formalisms adequate for the jobs given to these systems. They will use the theoretical basis and conceptual formalisms developed for cognitive science by physiology, psychology and philosophy (see Fig. 1), but they have to adjusted to the new situation, namely the intentional initiation of cognitive processes in technological systems. Informatics has already attempted to create formalisms for simpler information systems on this basis [2, 5]. In addition, elements of a cognitive approach are increasingly frequently cropping up in the structure of newgeneration pattern classification systems [3, 6], although the adequate terminology is not always used. On the other hand, some researchers believe that the cognitive domain can be conquered by IT systems just as the researchers of simple perception and classification mechanisms have managed to transplant selected biological observations into the technological domain, namely into artificial neural networks [4]. However, the authors have major doubts whether this route will be productive and efficient, as there is a huge difference in scale between neurobiological processes which are mapped by neural networks and mental processes which should be deployed in cognitive information systems or cognitive pattern recognition approaches. The reason is that whereas neural networks are based on the action of neurons numbering from several to several thousand (at the most), mental processes involve hundreds of millions of neurons in the brain, which is a significant hindrance in any attempt to imitate them with computers. This is why it seems appropriate and right to try to base the design of future Cognitive Information Systems on attempts at the behavioural modelling of psychological phenomena and not on the structural imitation of neurophysiological processes. The general foundations for the design of such systems have been the subject of earlier publications [6, 7, 8]. However, it must be said that the methodology of designing universal systems of cognitive interpretation has yet to be developed fully. This applies in particular to systems oriented towards the cognitive analysis of multimedia information. Overcoming the barrier between the form of multimedia information (e.g. the shape of objects in the picture or the tones of sounds) and the sense implicitly contained in this information requires more research initially oriented towards detailed goals. Possibly, after some time, it will be possible to aggregate the experience gained while executing these individual, detailed jobs into a comprehensive, consistent methodology. However, for the time being, we have to satisfy ourselves with achieving individual goals one after another. These goals are mainly about moving away from the analysis of data describing single objects to a more general and semantically deepened analysis of data presenting or describing various components of images or different images from the same video sequence. Some good examples of such visual data analysis will be presented in following chapters
M.R. Ogiela and L C. jain References 1. Bichindaritz, I, Vaidya, S, Jain, A, Jain, L C(eds ) Computational Intelligence in Healthcare 4. SCl, vol. 309, Pp. 347-369. Springer, Heidelberg (2010) Branquinho, J(ed ) The Foundations of Cognitive Science. Clarendon Press, Oxford (2001) 3. Davis, LS.(ed): Foundations of Image Understanding. Kluwer Academic Publishers 4. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. A Wiley Interscience Publication John Wiley Sons, Inc(2001) 5. Meystel, A M, Albus, J.S.: Intelligent Systems- Architecture, Design, and Control. A Wiley-Interscience Publication John Wiley Sons, Inc, Canada(2002) Ogiela, L, Ogiela, M.R.: Cognitive Techniques in Visual Data Interpretation. Sprin ger, Heidelberg(2009) 7. Ogiela, M.R., Tadeusiewicz, R. Modern Computational Intelligence Methods for the Interpretation of Medical Images. Springer, Heidelberg(2008) 8. Ogiela, M.R., Ogiela, L. Cognitive Informatics in Medical Image Semantic Content nderstanding. In: Kim, T.-H, Stoica, A, Chang, R.-S(eds )Security-Enriched Ur- ban Computing and Smart Grid. CCIS, vol. 78, pp. 131-138. Springer, Heidelberg 9. Tolk, A, Jain, L C: Intelligence-Based Systems Engineering. Intelligence Systems Reference Library 10 (2011) 10. Vernon, D, Metta, G, Sandin. cognitive systems: Imp tions for the autonomous development of mental capabilities in computational agents. IEEE Transactions on Evolutionary Computation 11(2), 151-180(2007)
4 M.R. Ogiela and L.C. Jain References 1. Bichindaritz, I., Vaidya, S., Jain, A., Jain, L.C. (eds.): Computational Intelligence in Healthcare 4. SCI, vol. 309, pp. 347–369. Springer, Heidelberg (2010) 2. Branquinho, J. (ed.): The Foundations of Cognitive Science. Clarendon Press, Oxford (2001) 3. Davis, L.S. (ed.): Foundations of Image Understanding. Kluwer Academic Publishers (2001) 4. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. A WileyInterscience Publication John Wiley & Sons, Inc. (2001) 5. Meystel, A.M., Albus, J.S.: Intelligent Systems – Architecture, Design, and Control. A Wiley-Interscience Publication John Wiley & Sons, Inc., Canada (2002) 6. Ogiela, L., Ogiela, M.R.: Cognitive Techniques in Visual Data Interpretation. Springer, Heidelberg (2009) 7. Ogiela, M.R., Tadeusiewicz, R.: Modern Computational Intelligence Methods for the Interpretation of Medical Images. Springer, Heidelberg (2008) 8. Ogiela, M.R., Ogiela, L.: Cognitive Informatics in Medical Image Semantic Content Understanding. In: Kim, T.-H., Stoica, A., Chang, R.-S. (eds.) Security-Enriched Urban Computing and Smart Grid. CCIS, vol. 78, pp. 131–138. Springer, Heidelberg (2010) 9. Tolk, A., Jain, L.C.: Intelligence-Based Systems Engineering. Intelligence Systems Reference Library 10 (2011) 10. Vernon, D., Metta, G., Sandini, G.: A survey of artificial cognitive systems: Implications for the autonomous development of mental capabilities in computational agents. IEEE Transactions on Evolutionary Computation 11(2), 151–180 (2007)
Chapter 2 Neural Networks for Handwriting Recognition Marcus Liwicki. Alex Graves. and Horst Bunke German Research Center for Artificial Intelligence, Trippstadter Str. 122, 67663 Kaiserslautern, Germany e-mail: marcus. liwicki@dfki. de Institute for Informatics 6, Technical University of Munich, Boltzmannstr 3, 85748 Garching bei Munchen, Germany e-mail: graves@in tum. de Institute for Computer Science and Applied Mathematics, Neubruickstr. 10. 3012 Bern. Switzerland e-mail: bunke@iam unibe. ch Abstract. In this chapter a novel kind of Recurrent Neural Networks(RNNs) is described. Bi- and Multidimensional rnns combined with connectionist tem- poral Classification allow for a direct recognition of raw stroke data or raw pixel data. In general, recognizing lines of unconstrained handwritten text is a challen for even the best current recognizers. Most recent progress in the field has been made either through improved preprocessing, or through advances in language modeling. Relatively little work has been done on the basic recognition algo- rithms. Indeed, most systems rely on the same hidden Markov models that have been used for decades in speech and handwriting recognition, despite their well- known shortcomings. This chapter describes an alternative approach based on a novel type of recurrent neural network, specifically designed for sequence labeling tasks where the data is hard to segment and contains long-range, bidirectional or multidirectional interdependencies. In experiments on two unconstrained handwriting databases, the new approach achieves word recognition accuracies of 79, 7% on on-line data and 74, 1%o on off-line data, significantly outperforming a state-of-the-art HMM-based system. Promising experimental results on various other datasets from different countries are also presented. A toolkit implementing he networks is freely available for public 1 Introduction Handwriting recognition is traditionally divided into on-line and off-line recogni tion. In on-line recognition a time ordered sequence of coordinates, representing M. R. Ogiela and L C. Jain(Eds): tational Intelligence Paradigms, SCI 386, pp springerlink.com e Springer-Verlag Berlin Heidelberg
M.R. Ogiela and L.C. Jain (Eds.): Computational Intelligence Paradigms, SCI 386, pp. 5–24. springerlink.com © Springer-Verlag Berlin Heidelberg 2012 Chapter 2 Neural Networks for Handwriting Recognition Marcus Liwicki1 , Alex Graves2 , and Horst Bunke3 1 German Research Center for Artificial Intelligence, Trippstadter Str. 122, 67663 Kaiserslautern, Germany e-mail: marcus.liwicki@dfki.de 2 Institute for Informatics 6, Technical University of Munich, Boltzmannstr. 3, 85748 Garching bei München, Germany e-mail: graves@in.tum.de 3 Institute for Computer Science and Applied Mathematics, Neubrückstr. 10, 3012 Bern, Switzerland e-mail: bunke@iam.unibe.ch Abstract. In this chapter a novel kind of Recurrent Neural Networks (RNNs) is described. Bi- and Multidimensional RNNs combined with Connectionist Temporal Classification allow for a direct recognition of raw stroke data or raw pixel data. In general, recognizing lines of unconstrained handwritten text is a challenging task. The difficulty of segmenting cursive or overlapping characters, combined with the need to assimilate context information, has led to low recognition rates for even the best current recognizers. Most recent progress in the field has been made either through improved preprocessing, or through advances in language modeling. Relatively little work has been done on the basic recognition algorithms. Indeed, most systems rely on the same hidden Markov models that have been used for decades in speech and handwriting recognition, despite their wellknown shortcomings. This chapter describes an alternative approach based on a novel type of recurrent neural network, specifically designed for sequence labeling tasks where the data is hard to segment and contains long-range, bidirectional or multidirectional interdependencies. In experiments on two unconstrained handwriting databases, the new approach achieves word recognition accuracies of 79,7% on on-line data and 74,1% on off-line data, significantly outperforming a state-of-the-art HMM-based system. Promising experimental results on various other datasets from different countries are also presented. A toolkit implementing the networks is freely available for public. 1 Introduction Handwriting recognition is traditionally divided into on-line and off-line recognition. In on-line recognition a time ordered sequence of coordinates, representing