PROC. OF THE IEEE, NOVEMBER 1998 Gradient-Based Learning Applied to Document Recognition Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner Abstract I. INTRODUCTION Multilayer Neural Networks trained with the backpropa- gation algorithm constitute the best example of a successful Over the last several years, machine learning techniques, Gradient-Based Learning technique. Given an appropriate particularly when applied to neural networks, have played network architecture, Gradient-Based Learning algorithms can be used to synthesize a complex decision surface that can an increasingly important role in the design of pattern classify high-dimensional patterns such as handwritten char- recognition systems. In fact, it could be argued that the acters, with minimal preprocessing. This paper reviews var availability of learning techniques has been a crucial fac- ious methods applied to handwritten character recognition and compares them on a standard handwritten digit recog- tor in the recent success of pattern recognition applica- nition task. Convolutional Neural Networks, that are specif- tions such as continuous speech recognition and handwrit- ically designed to deal with the variability of 2D shapes, are ing recognition. shown to outperform all other techniques. The main message of this paper is that better pattern Real-life document recognition systems are composed of multiple modules including field extraction, segmenta recognition systems can be built by relying more on auto- tion, recognition, and language modeling. A new learning matic learning, and less on hand-designed heuristics. This paradigm, called Graph Transformer Networks (GTN), al- is made possible by recent progress in machine learning lows such multi-module systems to be trained globally using Gradient-Based methods so as to minimize an overall per and computer technology. Using character recognition as formance meas ure. a case study, we show that hand-crafted feature extrac- Two systems for on-line handwriting recognition are de- tion can be advantageously replaced by carefully designed scribed. Experiments demonstrate the advantage of global learning machines that operate directly on pixel images. training, and the flexibility of Graph Transformer Networks A Graph Transformer Network for reading bank check is Using document understanding as a case study, we show also described. It uses Convolutional Neural Network char- that the traditional way of building recognition systems by acter recognizers combined with global training techniques manually integrating individually designed modules can be to provides record accuracy on business and personal checks. It is deployed commercially and reads several million checks replaced by a unified and well-principled design paradigm, per day. called Graph Transformer Networks, that allows training Keywords- Neural Networks, OCR, Document Recogni all the modules to optimize a global performance criterion. tion, Machine Learning, Gradient-Based Learning, Convo- Since the early days of pattern recognition it has been lutional Neural Networks, Graph Transformer Networks, Fi- known that the variability and richness of natural data, nite State Transducers be it speech, glyphs, or other types of patterns, make it almost impossible to build an accurate recognition system NOMENCLATURE entirely by hand. Consequently, most pattern recognition . GT Graph transformer. systems are built using a combination of automatic learn- . GTN Graph transformer network. ing techniques and hand-crafted algorithms. The usual . HMM Hidden Markov model. method of recognizing individual patterns consists in divid- . HOS Heuristic oversegmentation ing the system into two main modules shown in figure 1. The first module, called the feature extractor, transforms . K-NN K-nearest neighbor. .NN Neural network. the input patterns so that they can be represented by low- . OCR Optical character recognition. dimensional vectors or short strings of symbols that(a)can . PCA Principal component analysis. be easily matched or compared, and (b) are relatively in- . RBF Radial basis function. variant with respect to transformations and distortions of . RS-SVM Reduced-set support vector method. the input patterns that do not change their nature. The . SDNN Space displacement neural network. feature extractor contains most of the prior knowledge and . SVM Support vector method. is rather specific to the task. It is also the focus of most of . TDNN Time delay neural network. the design effort, because it is often entirely hand-crafted. . V-SVM Virtual support vector method. The classifier, on the other hand, is often general-purpose and trainable. One of the main problems with this ap- proach is that the recognition accuracy is largely deter The authors are with the Speech and Image Pro- cessing Services Research Laboratory, AT&T Labs- mined by the ability of the designer to come up with an Research, 100 Schulz Drive Red Bank, NJ 07701. E-mail: appropriate set of features. This turns out to be a daunt- {yann,leonb,yoshua,haffner}@research.att.com. Yoshua Bengio ing task which, unfortunately, must be redone for each new is also with the Departement d'Informatique et de Recherche Operationelle, Universite de Montreal, C.P. 6128 Succ. Centre-Ville, problem. A large amount of the pattern recognition liter- 2920 Chemin de la Tour, Montreal, Quebec, Canada H3C 3J7. ature is devoted to describing and comparing the relative
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✜ ★✩✫✪✭✬✯✮✟✰✲✱✴✳✶✵✶✷✪✹✸✺✰✻✬ ✼✽✰✾✪✭✩✿✱✯✮❀✱❂❁ ❃❄❅❄✯❆❇✮✟✰✻✬ ✳✫❈ ❉❈❋❊❍●❅■✰✲✱✴✳ ❏✰❍❊❑❈▲❁▼✱❅✮◆✳❖✮☛❈P✱ ◗❙❘◆❚❯❚❲❱❨❳✂❩❭❬❯❚✛❪❫❱✺❴❳❛❵✟❚✾❜❭❵◆❝❞❝❞❵✟❬✛❪❇◗✒❵◆❡❞❢✝❬❣❘❤❜❭❳✐❚❦❥◆❧♠❵❦❪❣❘✎❚❦♥❲♦♣❘✎❝rq✖❧♠s✠t❍✉✢❘✎✈✇❚❦❳✐q ①❣②④③⑥⑤✓⑦⑨⑧❶⑩✧⑤✓❷ ❸❺❹✂❻❽❼❿❾➀❻❽➁❞➂➄➃✖➅✛➆♣➃❞❹➇➅✥➁➈❻◆➆➉➃➊❼④➋❫➌➈➅❿➍✐➎➏❼❿➅✥➁➈❾➑➐✐➃❞➒➓➋➔❾→❼↔➣❭❼❿➣✐➃✌↕➇➁➄➙❿➍❛➛✂➅✥➌➈➛➇➁r➜ ➝➁r❼↔❾❽➌❛➐➞➁➈❻➝➌➈➅↔❾→❼↔➣✂➟✾➙❶➌❛➐➇➎➠❼↔❾→❼↔❹❛❼✧➃✛❼❿➣✐➃♣↕✙➃✖➎④❼❫➃➊➡✐➁➈➟❭➛✙❻❽➃❨➌➈➢✎➁➤➎✧❹➇➙❶➙❶➃✖➎✤➎✤➢⑨❹✂❻ ➥➅✥➁➄➒✙❾❽➃r➐✠❼✧➜♠➦✇➁➄➎④➃✖➒✫➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ❼✥➃✖➙↔➣✂➐✂❾➩➨➇❹✐➃➄➫ ➥❾➩➭➄➃r➐✿➁➈➐✺➁➄➛✂➛✂➅✥➌➈➛✂➅↔❾❽➁r❼✥➃ ➐✐➃❶❼④➋✇➌➈➅❿➍✿➁➄➅✥➙↔➣✂❾❽❼✧➃❞➙❿❼❿❹➇➅✥➃➄➯ ➥➅✥➁➄➒✙❾➩➃❞➐➄❼✤➜⑥➦❫➁➄➎✤➃✖➒➲➧◆➃➊➁➄➅↔➐✂❾➀➐➝ ➁➈❻➝➌➈➅↔❾❽❼❿➣✂➟❭➎ ➙↔➁➈➐♣↕✙➃✇❹➇➎✤➃✖➒➳❼✥➌➵➎✤➂➇➐➄❼❿➣✐➃❞➎✧❾➩➸❶➃➏➁✛➙↔➌❛➟❭➛✙❻❽➃✖➡➔➒➇➃✖➙➊❾➑➎✧❾➩➌❛➐♣➎✧❹➇➅✥➢➺➁➄➙↔➃❦❼↔➣✐➁r❼☛➙❶➁➈➐ ➙➊❻➩➁➄➎✤➎✥❾❽➢➺➂➤➣✂❾➝➣❛➜⑥➒✙❾➑➟❙➃r➐➇➎✧❾➩➌❛➐✐➁➈❻r➛➇➁r❼✤❼✥➃✖➅↔➐➇➎✝➎✧❹➇➙↔➣➤➁➄➎❇➣✐➁➈➐➇➒➇➋✛➅↔❾❽❼✤❼✥➃❞➐➳➙↔➣✐➁➄➅✤➜ ➁➄➙❿❼✥➃✖➅❿➎↔➯➄➋➔❾❽❼❿➣✌➟➻❾➑➐✂❾➀➟❙➁➈❻✠➛✂➅✥➃✖➛✂➅✥➌➇➙↔➃✖➎✧➎✧❾➀➐➝ ➫❯➼➵➣✂❾➩➎❯➛➇➁➄➛✙➃❞➅❣➅✧➃✖➭➇❾➩➃➊➋❨➎❇➭✖➁➄➅✧➜ ❾➩➌❛❹➇➎➉➟❙➃❶❼↔➣✐➌✐➒✂➎❨➁➄➛✂➛✙❻➀❾➩➃✖➒✢❼✧➌✶➣✐➁➈➐➇➒➇➋❨➅↔❾→❼✧❼✧➃r➐✢➙↔➣✐➁➄➅✥➁➄➙❿❼✥➃✖➅➉➅✥➃❞➙↔➌➝➐✂❾❽❼❿❾➩➌❛➐ ➁➈➐➇➒✶➙❶➌❛➟❭➛➇➁➄➅✧➃❞➎➏❼↔➣✐➃❞➟✭➌❛➐➽➁❙➎④❼✧➁➈➐➇➒➇➁➄➅❿➒✶➣✐➁➈➐➇➒➇➋❨➅↔❾→❼✧❼✧➃❞➐➽➒✙❾➝❾❽❼❨➅✧➃❞➙↔➌➝➜ ➐✂❾❽❼❿❾➩➌❛➐➉❼✧➁➄➎✤➍☎➫❯➾✇➌❛➐➈➭➄➌❛❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻r➆♣➃❞❹➇➅✥➁➈❻➈➆♣➃❶❼④➋✇➌➈➅❿➍❛➎❶➯✖❼↔➣✐➁r❼❀➁➄➅✥➃✇➎✧➛✙➃✖➙✖❾❽➢✓➜ ❾➑➙↔➁➈❻➑❻➩➂➞➒➇➃✖➎✥❾➝➐✐➃✖➒❙❼✧➌✌➒➇➃➊➁➈❻✙➋➔❾→❼↔➣✌❼↔➣✐➃❨➭✖➁➄➅↔❾➩➁➄↕✙❾➑❻➀❾→❼④➂➤➌➄➢☛➚✠➪✻➎✥➣✐➁➄➛✙➃❞➎↔➯➇➁➄➅✧➃ ➎✧➣✐➌r➋➔➐➻❼✥➌✒➌❛❹❛❼❿➛✙➃❞➅✧➢➺➌❛➅❿➟➶➁➈❻➀❻✎➌✠❼❿➣✐➃❞➅✛❼✧➃❞➙❿➣✂➐✂❾➑➨➇❹✐➃✖➎↔➫ ➹❨➃➊➁➈❻❽➜⑥❻➀❾❽➢➺➃❲➒➇➌✐➙✖❹✂➟✒➃r➐✠❼➘➅✥➃✖➙↔➌➝➐✂❾→❼↔❾❽➌❛➐▲➎✤➂✐➎④❼✧➃r➟❙➎❤➁➄➅✥➃❲➙↔➌✐➟❙➛☎➌➈➎④➃❞➒ ➌➄➢✒➟❙❹✂❻→❼↔❾➩➛✙❻➩➃❖➟❙➌✐➒✙❹✂❻➩➃✖➎✿❾➑➐➇➙➊❻➀❹➇➒✙❾➀➐➝➷➴➃❞❻➑➒❍➃➊➡❛❼✥➅✥➁➄➙❿❼↔❾❽➌❛➐✎➯➉➎✤➃➝➟❙➃r➐✠❼✥➁r➜ ❼❿❾➩➌❛➐✎➯❇➅✥➃❞➙↔➌➝➐✂❾❽❼❿❾➩➌❛➐✎➯❀➁➈➐➇➒➷❻❽➁➈➐➝❹✐➁➝➃➻➟❙➌➇➒➇➃❞❻➀❾➑➐➝ ➫✶➬▲➐✐➃✖➋✭❻➩➃➊➁➄➅↔➐✂❾➀➐➝ ➛➇➁➄➅✥➁➄➒✙❾➝➟➽➯☛➙↔➁➈❻➀❻❽➃❞➒ ➥➅✥➁➄➛✙➣➲➼☛➅✥➁➈➐➇➎④➢➺➌➈➅↔➟❙➃✖➅♣➆➉➃➊❼④➋❫➌➈➅❿➍✐➎❙➮➥➼❫➆➤➱❿➯☛➁➈❻→➜ ❻➩➌❞➋❨➎❯➎✧❹➇➙↔➣❭➟✒❹✂❻❽❼❿❾❽➜⑥➟❙➌✐➒✙❹✂❻➩➃❫➎✤➂✐➎④❼✧➃❞➟❭➎❀❼✧➌➤↕✙➃✛❼❿➅✧➁➈❾➀➐✐➃✖➒ ➝❻❽➌❛↕➇➁➈❻➑❻➩➂➤❹➇➎✥❾➑➐➝ ➥➅✥➁➄➒✙❾❽➃r➐✠❼✧➜♠➦✇➁➄➎④➃✖➒➲➟❙➃➊❼❿➣✐➌➇➒✂➎♣➎✤➌✶➁➄➎➉❼✧➌✿➟➻❾➀➐✂❾➑➟➻❾➩➸❶➃✌➁➈➐✫➌r➭➄➃❞➅✥➁➈❻➑❻❀➛✙➃✖➅✧➜ ➢➺➌➈➅↔➟❙➁➈➐➇➙↔➃➳➟❙➃✖➁➄➎✧❹➇➅✥➃➄➫ ➼➏➋❫➌➽➎✤➂✐➎④❼✧➃r➟❙➎➉➢➺➌➈➅➳➌❛➐❛➜⑥❻➀❾➀➐✐➃❭➣✐➁➈➐➇➒➇➋❨➅↔❾→❼↔❾➑➐➝ ➅✥➃✖➙❶➌➝➐✂❾❽❼❿❾➩➌❛➐✢➁➄➅✧➃❭➒➇➃❶➜ ➎✤➙➊➅❿❾➑↕✙➃❞➒◆➫✒✃❀➡➇➛✙➃❞➅↔❾➑➟❙➃❞➐➄❼✥➎➉➒➇➃❞➟❙➌❛➐➇➎➠❼❿➅✥➁r❼✧➃✌❼↔➣✐➃✒➁➄➒➇➭❞➁➈➐✠❼✥➁➝➃✒➌➄➢ ➝❻❽➌➈↕➇➁➈❻ ❼✥➅✥➁➈❾➀➐✂❾➑➐➝ ➯✖➁➈➐➇➒➳❼❿➣✐➃✇❐✙➃➊➡✂❾➑↕✙❾➑❻➀❾→❼④➂➉➌➄➢ ➥➅✧➁➄➛✙➣➳➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅❀➆♣➃❶❼④➋❫➌❛➅✥➍✐➎↔➫ ➬ ➥➅✥➁➄➛✙➣➽➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅✇➆♣➃❶❼④➋✇➌➈➅❿➍✒➢➺➌➈➅➉➅✧➃✖➁➄➒✙❾➑➐➝ ↕➇➁➈➐➇➍➻➙❿➣✐➃❞➙❿➍➽❾➩➎ ➁➈❻➑➎④➌✒➒➇➃❞➎✤➙❶➅↔❾➑↕✙➃✖➒◆➫➏❒♠❼➔❹➇➎④➃❞➎✛➾✇➌❛➐➈➭➄➌❛❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻➇➆♣➃❞❹➇➅✥➁➈❻◆➆➉➃➊❼④➋❫➌➈➅❿➍❭➙↔➣✐➁➄➅✤➜ ➁➄➙❿❼✥➃✖➅➳➅✧➃❞➙↔➌➝➐✂❾➩➸❶➃❞➅✥➎➉➙↔➌❛➟➞↕✙❾➀➐✐➃✖➒✢➋➔❾→❼↔➣ ➝❻❽➌❛↕➇➁➈❻◆❼✥➅✥➁➈❾➀➐✂❾➑➐➝ ❼✧➃❞➙↔➣✂➐✂❾➩➨➇❹✐➃✖➎ ❼✧➌➔➛✂➅✥➌r➭✂❾➩➒➇➃❞➎☛➅✧➃❞➙↔➌➈➅❿➒➳➁➄➙❶➙➊❹➇➅✥➁➄➙↔➂➔➌✐➐➤↕✙❹➇➎✧❾➀➐✐➃✖➎✤➎✝➁➈➐➇➒✌➛✙➃✖➅❿➎✤➌❛➐✐➁➈❻✠➙↔➣✐➃✖➙❿➍✐➎↔➫ ❒♠❼✇❾➑➎❣➒➇➃✖➛✙❻➩➌❞➂➈➃✖➒❭➙↔➌❛➟➻➟❙➃✖➅❿➙➊❾➩➁➈❻➑❻➩➂➉➁➈➐➇➒❭➅✥➃➊➁➄➒✂➎❣➎④➃✖➭➄➃✖➅✥➁➈❻✂➟➻❾➀❻➑❻➀❾❽➌✐➐➤➙❿➣✐➃❞➙❿➍❛➎ ➛✙➃❞➅➉➒➇➁❞➂✠➫ ❮✇❰✤Ï↔Ð◆Ñ ⑦⑨Ò↔③♠❷ ➆♣➃❞❹➇➅✥➁➈❻✇➆➉➃➊❼④➋❫➌➈➅❿➍✐➎↔➯➏Ó➔➾➵➹➤➯❯➪➉➌✐➙➊❹✂➟❙➃r➐✠❼➻➹❨➃✖➙↔➌➝➐✂❾→➜ ❼❿❾➩➌❛➐✎➯❦❸➲➁➄➙↔➣✂❾➑➐✐➃➽➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ➯ ➥➅✥➁➄➒✙❾➩➃❞➐➄❼✤➜⑥➦❫➁➄➎✤➃✖➒❖➧◆➃✖➁➄➅❿➐✂❾➀➐➝ ➯❯➾✇➌❛➐➈➭➄➌✠➜ ❻➀❹❛❼❿❾➩➌❛➐✐➁➈❻r➆➉➃r❹➇➅✥➁➈❻➄➆♣➃❶❼④➋❫➌❛➅✥➍✐➎↔➯ ➥➅✧➁➄➛✙➣➳➼☛➅✥➁➈➐➇➎④➢➺➌❛➅❿➟❙➃❞➅☛➆➉➃➊❼④➋❫➌➈➅❿➍✐➎↔➯❞Ô❇❾→➜ ➐✂❾❽❼✧➃➞Õ❛❼✧➁r❼✥➃➳➼☛➅✥➁➈➐➇➎✤➒✙❹➇➙❶➃✖➅❿➎↔➫ Ö➻×❀Ø♣Ù✙Ú❀Û◆Ü➈Ý✂Þ☛ß✝à☛Ù á➲â➤ã❲â➳ä✥å➈æ✙ç✿è✧ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä❞ð á➲â➤ã❨ñòâ➳ä❿å➄æ✙ç✢è✧ä❿å➄é✎ê④ëíì❛ä✧î➻ï➊ä✛é✙ï➊è④ó✇ì❛ä✧ô✟ð á✫õ♣ö➲ö÷õ➔ø✓ù✙ù✂ï➊é✫ö➲å➈ä✧ô❛ì✠ú➻î❭ì✂ù✂ï✖ûüð á✫õ➤ý➤þ✶õ➉ï➊ÿ✙ä✥ø➀ê✤è✧ø✁➤ì✠ú➈ï✖ä✥ê✧ï✄✂❛î➻ï➊é✐è✥å➄è✧ø➀ì➈é✝ð á✆☎✞✝➠ñ➉ñ✟☎✞✝⑥é✙ï✖å➈ä✧ï❞ê④è✛é✙ï➊ø✠✂➈ç☛✡◆ì➈ä❞ð á✫ñ♣ñ ñ➔ï➊ÿ☎ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á❖ý✌☞✎✍ ý♣æ✂è✧ø✁➊å➈û✏❿ç☎å➄ä❿å✑❶è✧ï✖ä➵ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é✝ð á✆✓✔☞✎✕✖✓❫ä✧ø➀é✗➊ø➑æ☎å➈û✏❶ì❛î❭æ◆ì➈é☎ï➊é✐è➔å➄é☎å➈û✙✘✂ê✧ø➀ê✖ð á✆✍✛✚✔✜✢✍➉å➈ù✂ø✓å➄û✏✡✎å➈ê✧ø➀ê✇ëíÿ✙é✗❶è✧ø➀ì➈é✝ð á✆✍➉þ☛✝➠þ☛✣➳ö✤✍❨ï✖ù✂ÿ✥❶ï✖ù✦✝⑥ê✧ï❶è➔ê✧ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï✒❶è✧ì❛ä➵î➻ï➊è✧ç✙ì✂ù☛ð á➲þ✦✧➳ñ➉ñòþ✂æ☎å✑➊ï➞ù✂ø➀ê✧æ✙û✓å✑➊ï➊î➻ï➊é✐è❨é✙ï➊ÿ☎ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á➲þ☛✣➳ö þ➇ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï★↔è✧ì❛ä✛î❭ï➊è✧ç✙ì✂ù☛ð á✫ã✩✧➳ñ➉ñ ã✛ø➀î➻ï➞ù✂ï➊û✓å✪✘➽é✙ï✖ÿ✙ä✥å➈û☛é✙ï❶è④ó➵ì➈ä✥ô✟ð á✫✣✬✝➠þ☛✣➳ö✭✣➉ø➀ä✤è✥ÿ☎å➄û✝ê✧ÿ✙æ✙æ◆ì➈ä✧è➔ú➈ï★↔è✧ì❛ä✛î❭ï➊è✧ç✙ì✂ù☛ð ✮✏✯✪✰✲✱✴✳✒✵✶✯✪✷✹✸✻✺✼✱✴✸✶✰✾✽❀✿ ✵✶✯ ✵✶✯✪✰❂❁✒❃❄✰❅✰❇❆✶✯ ✱✴❈★❉ ❊●❋✛✱✴❍✹✰❏■✥✸✶✷✴❑ ❆❅✰❅✺✶✺✶✿▲❈★❍ ❁✒✰❅✸✻▼✒✿▲❆❇✰❅✺ ◆❀✰❅✺✶✰❇✱✴✸✻❆❖✯ P✦✱✴◗❄✷✹✸❖✱❘✵✻✷✹✸✻❙❯❚ ❱✗✮❳❲❨✮ P☛✱✴◗✪✺✻❑ ◆✏✰❇✺✻✰❇✱✴✸✶❆❖✯❩❚❭❬❇❪✹❪❫❁✒❆❖✯✄✳✪❴▲❵✖❛❜✸✶✿▼❯✰❝◆✏✰❇❉❡❞❢✱✴❈✪❣❄❚✢❤❳✐❫❪❯❥✹❥✴❪★❬✹❦ ❧✗❑●❋✛✱✴✿▲❴✁♠ ♥ ❙✄✱✴❈✪❈❩❚ ❴▲✰❅✷✹❈✄◗❩❚ ❙❯✷✹✺✶✯✄✳♦✱★❚ ✯♦✱❘♣✑❈✪✰❅✸rq✴s❜✸✶✰❅✺✶✰❇✱✴✸✶❆✶✯❩❦ ✱❘✵✻✵❇❦ ❆❅✷✹❋t❦ ✉✈✷✹✺✻✯✒✳♦✱✭❞✇✰❇❈★❍✹✿▲✷ ✿▲✺①✱✴❴▲✺✻✷❝✽❀✿ ✵✻✯②✵✻✯✪✰❭❛④③✰❇❃✪✱✴✸✻✵✻✰❇❋✩✰❅❈❯✵✢❉❩⑤ ❊●❈★⑥✙✷✹✸✶❋✛✱❘✵✻✿▲⑦✒✳★✰⑧✰r✵✢❉✪✰⑨◆❀✰❅❆❖✯✪✰❅✸✻❆❖✯✪✰ ⑩❃✈③✰❅✸❖✱❘✵✻✿▲✷✹❈✪✰❅❴▲❴▲✰✹❚❄❶❳❈★✿▼❯✰❇✸✻✺✶✿ ✵✒③✰❷❉✪✰❹❸t✷✹❈❯✵✶✸✒③✰❺✱✴❴✁❚❄❻❨❦ ■❼❦♦❽★❬❇❾✹❿✛❁✄✳✪❆❅❆❯❦❢❻✇✰❇❈❯✵✻✸✶✰r❑✁➀❨✿▲❴▲❴▲✰✹❚ ❾✹➁✹❾✹❪✛❻✇✯✪✰❅❋✩✿▲❈✛❉✪✰❹❴➂✱➃✮✦✷✹✳✪✸❇❚♦❸t✷✹❈❯✵✶✸✒③✰❺✱✴❴✁❚❄➄❜✳✇③✰❅◗✑✰❅❆✹❚✑❻❢✱✴❈♦✱✴❉♦✱✩➅❳➆❯❻✫➆✹✐✄❥✒❦ ➇✪➈➉➇❿Ú☛Þ☛à✟×❀➊❇ß❀Û✎Þ❢➋í×❀Ú ý♣ú➈ï➊ä❦è✥ç✙ï➔û✓å➈ê✤è❫ê✤ï✖ú➈ï✖ä✥å➈û✦✘➈ï✖å➈ä✥ê✒➌➄î➓å✑❿ç☎ø➑é✙ï❨û➀ï✖å➈ä✧é☎ø➑é❼✂➳è✥ï✒❿ç✙é✙ø✁➍✐ÿ✙ï✖ê✒➌ æ☎å➈ä✤è✥ø✠➊ÿ✙û➀å➈ä✧û✠✘➽ó❨ç✙ï✖é✫å➄æ✙æ✙û➀ø➀ï✖ù✢è✧ì➓é✙ï✖ÿ✙ä❿å➄û✝é✙ï➊è④ó✇ì❛ä✧ô✂ê✒➌➇ç☎årú➈ï➤æ✙û✓å✪✘➈ï❞ù å➄é ø➀é✗➊ä✧ï❞å➈ê✧ø➑é❼✂❛û✙✘✲ø➑î➻æ◆ì➈ä✧è✥å➄é✐è❖ä✥ì➈û➀ï ø➀é è✧ç✙ï❑ù✂ï✖ê✧ø✙✂❛é ì➄ë➓æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é➷ê❺✘➇ê✤è✧ï✖î➓ê➊ð➐➏➠é➷ë⑨å✑❶è✒➌❯ø➩è➑➊ì➈ÿ✙û✓ù➒✡✎ï✢å➄ä❘✂➈ÿ✙ï❞ù❖è✥ç☎å✠è✒è✧ç✙ï årú✠å➄ø➀û➀å✑✡✙ø➀û➑ø➑è❅✘❖ì➈ë➔û➑ï❞å➄ä✥é✙ø➑é✗✂✫è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✖ê➞ç✎å➈ê➓✡◆ï➊ï✖é❤å➔➊ä✧ÿ✥❶ø✓å➄û➏ë⑨å➎✹✝ è✧ì❛ä✢ø➀é✻è✥ç✙ï ä✧ï★❶ï➊é✐è✺ê✤ÿ✗✒❶ï❞ê✧ê✢ì➄ë➞æ☎å➄è✤è✥ï➊ä✥é✲ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✻å➈æ✙æ✙û➀ø✠✖å♦✝ è✧ø➀ì➈é✎ê❨ê✤ÿ✗❿ç✿å➈ê✩➊ì➈é✐è✧ø➀é➇ÿ✙ì➈ÿ☎ê✛ê✧æ✎ï✖ï✒❿ç✢ä✧ï★❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✢å➈é☎ù✢ç☎å➄é☎ù✙ó❨ä✧ø➑è❇✝ ø➀é❼✂➻ä✧ï★❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✝ð ã✛ç✙ï✶î➓å➄ø➀é î➻ï✖ê✥ê✥å❄✂➈ï➓ì➈ë❨è✧ç✙ø✓ê❙æ☎å➄æ◆ï➊ä❙ø➀ê✒è✧ç☎å➄è➑✡◆ï❶è✧è✧ï➊ä❭æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ê❺✘➇ê✤è✧ï✖î➓ê✬➊å➈é✫✡✎ï➓✡☎ÿ✙ø➑û➑è✬✡☛✘✿ä✧ï✖û✙✘➇ø➀é❼✂➓î➻ì➈ä✥ï➳ì❛é➲å➄ÿ✂è✥ì❄✝ î➓å✠è✥ø✠➤û➀ï✖å➈ä✧é✙ø➀é❼✂✥➌✙å➄é☎ù✢û➑ï❞ê✧ê➔ì➈é✿ç☎å➄é☎ù✦✝⑥ù✂ï❞ê✤ø✠✂➈é☎ï✖ù✢ç✙ï➊ÿ✙ä✥ø✓ê④è✥ø✠✖ê➊ð➏ã✛ç✙ø✓ê ø✓ê✶î➓å➈ù✙ï➲æ✎ì✐ê✧ê✧ø✠✡✙û➑ï➔✡❩✘➘ä✥ï✒➊ï➊é✐è✶æ✙ä✥ì✑✂❛ä✧ï❞ê✧ê➻ø➀é✻î➓å✑❿ç✙ø➀é✙ï➲û➀ï✖å➈ä✧é☎ø➑é❼✂ å➄é✎ù➒➊ì➈î➻æ✙ÿ✂è✥ï➊ä➤è✥ï✒❿ç✙é✙ì❛û➑ì➎✂✑✘❛ð➣→➉ê✧ø➑é❼✂✆❿ç☎å➈ä✥å➎↔è✥ï➊ä➤ä✧ï★❶ì➎✂➈é✙ø➑è✧ø➀ì➈é➷å➈ê å✢✖å➈ê✧ï❖ê④è✥ÿ☎ù✦✘➎➌❨ó✇ï❖ê✧ç✙ì✠ó▼è✧ç✎å✠è✢ç✎å➄é☎ù☛✝r❶ä❿å✠ë➺è✥ï✖ù❑ëíï✖å➄è✧ÿ✙ä✥ï➲ï✄↔✐è✥ä✥å➎✹✝ è✧ø➀ì➈é↕✖å➄é✫✡◆ï✒å➈ù✂ú✠å➈é❛è❿å❄✂❛ï➊ì➈ÿ✎ê✤û✠✘➽ä✥ï➊æ✙û✓å✑➊ï✖ù➐✡☛✘➙➊å➈ä✧ï➊ëíÿ✙û➀û✙✘✺ù✂ï✖ê✧ø✠✂➈é✙ï❞ù û➀ï✖å➄ä✥é✙ø➀é❼✂➷î➓å✑❿ç✙ø➀é✙ï✖ê❭è✥ç☎å✠è➽ì❛æ✎ï✖ä✥å➄è✧ï➲ù✂ø➀ä✥ï✒↔è✥û✙✘❤ì➈é➘æ✙ø✙↔➇ï✖û➔ø➀î➻å✑✂➈ï❞ê➊ð →➉ê✧ø➀é❼✂❺ù✂ì✦➊ÿ✙î➻ï➊é✐è❙ÿ✙é✎ù✂ï➊ä❿ê④è❿å➄é☎ù✙ø➑é❼✂❺å➈ê✒å➛➊å❛ê✤ï✶ê✤è✧ÿ✎ù✦✘✑➌❦ó➵ï✢ê✧ç✙ì✠ó è✧ç✎å✠è➏è✥ç✙ï❨è✧ä❿å➈ù✙ø➩è✥ø➑ì❛é☎å➄û✙ó✛å✪✘➞ì➈ë✇✡☎ÿ✙ø➑û✓ù✂ø➀é❼✂➞ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❭ê❺✘✂ê④è✥ï➊î➓ê❷✡☛✘ î➓å➄é➇ÿ☎å➈û➑û✠✘➞ø➀é❛è✥ï✄✂❛ä✥å➄è✧ø➀é❼✂➤ø➑é☎ù✙ø➑ú➇ø✓ù✂ÿ☎å➄û➀û✠✘❭ù✂ï❞ê✤ø✠✂➈é☎ï✖ù❙î❭ì✂ù✂ÿ☎û➑ï❞ê➜➊å➈é➝✡✎ï ä✥ï➊æ✙û✓å✑➊ï✖ù➣✡☛✘➽å❙ÿ✙é✙ø✙➞☎ï✖ù✿å➄é✎ù➽ó➵ï➊û➀û➟✝⑥æ✙ä✥ø➑é✗➊ø➑æ☎û➑ï❞ù✶ù✂ï❞ê✤ø✠✂➈é✢æ☎å➄ä❿å➈ù✂ø✠✂➈î✫➌ ➊å➈û➑û➀ï✖ù②➠✎➡❘➢❘➤❼➥➧➦✦➡❺➢❄➨❼➩➭➫❯➯♦➡✹➲➵➳✄➡➝➸➑➳❯➺✶➻➜➯♦➡❺➼♦➩✹➌❣è✧ç☎å➄è➻å➈û➑û➀ì✠ó➔ê✌è✥ä✥å➈ø➑é☎ø➑é❼✂ å➄û➀û✙è✥ç✙ï➉î➻ì✂ù✂ÿ✙û➀ï✖ê➏è✧ì➞ì❛æ✂è✧ø➀î➻ø✙➽✖ï♣å✌✂❛û➑ì➎✡☎å➄û☎æ✎ï✖ä✤ëíì❛ä✧î➓å➄é✥❶ï✛❶ä✥ø➑è✧ï➊ä✥ø➀ì➈é✝ð þ➇ø➑é✥❶ï➓è✥ç✙ï➽ï✖å➈ä✧û✠✘❺ù☎å✪✘➇ê➞ì➄ë❨æ☎å✠è✧è✧ï✖ä✧é ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❺ø➑è✒ç☎å❛ê➓✡◆ï➊ï➊é ô➇é✙ì✠ó❨é è✧ç☎å➄è➽è✧ç☎ï➲ú✠å➄ä✥ø➀å✑✡✙ø➑û➀ø➑è❅✘❑å➄é✎ù❑ä✥ø✁❿ç✙é✙ï✖ê✥ê➓ì➈ë➤é✎å✠è✧ÿ☎ä✥å➈û➔ù✙å✠è❿å❼➌ ✡◆ï✢ø➑è➽ê✤æ◆ï➊ï★❿ç✏➌❹✂➈û✠✘➇æ✙ç☎ê✒➌❣ì➈ä❭ì➄è✥ç✙ï➊ä❭è❅✘➇æ✎ï❞ê❭ì➄ë➉æ☎å➄è✤è✥ï➊ä✥é☎ê✄➌❣î➓å➈ô➈ï✿ø➩è å➄û➀î➻ì❛ê✤è❨ø➑î➻æ◆ì❛ê✥ê✤ø✠✡✙û➀ï➤è✧ì➵✡✙ÿ✙ø➀û✓ù✿å➈é✫å✑✄➊ÿ✙ä❿å✠è✧ï➤ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✢ê❺✘➇ê✤è✧ï✖î ï➊é✐è✥ø➑ä✥ï➊û✠✘✫✡❩✘✫ç✎å➄é☎ù☛ð➉☞✇ì❛é☎ê✤ï★➍✐ÿ✙ï➊é✐è✧û✠✘✑➌✟î➻ì❛ê✤è➤æ☎å✠è✧è✧ï➊ä✥é➲ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é ê❺✘➇ê✤è✧ï✖î➓ê➉å➄ä✥ï✌✡✙ÿ✙ø➀û➑è♣ÿ☎ê✧ø➑é✗✂✶å➣❶ì❛î➉✡✙ø➀é☎å➄è✧ø➀ì➈é✫ì➄ë➏å➈ÿ✂è✧ì❛î➻å➄è✧ø✁✌û➀ï✖å➄ä✥é✦✝ ø➀é❼✂ è✧ï✒❿ç☎é✙ø✠➍✐ÿ✙ï❞ê✫å➄é☎ù✾ç☎å➈é☎ù☛✝r❶ä❿å✠ë➺è✧ï❞ù✲å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➓ê➊ð▼ã✛ç✙ï➷ÿ☎ê✤ÿ✎å➄û î➻ï❶è✥ç✙ì✂ù➞ì➄ë✎ä✧ï★❶ì✑✂❛é✙ø✠➽➊ø➀é❼✂➔ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û➇æ☎å✠è✧è✧ï➊ä✥é☎ê❜❶ì❛é☎ê✤ø✓ê✤è✥ê❇ø➑é❭ù✂ø➑ú➇ø✓ù☛✝ ø➀é❼✂✺è✧ç✙ï➽ê❺✘✂ê④è✥ï➊î❂ø➑é✐è✧ì✺è④ó➵ì✺î➓å➄ø➀é î➻ì✂ù✂ÿ✙û➀ï✖ê✒ê✤ç✙ì✠ó❨é ø➀é➛➞✥✂➈ÿ✙ä✥ï✫➾➈ð ã✛ç✙ï➚➞☎ä❿ê④è➤î➻ì✂ù✂ÿ✙û➀ï✑➌❀✖å➄û➀û➑ï❞ù✫è✧ç☎ï❙ëíï❞å✠è✥ÿ✙ä✧ï❭ï❯↔➇è✥ä✥å➎↔è✧ì❛ä✒➌◆è✥ä✥å➈é☎ê④ëíì❛ä✧î➓ê è✧ç☎ï♣ø➀é✙æ✙ÿ✂è✛æ☎å➄è✤è✧ï✖ä✧é✎ê➵ê✧ì✌è✥ç☎å✠è➵è✧ç✙ï✒✘➣➊å➈é➣✡✎ï➳ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù➵✡☛✘➻û➑ì✠ó✔✝ ù✂ø➀î➻ï➊é☎ê✧ø➑ì❛é☎å➄û➇ú❛ï✒↔è✥ì➈ä❿ê✝ì❛ä❯ê✧ç✙ì➈ä✧è❯ê✤è✧ä✥ø➀é❼✂❛ê❯ì➄ë◆ê❇✘➇î➉✡◆ì➈û✓ê❀è✧ç☎å➄è✔➪üå➎➶❳➊å➄é ✡◆ï❭ï❞å➈ê✧ø➑û✠✘✫î➓å✠è✴❿ç✙ï✖ù❖ì➈ät❶ì➈î➻æ☎å➈ä✧ï❞ù❢➌✝å➄é✎ù➹➪➭✡✈➶♣å➈ä✧ï❭ä✥ï➊û✓å✠è✧ø➀ú➈ï✖û✙✘✺ø➀é✦✝ ú✠å➄ä✥ø➀å➈é✐è➉ó❨ø➩è✥ç❖ä✥ï✖ê✧æ✎ï★↔è♣è✧ì➽è✥ä✥å➈é☎ê✤ëíì➈ä✥î➻å➄è✧ø➀ì➈é☎ê➉å➄é☎ù➲ù✙ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê➔ì➈ë è✧ç☎ï➽ø➑é☎æ✙ÿ✂è✒æ☎å➄è✤è✥ï➊ä✥é☎ê✌è✥ç☎å✠è❙ù✙ì✫é✙ì➄è➉❿ç☎å➈é❼✂➈ï➻è✥ç✙ï➊ø➀ä✒é✎å✠è✧ÿ☎ä✧ï❛ð✢ã✛ç✙ï ëíï✖å➄è✧ÿ✙ä✥ï➔ï✄↔✐è✥ä✥å➎↔è✥ì➈ä❹❶ì❛é✐è✥å➄ø➀é☎ê❣î➻ì✐ê④è❫ì➈ë✟è✧ç✙ï➔æ☎ä✧ø➀ì➈ä➏ô➇é✙ì✠ó❨û➀ï✖ù❼✂➈ï❨å➄é☎ù ø✓ê✇ä❿å✠è✥ç✙ï➊ä➵ê✤æ◆ï✒➊ø➟➞✈➔è✥ì➞è✧ç☎ï♣è❿å➈ê✧ô✟ð❳➏⑥è✛ø➀ê✛å➈û➀ê✧ì✌è✧ç☎ï➉ëíì✦❶ÿ✎ê✇ì➈ë❀î➻ì❛ê✤è✇ì➈ë è✧ç☎ï✒ù✂ï✖ê✧ø✠✂➈é✺ï❯➘✟ì➈ä✧è✒➌✥✡✎ï★➊å➄ÿ✎ê✤ï✌ø➑è➉ø✓ê❨ì➄ë➺è✥ï➊é✫ï➊é✐è✥ø➑ä✥ï➊û✠✘✶ç☎å➈é☎ù☛✝r❶ä❿å✠ë➺è✥ï✖ù☛ð ã✛ç✙ï➚❶û✓å➈ê✥ê✤ø✙➞☎ï➊ä★➌✙ì❛é✺è✧ç✙ï❙ì➄è✧ç☎ï➊ä➉ç✎å➄é☎ù❢➌◆ø✓ê➉ì➈ë➺è✧ï➊é✆✂❛ï➊é✙ï✖ä✥å➈û➟✝⑥æ✙ÿ✙ä✥æ◆ì❛ê✧ï å➄é✎ù è✧ä❿å➄ø➀é☎å❄✡✙û➀ï➈ðòý♣é✙ï➲ì➈ë➤è✧ç✙ï➲î➓å➈ø➑é❍æ✙ä✥ì✑✡☎û➑ï✖î➻ê✶ó❨ø➩è✥ç➘è✧ç✙ø✓ê✿å➄æ✦✝ æ✙ä✥ì❛å➎❿ç ø✓ê✶è✧ç☎å➄è✶è✧ç☎ï❖ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✻å➎✄➊ÿ✙ä✥å➎❯✘❑ø➀ê✶û➀å➈ä❺✂❛ï➊û✠✘ ù✂ï➊è✧ï✖ä❇✝ î➻ø➑é☎ï✖ù①✡☛✘❺è✧ç✙ï✿å❄✡✙ø➀û➑ø➑è❅✘ ì➄ë❨è✧ç☎ï✢ù✂ï❞ê✤ø✠✂➈é✙ï✖ä✌è✧ì➛❶ì➈î➻ï✢ÿ✙æ ó❨ø➩è✥ç❤å➄é å➄æ☎æ✙ä✧ì❛æ✙ä✥ø➀å➄è✧ï➞ê✧ï❶è➉ì➈ë❯ëíï❞å✠è✥ÿ✙ä✧ï❞ê➊ð➔ã✛ç☎ø➀ê➔è✧ÿ✙ä✥é☎ê➔ì❛ÿ✂è➉è✥ì➣✡◆ï❙å➓ù✙å➈ÿ✙é✐è❇✝ ø➀é❼✂➳è✥å➈ê✧ô➞ó❨ç✙ø✁❿ç✏➌➄ÿ☎é✂ëíì➈ä✧è✧ÿ✙é✎å✠è✧ï✖û✙✘➎➌➈î❙ÿ☎ê④è❷✡◆ï❨ä✧ï❞ù✂ì➈é☎ï✇ëíì❛ä❦ï✖å✑❿ç❭é✙ï✖ó æ✙ä✥ì✑✡✙û➀ï➊î✺ð✞✕òû✓å➄ä❘✂➈ï✒å➈î❭ì❛ÿ✙é✐è➉ì➄ë❣è✥ç✙ï❭æ☎å✠è✧è✧ï➊ä✥é➲ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✿û➀ø➑è✧ï➊ä❺✝ å✠è✥ÿ✙ä✥ï❙ø✓ê➤ù✂ï➊ú❛ì➄è✧ï❞ù✫è✥ì✿ù✂ï❞ê❺➊ä✧ø✠✡✙ø➀é❼✂✺å➄é☎ù➛❶ì➈î➻æ☎å➈ä✧ø➀é❼✂✶è✧ç✙ï➻ä✥ï➊û✓å✠è✧ø➀ú➈ï
PROC.OF THE IEEE,NOVEMBER 1998 Class scores manipulate directed graphs.This leads to the concept of trainable Graph Transformer Network (GTN)also intro- TRAINABLE CLASSIFIER MODULE duced in Section IV.Section V describes the now clas- sical method of heuristic over-segmentation for recogniz- Feature vector ing words or other character strings.Discriminative and non-discriminative gradient-based techniques for training a recognizer at the word level without requiring manual FEATURE EXTRACTION MODULE segmentation and labeling are presented in Section VI.Sec- tion VII presents the promising Space-Displacement Neu- Raw input ral Network approach that eliminates the need for seg- mentation heuristics by scanning a recognizer at all pos Fig.1.Traditional pattern recognition is performed with two mod- ules:a fixed feature extractor,and a trainable classifier. sible locations on the input.In section VIIL,it is shown that trainable Graph Transformer Networks can be for- mulated as multiple generalized transductions,based on a merits of different feature sets for particular tasks general graph composition algorithm.The connections be- Historically,the need for appropriate feature extractors tween GTNs and Hidden Markov Models,commonly used was due to the fact that the learning techniques used by in speech recognition is also treated.Section IX describes the classifiers were limited to low-dimensional spaces with a globally trained GTN system for recognizing handwrit- easily separable classes [1].A combination of three factors ing entered in a pen computer.This problem is known as have changed this vision over the last decade.First,the "on-line"handwriting recognition,since the machine must availability of low-cost machines with fast arithmetic units produce immediate feedback as the user writes.The core of allows to rely more on brute-force "numerical"methods the system is a Convolutional Neural Network.The results than on algorithmic refinements.Second,the availability clearly demonstrate the advantages of training a recognizer of large databases for problems with a large market and at the word level,rather than training it on pre-segmented, wide interest,such as handwriting recognition,has enabled hand-labeled,isolated characters.Section X describes a designers to rely more on real data and less on hand-crafted complete GTN-based system for reading handwritten and feature extraction to build recognition systems.The third machine-printed bank checks.The core of the system is the and very important factor is the availability of powerful ma- Convolutional Neural Network called LeNet-5 described in chine learning techniques that can handle high-dimensional Section II.This system is in commercial use in the NCR. inputs and can generate intricate decision functions when Corporation line of check recognition systems for the bank- fed with these large data sets.It can be argued that the ing industry.It is reading millions of checks per month in recent progress in the accuracy of speech and handwriting several banks across the United States. recognition systems can be attributed in large part to an increased reliance on learning techniques and large training A.Learning from Data data sets.As evidence to this fact,a large proportion of There are several approaches to automatic machine modern commercial OCR.systems use some form of multi- learning,but one of the most successful approaches,pop- layer Neural Network trained with back-propagation. ularized in recent years by the neural network community, In this study,we consider the tasks of handwritten char- can be called "numerical"or gradient-based learning.The acter recognition (Sections I and II)and compare the per- learning machine computes a function Yp F(ZP,W) formance of several learning techniques on a benchmark where ZP is the p-th input pattern,and W represents the data set for handwritten digit recognition (Section III). collection of adjustable parameters in the system.In a While more automatic learning is beneficial,no learning pattern recognition setting,the output Yp may be inter- technique can succeed without a minimal amount of prior preted as the recognized class label of pattern ZP,or as knowledge about the task.In the case of multi-layer neu- scores or probabilities associated with each class.A loss ral networks,a good way to incorporate knowledge is to function EP =D(DP,F(W,ZP)),measures the discrep- tailor its architecture to the task.Convolutional Neu- ancy between DP,the "correct"or desired output for pat- ral Networks [2]introduced in Section II are an exam-tern ZP,and the output produced by the system.The ple of specialized neural network architectures which in- average loss function Etrain(W)is the average of the er- corporate knowledge about the invariances of 2D shapes rors EP over a set of labeled examples called the training by using local connection patterns,and by imposing con- set {D),....(,DP)}.In the simplest setting,the straints on the weights.A comparison of several methods learning problem consists in finding the value of W that for isolated handwritten digit recognition is presented in minimizes Etrain(W).In practice,the performance of the section III.To go from the recognition of individual char- system on a training set is of little interest.The more rel- acters to the recognition of words and sentences in docu-evant measure is the error rate of the system in the field, ments,the idea of combining multiple modules trained to where it would be used in practice.This performance is reduce the overall error is introduced in Section IV.Rec- estimated by measuring the accuracy on a set of samples ognizing variable-length objects such as handwritten words disjoint from the training set,called the test set.Much using multi-module systems is best done if the modules theoretical and experimental work [3],[4],[5]has shown
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ TRAINABLE CLASSIFIER MODULE FEATURE EXTRACTION MODULE Class scores Feature vector Raw input ✁❼✿▲❍✪❦✦❬✹❦↕✮✦✸❖✱✴❉✪✿ ✵✻✿▲✷✹❈♦✱✴❴✦❃♦✱❘✵ ✵✻✰❇✸✻❈✌✸✶✰❅❆❅✷✹❍✹❈✪✿ ✵✻✿▲✷✹❈✌✿▲✺❳❃❄✰❅✸✻⑥✙✷✹✸✶❋✩✰❅❉④✽❀✿ ✵✻✯✞✵●✽❢✷✩❋✩✷✄❉★❑ ✳✪❴▲✰❅✺❺♠✥✱✄✂✆☎✒✰❅❉④⑥✙✰❺✱❘✵✻✳✪✸✻✰➜✰✝☎✄✵✶✸✶✱✴❆r✵✶✷✹✸❇❚❄✱✴❈✪❉t✱➜✵✶✸✶✱✴✿▲❈♦✱✴◗✪❴▲✰❷❆❅❴➂✱✴✺✶✺✻✿✂♦✰❅✸❺❦ î➻ï➊ä✥ø➩è❿ê❨ì➄ë❦ù✂ø✙➘◆ï✖ä✧ï✖é❛è❨ëíï❞å✠è✥ÿ✙ä✧ï✌ê✧ï❶è❿ê➵ëíì➈ä❨æ☎å➈ä✤è✥ø✠➊ÿ✙û➀å➈ä✛è✥å❛ê✤ô✂ê✖ð õ➔ø✓ê④è✥ì➈ä✥ø✠✖å➄û➀û✙✘➎➌☎è✥ç✙ï❙é☎ï➊ï✖ù✫ëíì➈ä➤å➄æ✙æ✙ä✥ì➈æ☎ä✧ø✓å✠è✥ï➤ëíï✖å✠è✥ÿ✙ä✥ï✒ï❯↔➇è✥ä✥å➎↔è✧ì❛ä✥ê ó✛å➈ê➞ù✂ÿ☎ï➓è✧ì✫è✥ç✙ï➓ë⑨å➎↔è✒è✥ç☎å✠è✒è✧ç✙ï➽û➀ï✖å➈ä✧é☎ø➑é❼✂✺è✥ï✒❿ç✙é☎ø✠➍✐ÿ✙ï❞ê➞ÿ☎ê✧ï✖ù ✡☛✘ è✧ç☎ï➉❶û✓å➈ê✥ê✤ø✙➞☎ï✖ä✥ê✛ó➵ï➊ä✥ï✌û➑ø➀î➻ø➩è✥ï✖ù✿è✥ì➓û➑ì✠ó✔✝➠ù✂ø➀î❭ï✖é☎ê✧ø➑ì❛é☎å➄û✝ê✧æ☎å✑➊ï✖ê❨ó❨ø➑è✧ç ï✖å❛ê✤ø➀û✠✘➽ê✤ï✖æ☎å➄ä❿å❄✡✙û➀ït❶û✓å➈ê✥ê✤ï❞ê✟✞✙➾✡✠⑥ð❹✕✟❶ì➈î➑✡✙ø➀é☎å✠è✥ø➑ì❛é✶ì➄ë❇è✧ç✙ä✥ï➊ï♣ë⑨å✑❶è✧ì❛ä✥ê ç☎årú❛ï➣❿ç☎å➄é❼✂❛ï✖ù è✥ç✙ø➀ê❙ú➇ø➀ê✧ø➀ì➈é ì✠ú➈ï✖ä➤è✧ç✙ï✢û➀å❛ê④è❭ù✂ï✒✖å➈ù✂ï❛ð➔✜❯ø➑ä❿ê④è★➌❯è✧ç✙ï årú✠å➄ø➀û➀å✑✡✙ø➀û➑ø➑è❅✘❙ì➈ë❀û➀ì✠ó✔✝❖➊ì❛ê✤è➏î➻å➎❿ç✙ø➀é✙ï✖ê✇ó❨ø➩è✥ç➽ë⑨å❛ê④è✛å➄ä✥ø➩è✥ç✙î➻ï❶è✧ø✁♣ÿ✙é✙ø➑è✥ê å➄û➀û➀ì✠ó➔ê➻è✧ì❑ä✧ï✖û✙✘ î➻ì➈ä✥ï✫ì➈é ✡✙ä✥ÿ✂è✥ï❯✝♠ëíì➈ä✴❶ï☞☛✤é➇ÿ✙î➻ï✖ä✧ø✁➊å➈û✍✌➷î➻ï❶è✥ç✙ì✂ù✙ê è✧ç✎å➄é ì➈é å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➻ø✠❭ä✥ï❯➞☎é✙ï✖î➻ï➊é✐è✥ê✖ð✿þ➇ï★❶ì❛é☎ù❢➌❇è✧ç✙ï➽årú✠å➄ø➀û✓å❄✡✙ø➀û➑ø➑è❅✘ ì➄ë♣û➀å➈ä❺✂❛ï✢ù✙å➄è✥å❄✡✎å➈ê✧ï✖ê✌ëíì❛ä➻æ✙ä✧ì➎✡✙û➀ï➊î➓ê✒ó❨ø➑è✧ç➘å❺û➀å➈ä❺✂❛ï➽î➓å➄ä✥ô➈ï❶è➻å➄é☎ù ó❨ø✓ù✂ï❨ø➑é✐è✥ï➊ä✥ï✖ê✤è✒➌➈ê✧ÿ✗❿ç➻å➈ê❦ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂➳ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❀➌➄ç☎å➈ê❯ï✖é☎å❄✡✙û➀ï✖ù ù✂ï❞ê✤ø✠✂➈é✙ï✖ä✥ê✝è✧ì➳ä✧ï✖û✙✘➳î➻ì➈ä✥ï❫ì❛é➞ä✧ï❞å➄û➇ù✙å✠è❿å➉å➈é☎ù✌û➀ï✖ê✥ê❀ì❛é➞ç☎å➄é✎ù☛✝❖➊ä✥å➄ë➺è✧ï❞ù ëíï✖å➄è✧ÿ✙ä✥ï✌ï❯↔➇è✥ä✥å➎↔è✧ø➀ì➈é✢è✧ì➵✡☎ÿ✙ø➑û✓ù✺ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ê❺✘➇ê✤è✧ï✖î➓ê➊ð❫ã✛ç☎ï➤è✧ç✙ø➀ä✥ù å➄é✎ù➤ú➈ï➊ä❘✘➔ø➀î➻æ✎ì❛ä✤è❿å➄é✐è☛ë⑨å✑❶è✧ì➈ä❀ø➀ê✟è✥ç✙ï❫årú✠å➄ø➀û✓å❄✡✙ø➀û➑ø➑è❅✘➔ì➈ë➇æ◆ì✠ó➵ï➊ä✧ëíÿ✙û➄î➓å♦✝ ❿ç✙ø➀é✙ï➵û➑ï❞å➄ä✥é✙ø➑é✗✂➔è✧ï★❿ç✙é✙ø✁➍✐ÿ✙ï✖ê❇è✧ç✎å✠è❷➊å➄é✒ç☎å➄é☎ù✙û➑ï➵ç✙ø✠✂➈ç✦✝➠ù✂ø➀î❭ï✖é☎ê✧ø➑ì❛é☎å➄û ø➀é✙æ✙ÿ✂è❿ê➳å➈é☎ù➔✖å➄é↕✂❛ï➊é✙ï✖ä✥å➄è✧ï❙ø➑é✐è✧ä✥ø✁➊å✠è✥ï❙ù✂ï★❶ø✓ê✤ø➀ì➈é➲ëíÿ✙é✗↔è✥ø➑ì❛é☎ê➳ó❨ç✙ï➊é ëíï✖ù ó❨ø➑è✧ç è✧ç☎ï✖ê✧ï➓û➀å➈ä❺✂❛ï❭ù✙å➄è✥å✺ê✧ï❶è✥ê✖ð➵➏⑥è➉➊å➈é➛✡◆ï✶å➈ä❺✂❛ÿ✙ï✖ù➲è✧ç☎å➄è✌è✧ç✙ï ä✥ï✒❶ï✖é✐è✛æ☎ä✧ì➎✂➈ä✥ï✖ê✥ê❫ø➀é✿è✧ç✙ï➞å➎✄❶ÿ☎ä✥å➎❯✘➓ì➄ë❣ê✧æ✎ï✖ï✒❿ç✺å➄é☎ù✿ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂ ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é ê❇✘✂ê✤è✧ï✖î➻ê✌✖å➄é➒✡✎ï➽å➄è✤è✥ä✧ø✠✡✙ÿ✂è✥ï✖ù ø➑é➷û✓å➄ä❘✂➈ï❭æ☎å➄ä✧è➤è✧ì➲å➈é ø➀é✗❶ä✥ï✖å❛ê✤ï❞ù✌ä✧ï✖û➑ø✓å➄é✥❶ï➵ì➈é✒û➀ï✖å➈ä✧é☎ø➑é❼✂➉è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❣å➄é✎ù➞û✓å➄ä❘✂➈ï➏è✥ä✥å➈ø➑é☎ø➑é❼✂ ù✙å➄è✥å✫ê✧ï❶è✥ê✖ð➙✕♣ê✌ï➊ú➇ø✓ù✂ï➊é✥❶ï➻è✧ì✺è✥ç✙ø✓ê✌ë⑨å✑❶è✒➌❦å✿û➀å➈ä❺✂❛ï❭æ☎ä✧ì❛æ✎ì❛ä✤è✥ø➑ì❛é❺ì➈ë î➻ì✂ù✂ï➊ä✥é➙➊ì➈î➻î➻ï➊ä✴❶ø✓å➄û✝ý✌☞✎✍✲ê❺✘➇ê✤è✧ï✖î➓ê✇ÿ✎ê✤ï➤ê✤ì❛î➻ï➉ëíì❛ä✧î ì➈ë❀î✒ÿ✙û➑è✧ø✙✝ û✓å✪✘➈ï➊ä❨ñ➉ï➊ÿ✙ä❿å➄û✝ñ➉ï❶è④ó➵ì➈ä✥ô➻è✧ä❿å➄ø➀é✙ï✖ù✿ó❨ø➑è✧ç✫✡☎å✑❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é✝ð ➏➠é➓è✧ç✙ø✓ê➵ê④è✥ÿ☎ù✦✘✑➌✐ó➵ï✬❶ì❛é☎ê✤ø✓ù✂ï✖ä❣è✧ç✙ï➉è✥å❛ê✤ô✂ê➏ì➈ë☛ç☎å➄é☎ù✙ó❨ä✧ø➑è✤è✥ï➊é ❿ç☎å➈ä❇✝ å✑❶è✧ï✖ä❨ä✧ï★❶ì✑✂❛é✙ø➑è✧ø➀ì➈é➔➪♠þ➇ï✒❶è✧ø➀ì➈é☎ê✩➏➵å➄é☎ù➙➏❇➏❇➶✛å➈é☎ù➐❶ì➈î➻æ☎å➈ä✧ï♣è✧ç☎ï✌æ✎ï✖ä❇✝ ëíì➈ä✥î➓å➄é✗➊ï✫ì➄ë➞ê✧ï➊ú➈ï✖ä✥å➈û❨û➑ï❞å➄ä✥é✙ø➀é❼✂➷è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✖ê➽ì❛é❍å①✡✎ï✖é✗❿ç✙î➓å➄ä✥ô ù✙å➄è✥å➘ê✤ï➊è✺ëíì➈ä✺ç☎å➄é✎ù✂ó❨ä✧ø➑è✤è✥ï➊é ù✂ø✠✂➈ø➑è✺ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❫➪♠þ➇ï✒❶è✧ø➀ì➈é✖➏❇➏❇➏❇➶❶ð ✎ç✙ø➀û➑ï✫î➻ì❛ä✧ï✫å➄ÿ✙è✧ì➈î➓å➄è✧ø✁✢û➀ï✖å➈ä✧é☎ø➑é❼✂➷ø✓ê➵✡◆ï➊é✙ï✄➞✥❶ø✓å➄û✶➌✇é☎ì û➀ï✖å➈ä✧é☎ø➑é❼✂ è✧ï★❿ç✙é✙ø✁➍✐ÿ✙ï➉➊å➈é➲ê✤ÿ✥✄❶ï✖ï✖ù✺ó❨ø➑è✧ç✙ì❛ÿ✂è➳å➽î❭ø➀é✙ø➀î➓å➄û❇å➈î➻ì➈ÿ✙é✐è➉ì➈ë❦æ☎ä✧ø➀ì➈ä ô➇é✙ì✠ó❨û➀ï✖ù✦✂❛ï✒å❄✡◆ì➈ÿ✂è♣è✧ç✙ï✒è✥å❛ê✤ô✟ð✬➏➠é➲è✧ç✙ï➝➊å❛ê✤ï➞ì➈ë➏î✒ÿ✙û➑è✧ø✙✝⑥û➀å✪✘❛ï➊ä➔é✙ï✖ÿ✦✝ ä❿å➄û➵é✙ï❶è④ó➵ì➈ä✥ô✂ê✄➌❣å↕✂❛ì➇ì➇ù ó➵å✪✘ è✧ì❺ø➑é✥❶ì➈ä✥æ◆ì➈ä❿å✠è✧ï✶ô✐é☎ì✠ó❨û➑ï❞ù✦✂➈ï✢ø➀ê✒è✧ì è✥å➈ø➑û➀ì➈ä❺ø➩è❿ê❺å➈ä❘❿ç☎ø➩è✥ï✒↔è✥ÿ✙ä✥ï è✧ì✲è✧ç✙ï❤è✥å➈ê✧ô✟ð ☞✇ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✒ñ➔ï✖ÿ✦✝ ä❿å➄û✒ñ➉ï❶è④ó➵ì➈ä✥ô➇ê✏✞✑✆✠❙ø➑é✐è✧ä✥ì✂ù✂ÿ✗➊ï✖ù❲ø➑éòþ➇ï★↔è✥ø➑ì❛é ➏❇➏✫å➄ä✥ï å➄é ï❯↔✙å➄î➚✝ æ✙û➀ï➲ì➄ë✌ê✧æ✎ï★❶ø✓å➄û➀ø✙➽✖ï✖ù é✙ï➊ÿ☎ä✥å➈û❨é✙ï❶è④ó➵ì➈ä✥ô❤å➄ä✴❿ç✙ø➑è✧ï★↔è✧ÿ☎ä✧ï❞ê➻ó❨ç✙ø✠❿ç➘ø➀é✦✝ ❶ì❛ä✧æ◆ì➈ä❿å✠è✥ï✢ô➇é✙ì✠ó❨û➀ï✖ù❼✂➈ï✺å❄✡◆ì➈ÿ✂è➻è✧ç☎ï✫ø➑é➇ú✠å➄ä✥ø➀å➈é✗❶ï❞ê❭ì➄ë✒✑✑✧ ê✧ç☎å➄æ◆ï✖ê ✡☛✘➲ÿ☎ê✧ø➑é✗✂✢û➀ì✦➊å➄û❹➊ì➈é✙é☎ï✒↔è✥ø➑ì❛é❖æ☎å➄è✤è✥ï➊ä✥é☎ê✄➌❇å➄é☎ù➛✡☛✘✫ø➑î➻æ◆ì❛ê✧ø➑é❼✂✆➊ì➈é✦✝ ê✤è✧ä❿å➄ø➀é❛è❿ê➔ì❛é✫è✧ç✙ï❭ó➵ï➊ø✠✂➈ç✐è✥ê✖ð✬✕➧➊ì➈î➻æ☎å➈ä✧ø✓ê✤ì❛é✫ì➄ë❫ê✧ï➊ú❛ï➊ä❿å➄û✝î➻ï❶è✥ç✙ì✂ù✙ê ëíì➈ä➻ø✓ê✤ì❛û➀å➄è✧ï❞ù ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï➊é➘ù✂ø✠✂➈ø➑è❭ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❤ø✓ê❭æ✙ä✧ï❞ê✤ï✖é✐è✧ï✖ù❤ø➑é ê✧ï✒↔è✥ø➑ì❛é✆➏❇➏❺➏↔ð✛ã❇ì➵✂➈ì➻ëíä✥ì➈î✴è✧ç✙ï❙ä✧ï★❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✿ì➄ë❣ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û❳❿ç☎å➈ä❇✝ å✑❶è✧ï✖ä✥ê➳è✥ì✢è✥ç✙ï➓ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❺ì➄ë➵ó✇ì❛ä✥ù✙ê✌å➈é☎ù ê✤ï✖é❛è✥ï➊é✗➊ï✖ê➤ø➀é ù✂ì✦❶ÿ✦✝ î➻ï➊é✐è✥ê✒➌✟è✧ç☎ï❙ø✓ù✂ï✖å✶ì➄ë➜➊ì➈î➉✡☎ø➑é✙ø➀é❼✂✢î✒ÿ✙û➑è✧ø➀æ✙û➀ï❭î➻ì✂ù✂ÿ✙û➀ï✖ê➉è✧ä❿å➄ø➀é✙ï❞ù✺è✧ì ä✥ï✖ù✂ÿ✗➊ï✒è✥ç✙ï❭ì✠ú➈ï➊ä❿å➄û➀û❀ï✖ä✧ä✥ì➈ä♣ø➀ê➳ø➀é✐è✧ä✥ì➇ù✙ÿ✗❶ï❞ù✫ø➑é➷þ➇ï✒❶è✧ø➀ì➈é➛➏r✣✒ð➉✍❨ï★✹✝ ì✑✂❛é✙ø✠➽➊ø➀é❼✂♣ú✠å➈ä✧ø✓å❄✡✙û➀ï❯✝⑥û➀ï➊é❼✂➈è✧ç➞ì➎✡✔✓④ï✒❶è✥ê❣ê✧ÿ✗❿ç❙å❛ê❦ç☎å➄é☎ù✙ó❨ä✧ø➑è✤è✥ï➊é❭ó✇ì❛ä✥ù✙ê ÿ☎ê✧ø➑é✗✂❤î✒ÿ☎û➩è✥ø➟✝⑥î➻ì➇ù✙ÿ✙û➑ï ê❇✘✂ê✤è✧ï✖î➻ê✢ø✓ê➣✡◆ï✖ê✤è✺ù✂ì❛é✙ï➲ø➑ë➞è✧ç☎ï❖î➻ì✂ù✂ÿ✙û➀ï✖ê î➓å➄é✙ø➀æ✙ÿ✙û✓å✠è✥ï➓ù✂ø➑ä✥ï✒❶è✧ï❞ù➔✂❛ä✥å➈æ✙ç☎ê✖ð❭ã✛ç✙ø✓ê➤û➀ï✖å➈ù☎ê➳è✧ì✿è✧ç✙ï➣➊ì➈é✗➊ï➊æ✂è✌ì➈ë è✧ä❿å➄ø➀é☎å✑✡✙û➑ï⑨➠✔➡❺➢❘➤✗➥➧➦❼➡❺➢♦➨✗➩➭➫❯➯♦➡✹➲➵➳❯➡ ➸➑➳❯➺✶➻➜➯♦➡❺➼ ➪♠â➤ã❨ñ④➶➻å➈û➀ê✧ì❺ø➀é✐è✧ä✥ì❄✝ ù✂ÿ✗➊ï✖ù➘ø➀é✲þ➇ï✒❶è✧ø➀ì➈é ➏r✣✒ðòþ➇ï★↔è✥ø➑ì❛é⑨✣❂ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ê➓è✧ç✙ï➲é☎ì✠ó ➊û➀å❛ê❅✝ ê✧ø✠✖å➄û❨î➻ï❶è✥ç✙ì✂ù❑ì➈ë➳ç☎ï➊ÿ✙ä✥ø➀ê✤è✧ø✁✿ì✠ú➈ï✖ä❇✝➠ê✤ï✒✂➈î➻ï➊é✐è❿å✠è✧ø➀ì➈é ëíì➈ä➓ä✥ï✒❶ì➎✂➈é✙ø✠➽❯✝ ø➀é❼✂❖ó➵ì➈ä❿ù✙ê➞ì❛ä✒ì➈è✧ç✙ï✖ä➑❿ç☎å➈ä✥å➎↔è✧ï✖ä✒ê✤è✧ä✥ø➑é❼✂✐ê➊ð➒✧➉ø✓ê❘❶ä✥ø➑î➻ø➀é☎å✠è✥ø➑ú❛ï✶å➄é☎ù é✙ì❛é✦✝⑥ù✙ø➀ê❘❶ä✥ø➑î➻ø➀é☎å✠è✥ø➑ú❛ï➙✂❛ä✥å❛ù✂ø➀ï➊é✐è❇✝❖✡☎å➈ê✧ï✖ù è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❭ëíì➈ä➻è✥ä✥å➈ø➑é☎ø➑é❼✂ å➷ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï✖ä➓å✠è➓è✧ç☎ï➲ó✇ì❛ä✥ù❑û➀ï➊ú➈ï✖û❨ó❨ø➩è✥ç✙ì➈ÿ✙è✶ä✧ï★➍✐ÿ✙ø➑ä✥ø➀é❼✂➷î➻å➈é➇ÿ☎å➄û ê✧ï✄✂➈î➻ï✖é❛è❿å✠è✥ø➑ì❛é➤å➈é☎ù➤û✓å❄✡◆ï➊û➀ø➑é❼✂➔å➈ä✧ï❣æ☎ä✧ï❞ê✤ï✖é❛è✥ï✖ù➤ø➀é✒þ➇ï✒❶è✧ø➀ì➈é✌✣④➏↔ð❯þ➇ï★✹✝ è✧ø➀ì➈é➛✣✬➏❺➏➉æ✙ä✥ï✖ê✧ï➊é✐è✥ê➉è✧ç☎ï❙æ✙ä✥ì➈î➻ø✓ê✤ø➀é❼✂✺þ➇æ☎å✑➊ï❯✝r✧➉ø✓ê✧æ✙û➀å➎❶ï✖î❭ï✖é✐è➳ñ➔ï✖ÿ✦✝ ä❿å➄û➞ñ➔ï➊è④ó✇ì❛ä✧ô✾å➄æ✙æ✙ä✥ì❛å➎❿ç✻è✥ç☎å✠è➲ï✖û➑ø➀î➻ø➑é☎å➄è✧ï❞ê✺è✧ç✙ï❤é✙ï➊ï❞ù✲ëíì❛ä➲ê✤ï✒✂❄✝ î➻ï➊é✐è✥å➄è✧ø➀ì➈é ç☎ï➊ÿ✙ä✥ø➀ê✤è✧ø✁➊ê➑✡☛✘ ê❺✖å➄é✙é✙ø➀é❼✂ å➲ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï✖ä✒å✠è➻å➄û➀û✛æ✎ì✐ê❅✝ ê✧ø✙✡✙û➀ï➽û➀ì✦➊å✠è✥ø➑ì❛é☎ê➞ì❛é➷è✧ç✙ï✢ø➑é☎æ✙ÿ✂è✖ð➔➏➠é❤ê✤ï★↔è✥ø➑ì❛é①✣✬➏❇➏❺➏✹➌❣ø➩è❭ø➀ê❙ê✤ç☎ì✠ó❨é è✧ç✎å✠è✿è✧ä❿å➄ø➀é☎å❄✡✙û➀ï â➳ä✥å➈æ✙ç✲ã❇ä✥å➈é☎ê④ëíì❛ä✧î➻ï✖ä✢ñ➔ï➊è④ó✇ì❛ä✧ô✂ê➐➊å➈é ✡✎ï ëíì❛ä❇✝ î✒ÿ☎û➀å➄è✧ï✖ù✫å❛ê❨î✒ÿ☎û➩è✥ø➑æ✙û➀ï➓✂❛ï➊é✙ï✖ä✥å➈û➑ø✠➽➊ï❞ù➽è✥ä✥å➈é☎ê✧ù✙ÿ✗↔è✥ø➑ì❛é☎ê✄➌✗✡☎å➈ê✧ï✖ù✿ì➈é✫å ✂➈ï✖é✙ï➊ä❿å➄û✗✂➈ä❿å➄æ✙ç➣➊ì➈î➻æ✎ì✐ê✤ø➑è✧ø➀ì➈é➽å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î✺ð❯ã✛ç☎ï④❶ì❛é✙é✙ï★↔è✧ø➀ì➈é✎ê❹✡◆ï❯✝ è④ó➵ï➊ï➊é➲â➤ã❨ñ♣ê➉å➄é✎ù✫õ➔ø✓ù✙ù✂ï➊é➲ö❖å➄ä✥ô➈ì✠ú➽ö✫ì✂ù✂ï✖û➀ê✒➌✥➊ì➈î➻î➻ì➈é✙û✠✘✶ÿ☎ê✧ï✖ù ø➀é❖ê✤æ◆ï➊ï★❿ç✫ä✧ï★❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✺ø✓ê➉å➄û✓ê✧ì➓è✧ä✥ï✖å➄è✧ï✖ù✝ð➉þ➇ï✒❶è✧ø➀ì➈é✆➏✝✕➶ù✙ï✖ê❘❶ä✥ø✙✡◆ï✖ê å✫✂➈û➀ì✑✡✎å➄û➀û✙✘✫è✥ä✥å➈ø➑é☎ï✖ù â➤ã❨ñ ê❇✘✂ê✤è✧ï✖î ëíì➈ä✒ä✧ï★❶ì✑✂❛é✙ø✠➽➊ø➀é❼✂✺ç☎å➄é☎ù✙ó❨ä✧ø➑è❇✝ ø➀é❼✂➽ï➊é✐è✥ï➊ä✥ï✖ù✺ø➀é❖å➓æ◆ï➊é➔➊ì➈î➻æ✙ÿ✂è✥ï➊ä❞ð➔ã✛ç✙ø➀ê➉æ✙ä✥ì✑✡✙û➀ï➊î▼ø➀ê➉ô➇é✙ì✠ó❨é➲å➈ê ☛✤ì❛é✦✝⑥û➑ø➀é✙ï✖✌➞ç☎å➄é☎ù✙ó❨ä✧ø➑è✧ø➀é❼✂❭ä✧ï★❶ì✑✂❛é✙ø➑è✧ø➀ì➈é✏➌➇ê✤ø➀é✗➊ï➔è✧ç☎ï➳î➓å➎❿ç✙ø➑é☎ï➉î✒ÿ✎ê④è æ✙ä✥ì✂ù✂ÿ✗❶ï✇ø➑î➻î➻ï✖ù✙ø➀å➄è✧ï✇ëíï➊ï✖ù❼✡☎å✑❿ô✌å➈ê✝è✧ç✙ï➵ÿ☎ê✧ï➊ä❯ó❨ä✧ø➑è✧ï❞ê➊ð❯ã✛ç✙ï✔❶ì➈ä✥ï➏ì➈ë è✧ç☎ï➉ê❇✘✂ê✤è✧ï✖îòø➀ê✇å➓☞✇ì➈é➇ú➈ì❛û➑ÿ✂è✥ø➑ì❛é☎å➄û✙ñ➉ï➊ÿ✙ä❿å➄û✙ñ➉ï❶è④ó➵ì➈ä✥ô✟ð❇ã✛ç✙ï➔ä✥ï✖ê✧ÿ✙û➑è✥ê ❶û➀ï✖å➈ä✧û✠✘➞ù✂ï✖î❭ì❛é☎ê✤è✧ä❿å✠è✧ï✇è✧ç☎ï➔å➈ù✂ú✠å➄é✐è❿å❄✂➈ï❞ê❀ì➈ë☎è✥ä✥å➈ø➑é✙ø➀é❼✂➤å➳ä✧ï★❶ì✑✂❛é✙ø✠➽➊ï➊ä å✠è❦è✥ç✙ï❨ó✇ì❛ä✥ù✒û➑ï✖ú➈ï✖û✻➌➄ä❿å✠è✥ç✙ï➊ä❯è✥ç☎å➄é❙è✧ä❿å➄ø➀é✙ø➀é❼✂➤ø➩è❣ì➈é❭æ✙ä✥ï❯✝➠ê✤ï✒✂➈î➻ï➊é✐è✥ï✖ù❢➌ ç☎å➈é☎ù☛✝⑥û➀å✑✡✎ï✖û➑ï❞ù❢➌✛ø➀ê✧ì➈û✓å✠è✥ï✖ù ❿ç☎å➄ä❿å✑❶è✧ï✖ä✥ê✖ð✲þ➇ï✒❶è✧ø➀ì➈é☞✕❋ù✙ï✖ê❘❶ä✥ø✙✡◆ï✖ê➓å ❶ì❛î➻æ✙û➑ï➊è✧ï➻â➤ã❨ñ✩✝❖✡☎å➈ê✧ï✖ù✫ê❺✘✂ê④è✥ï➊î▼ëíì➈ä♣ä✧ï❞å➈ù✂ø➀é❼✂➽ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï✖é❖å➄é☎ù î➓å✑❿ç✙ø➀é✙ï✄✝♠æ✙ä✥ø➀é❛è✥ï✖ù✌✡✎å➄é✙ô➓❿ç✙ï★❿ô✂ê➊ð❯ã✛ç✙ï✎➊ì➈ä✥ï✇ì➈ë✙è✧ç☎ï➵ê❺✘✂ê④è✥ï➊îòø✓ê❇è✧ç✙ï ☞✇ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û✎ñ➉ï➊ÿ✙ä❿å➄û✟ñ➔ï➊è④ó✇ì❛ä✧ô➵➊å➈û➑û➀ï✖ù✘✗✝ï✖ñ➉ï❶è❺✝✝✙✒ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù✶ø➑é þ➇ï★↔è✧ø➀ì➈é➒➏❇➏↔ð❭ã✛ç✙ø✓ê✌ê❇✘✂ê✤è✧ï➊î❋ø➀ê➳ø➀é➒➊ì➈î➻î➻ï➊ä✴❶ø✓å➄û❦ÿ☎ê✧ï❙ø➀é è✧ç✙ï➓ñ✞☞✎✍ ☞✇ì➈ä✥æ◆ì➈ä❿å✠è✧ø➀ì➈é✒û➑ø➀é✙ï❨ì➄ë✇❿ç✙ï✒❿ô✌ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é❙ê❺✘✂ê④è✥ï➊î➓ê❯ëíì➈ä❯è✧ç☎ï✩✡☎å➄é✙ô❩✝ ø➀é❼✂➽ø➀é☎ù✂ÿ☎ê✤è✧ä❘✘➈ð✎➏⑥è➉ø✓ê➔ä✥ï✖å➈ù✙ø➑é❼✂➻î➻ø➀û➑û➀ø➑ì❛é☎ê➔ì➈ë❹❿ç✙ï✒❿ô✂ê✛æ◆ï➊ä➉î➻ì❛é❛è✥ç✫ø➑é ê✧ï➊ú➈ï✖ä✥å➈û✇✡☎å➄é✙ô✂ê❨å➎❶ä✥ì❛ê✥ê➏è✥ç✙ï➓→➔é✙ø➑è✧ï❞ù➲þ✐è✥å➄è✧ï❞ê➊ð ✚✜✛✒✢➳❘➢❄➡✹➨✤✣●➨✦✥✬➫✴➡❺➯❄➲★✧➚➢♦➺r➢ ã✛ç✙ï➊ä✥ï✹å➄ä✥ïòê✧ï➊ú➈ï✖ä✥å➈û❖å➄æ☎æ✙ä✧ì✐å✑❿ç✙ï❞ê è✧ì✴å➈ÿ✂è✧ì❛î➓å✠è✧ø✁ î➓å✑❿ç✙ø➀é✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂✗➌❢✡☎ÿ✂è✌ì➈é✙ï❭ì➄ë✇è✧ç✙ï➻î➻ì❛ê✤è➤ê✧ÿ✗✒❶ï✖ê✥ê✤ëíÿ✙û➏å➄æ✙æ☎ä✧ì✐å✑❿ç✙ï❞ê✄➌◆æ✎ì❛æ✦✝ ÿ✙û✓å➄ä✥ø✙➽✖ï✖ù✶ø➑é✿ä✧ï★❶ï✖é❛è✩✘❛ï✖å➄ä❿ê➃✡❩✘➻è✥ç✙ï✌é✙ï✖ÿ✙ä✥å➈û✟é✙ï❶è④ó➵ì➈ä✥ô➵❶ì➈î➻î❙ÿ✙é✙ø➑è❅✘✑➌ ➊å➈é✆✡✎ï➚➊å➈û➑û➀ï✖ù✩☛✧é✐ÿ☎î❭ï✖ä✧ø✁➊å➈û✍✌➻ì❛ä✟✥✑➡❺➢✫✪✬✣ ➳❯➨✥➺✮✭✰✯✴➢✪➩✄➳✱✪✳✲✙➳✴➢♦➡✹➨✴✣●➨✵✥➈ð➉ã✛ç✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂➘î➓å✑❿ç✙ø➀é✙ï✢❶ì❛î➻æ✙ÿ✂è✧ï❞ê❖å ëíÿ✙é✥↔è✧ø➀ì➈é✷✶✹✸✻✺✽✼➵➪✿✾❀✸✔❁❃❂➶ ó❨ç✙ï✖ä✧ï❄✾❅✸❭ø✓ê❨è✥ç✙ï❇❆✗✝üè✥ç✫ø➑é☎æ✙ÿ✂è♣æ✎å✠è✤è✥ï➊ä✥é✏➌✟å➄é☎ù❈❂ ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥ê❨è✧ç✙ï ❶ì❛û➑û➀ï✒❶è✧ø➀ì➈é✻ì➈ë✒å➈ù✆✓④ÿ✎ê④è❿å❄✡✙û➀ï❖æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê✶ø➑é✻è✧ç✙ï➷ê❇✘✂ê✤è✧ï✖î✿ð ➏➠é✲å æ☎å➄è✤è✧ï✖ä✧é ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é ê✤ï➊è✤è✧ø➀é❼✂✥➌❇è✥ç✙ï✶ì❛ÿ✂è✧æ☎ÿ✂è❄✶✹✸✿î➓å✪✘➛✡◆ï✶ø➀é✐è✧ï✖ä❇✝ æ✙ä✥ï❶è✥ï✖ù❤å❛ê❙è✧ç✙ï✺ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï❞ù✢❶û✓å➈ê✥ê❙û➀å✑✡✎ï✖û❨ì➄ë➉æ☎å➄è✤è✥ï➊ä✥é✩✾✸ ➌✇ì➈ä➓å➈ê ê❘❶ì➈ä✥ï✖ê✒ì➈ä❙æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➩è✥ø➑ï❞ê✒å❛ê✧ê✧ì✦❶ø✓å✠è✧ï❞ù➷ó❨ø➩è✥ç❤ï❞å✑❿ç✢❶û✓å➈ê✥ê✖ð➛✕Pû➑ì✐ê✧ê ëíÿ✙é✗❶è✧ø➀ì➈é✷❉❊✸❋✺❍●➙➪❏■❑✸✔❁❃✼➵➪✮❂▲❁▼✾❀✸✑➶❇➶❯➌➞î➻ï✖å❛ê✤ÿ✙ä✥ï✖ê✿è✥ç✙ï❤ù✂ø✓ê❘❶ä✥ï➊æ✦✝ å➄é✥❯✘➣✡◆ï❶è④ó➵ï➊ï✖é◆■❑✸✦➌✙è✧ç✙ï❖☛❘❶ì➈ä✥ä✥ï✒↔è✱✌➞ì❛ä➔ù✂ï✖ê✧ø➑ä✥ï✖ù✿ì❛ÿ✂è✧æ✙ÿ✙è❨ëíì➈ä❨æ☎å➄è❇✝ è✧ï✖ä✧éP✾❀✸✦➌➉å➄é✎ù➘è✧ç✙ï❺ì➈ÿ✂è✥æ✙ÿ✂è✿æ✙ä✧ì✂ù✂ÿ✥❶ï✖ù ✡❩✘❑è✧ç☎ï ê❺✘✂ê④è✥ï➊î✺ð✹ã✛ç✙ï årú➈ï✖ä✥å✑✂➈ï➻û➀ì❛ê✥ê➞ëíÿ✙é✥↔è✧ø➀ì➈é◗❉❙❘❯❚❃❱✡❲❨❳✏➪❩❂➶➞ø✓ê✒è✧ç✙ï✿årú➈ï➊ä❿å❄✂❛ï➻ì➄ë❨è✧ç☎ï✢ï✖ä❇✝ ä✥ì➈ä❿ê✟❉❊✸➽ì✠ú❛ï➊ä➤å✢ê✧ï❶è➤ì➄ë❫û✓å❄✡◆ï➊û➀ï✖ù❺ï❯↔✙å➈î❭æ☎û➑ï❞ê④✖å➄û➀û➑ï❞ù✫è✧ç☎ï❙è✥ä✥å➈ø➑é☎ø➑é❼✂ ê✧ï❶è◆❬❩➪✿✾ ✜ ❁❃■✜ ➶✡❁❪❭✍❭❨❭✍❭✁➪❩✾❇❫❙❁❴■❵❫❹➶▼❛❛ð⑧➏➠é❤è✥ç✙ï✫ê✤ø➀î➻æ✙û➑ï❞ê④è➽ê✧ï❶è✧è✧ø➀é❼✂✗➌❫è✧ç✙ï û➀ï✖å➄ä✥é✙ø➀é❼✂➲æ✙ä✥ì✑✡☎û➑ï✖î✤➊ì➈é☎ê✧ø✓ê④è❿ê✒ø➀é①➞☎é☎ù✙ø➑é❼✂❺è✧ç☎ï✢ú✠å➄û➀ÿ✙ï✢ì➄ë✟❂ è✥ç☎å✠è î➻ø➑é☎ø➑î➻ø✠➽➊ï✖ê❊❉❙❘❯❚✱❱▼❲✍❳✏➪✮❂➶❶ð✛➏➠é✫æ✙ä❿å✑↔è✥ø✠➊ï✑➌✎è✧ç✙ï❙æ✎ï✖ä✤ëíì❛ä✧î➓å➈é✗❶ï✌ì➈ë❣è✧ç✙ï ê❺✘➇ê✤è✧ï✖î✴ì❛é✿å❭è✥ä✥å➈ø➑é✙ø➀é❼✂➓ê✧ï❶è➔ø✓ê❨ì➄ë❦û➑ø➑è✤è✥û➑ï✌ø➀é✐è✧ï✖ä✧ï❞ê④è❞ð❫ã✛ç✙ï✌î➻ì➈ä✥ï➤ä✧ï✖û➟✝ ï➊ú✠å➈é❛è➤î➻ï✖å❛ê✤ÿ✙ä✥ï✒ø✓ê♣è✧ç✙ï➻ï✖ä✧ä✥ì➈ä♣ä✥å➄è✧ï✒ì➈ë➏è✧ç☎ï➻ê❺✘✂ê④è✥ï➊î ø➀é➲è✥ç✙ï➚➞☎ï➊û✓ù❢➌ ó❨ç✙ï✖ä✧ï➽ø➑è➻ó✇ì❛ÿ✙û➀ù ✡◆ï✿ÿ☎ê✤ï❞ù ø➀é æ✙ä❿å✑↔è✥ø✠➊ï➈ð❖ã✛ç☎ø➀ê❙æ✎ï✖ä✤ëíì❛ä✧î➓å➈é✗❶ï➽ø✓ê ï✖ê✤è✧ø➀î➓å✠è✥ï✖ù➒✡☛✘➲î➻ï✖å➈ê✧ÿ✙ä✥ø➑é✗✂✶è✥ç✙ï➽å✑✒❶ÿ✙ä❿å✑✄✘✫ì➈é➷å✿ê✧ï❶è✌ì➈ë✛ê✥å➄î➻æ✙û➀ï✖ê ù✂ø✓ê❯✓④ì❛ø➑é✐è➻ëíä✥ì➈î è✥ç✙ï✿è✧ä❿å➄ø➀é✙ø➑é✗✂➷ê✧ï❶è★➌➃✖å➄û➀û➑ï❞ù è✥ç✙ï✿è✧ï✖ê✤è✶ê✧ï❶è✖ð➘ö✫ÿ✗❿ç è✧ç☎ï➊ì➈ä✥ï❶è✥ø✠✖å➄û❨å➄é✎ù ï❯↔✂æ◆ï➊ä✥ø➑î➻ï✖é❛è❿å➄û❨ó➵ì➈ä✥ô☞✞❜✬✠✶➌❊✞❝✫✠✶➌❇✞✙✆✠➔ç✎å➈ê➻ê✤ç☎ì✠ó❨é
PROC.OF THE IEEE,NOVEMBER 1998 that the gap between the expected error rate on the test Hessian matrix as in Newton or Quasi-Newton methods. set d trt and the error rate on the training set dt,Ano de- The ConRigate Gradient method [can also be used. creases with the number of training samples approximately However,Appendix B shows that despite many claims as to the contrary in the literature,the usefulness of these dtt-d,A[(]“”)W (1) second-order methods to large learning machines is very limited. where"is the number of training samples,is a measure of A popular minimization procedure is the stochastic gra- “effective capacity'”or complexity of the machine[j],[☑，a dient algorithm,also called the on-line update.It consists is a number between 0.5 and 1.0,and is a constant.This in updating the parameter vector using a noisy,or approx- gap always decreases when the number of training samples increases.Furthermore,as the capacity increases,dt,Ao imated,version of the average gradient.In the most com- mon instance of it,W is updated on the basis of a single decreases.Therefore,when increasing the capacity],there sample: is a trade-off between the decrease of dt,As and the in- crease of the gap,with an optimal value of the capacity W视sW-1-e arRW) ∂W (3) that achieves the lowest generalization error dtt.Most learning algorithms attempt to minimize dt,Ao as well as With this procedure the parameter vector Zuctuates some estimate of the gap.A formal version of this is called around an average tralectory,but usually converges consid- structural risk minimization [j],[7],and is based on defin- erably faster than regular gradient descent and second or- ing a sequence of learning machines of increasing capacity, der methods on large training sets with redundant samples corresponding to a sequence of subsets of the parameter (such as those encountered in speech or character recogni- space such that each subset is a superset of the previous tion).The reasons for this are explained in Appendix B subset.In practical terms,Structural Risk Minimization The properties of such algorithms applied to learning have is implemented by minimizing dt,AsoigD(W),where the been studied theoretically since the 19j0,s [9],[10],[11], function D(W)is called a regularization function,and g is but practical successes for non-trivial tasks did not occur a constant.D(W)is chosen such that it takes large val- until the mid eighties. ues on parameters W that belong to high-capacity subsets C.Grad ent Back-Propagat on of the parameter space.Minimizing D(W)in effect lim- its the capacity of the accessible subset of the parameter Gradient-Based Learning procedures have been used space,thereby controlling the tradeoff between minimiz- since the late 1950,s,but they were mostly limited to lin- ing the training error and minimizing the expected gap ear systems [1].The surprising usefulness of such sim- between the training error and test error. ple gradient descent techniques for complex machine learn- ing tasks was not widely realized until the following three B.Gradent-Based Learn Vig events occurred.The first event was the realization that, The general problem of minimizing a function with re- despite early warnings to the contrary [12],the presence of local minima in the loss function does not seem to spect to a set of parameters is at the root of many issues in computer science.Gradient-Based Learning draws on the be a mapor problem in practice.This became apparent fact that it is generally much easier to minimize a reason- when it was noticed that local minima did not seem to be a mabor impediment to the success of early non-linear ably smooth,continuous function than a discrete (combi- natorial)function.The loss function can be minimized by gradient-based Learning techniques such as Boltzmann ma- chines [13],[14].The second event was the popularization estimating the impact of small variations of the parame- ter values on the loss function.This is measured by the by Rumelhart,Hinton and Williams [15]and others of a gradient of the loss function with respect to the param- simple and eb cient procedure,the back-propagation al- gorithm,to compute the gradient in a non-linear system eters.-b cient learning algorithms can be devised when the gradient vector can be computed analytically (as op- composed of several layers of processing.The third event posed to numerically through perturbations).This is the was the demonstration that the back-propagation proce- basis of numerous gradient-based learning algorithms with dure applied to multi-layer neural networks with sigmoidal continuous-valued parameters.In the procedures described units can solve complicated learning tasks.The basic idea in this article,the set of parameters W is a real-valued vec- of back-propagation is that gradients can be computed eb- tor,with respect to which d(W)is continuous,as well as ciently by propagation from the output to the input.This idea was described in the control theory literature of the differentiable almost everywhere.The simplest minimiza- tion procedure in such a setting is the gradient descent early sixties [1j],but its application to machine learning algorithm where W is iteratively adRuisted as follows: was not generally realized then.Interestingly,the early derivations of back-propagation in the context of neural a(W) network learning did not use gradients,but "virtual tar- WIs WI-1-e- aw (2) gets”for units in intermediate layers[lT],[l月，or minimal disturbance arguments [19].The Lagrange formalism used In the simplest case,e is a scalar constant.More sophisti- in the control theory literature provides perhaps the best cated procedures use variable e,or substitute it for a diag- rigorous method for deriving back-propagation [20],and for onal matrix,or substitute it for an estimate of the inverse deriving generalizations of back-propagation to recurrent
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ è✧ç✎å✠è➞è✥ç✙ï➵✂❛å➈æ➒✡◆ï❶è④ó➵ï➊ï✖é è✧ç✙ï➽ï✄↔✂æ✎ï★↔è✧ï❞ù ï➊ä✥ä✧ì❛ä✌ä✥å➄è✧ï➻ì➈é è✧ç☎ï➓è✧ï❞ê④è ê✧ï❶è✹❉❘✂✁☎✄✿❘ å➈é☎ù✫è✥ç✙ï❭ï➊ä✥ä✧ì❛ä➉ä❿å✠è✧ï❙ì➈é✫è✥ç✙ï❙è✧ä❿å➄ø➀é✙ø➑é✗✂✢ê✧ï❶è✹❉❘❯❚❃❱✡❲❨❳ ù✙ï❯✝ ❶ä✥ï✖å❛ê✤ï❞ê❯ó❨ø➩è✥ç❙è✥ç✙ï✛é➇ÿ✙î➉✡◆ï➊ä❣ì➈ë✎è✥ä✥å➈ø➑é✙ø➀é❼✂➤ê✧å➈î➻æ✙û➑ï❞ê❣å➄æ✙æ☎ä✧ì✪↔✂ø➀î➻å➄è✧ï✖û✙✘ å➈ê ❉❘✂✁☎✄✿❘✝✆ ❉❘❯❚❃❱✡❲❨❳ ✺✟✞❢➪✡✠☞☛✍✌➓➶✏✎ ➪❇➾★➶ ó❨ç✙ï✖ä✧ï✑✌❑ø➀ê✝è✧ç✙ï❣é➇ÿ✙î➑✡✎ï✖ä❀ì➈ë❛è✥ä✥å➈ø➑é✙ø➀é❼✂➔ê✥å➄î➻æ✙û➀ï✖ê✒➌✒✠➞ø➀ê✝å✛î➻ï❞å➈ê✧ÿ✙ä✧ï❦ì➈ë ☛✤ï✄➘◆ï★↔è✥ø➑ú❛ï✛➊å➄æ✎å✑❶ø➑è❅✘✔✌➳ì➈ä➜➊ì➈î➻æ✙û➀ï❯↔✂ø➩è❅✘❙ì➄ë◆è✧ç✙ï➉î➻å➎❿ç✙ø➀é✙ï✜✞✓✬✠✶➌ ✞✕✔✠❖➌✗✖ ø✓ê✇å✌é➇ÿ✙î➑✡✎ï✖ä➃✡◆ï❶è④ó➵ï➊ï✖é✙✘ ❭ ✙➤å➄é✎ù✆➾✫❭ ✘❼➌➇å➄é☎ù✚✞➓ø➀ê➵å➓❶ì❛é☎ê✤è✥å➄é✐è❞ð❯ã✛ç✙ø✓ê ✂❛å➈æ➽å➄û➀ó➵å✪✘✂ê➏ù✙ï✒❶ä✥ï✖å❛ê✤ï❞ê❣ó❨ç✙ï✖é➓è✧ç✙ï♣é➇ÿ✙î➉✡◆ï➊ä✇ì➄ë☛è✧ä❿å➄ø➀é✙ø➀é❼✂✒ê✥å➄î➻æ✙û➀ï✖ê ø➀é✗❶ä✥ï✖å❛ê✤ï❞ê➊ð❨✜✙ÿ✙ä✧è✧ç☎ï➊ä✥î❭ì❛ä✧ï➎➌✐å❛ê❣è✥ç✙ï✞✖å➄æ☎å➎❶ø➑è❅✘✚✠➽ø➀é✗❶ä✥ï✖å❛ê✤ï❞ê✄➌✵❉❙❘❯❚✱❱▼❲❨❳ ù✂ï★❶ä✥ï✖å➈ê✧ï✖ê✖ð❀ã✛ç✙ï✖ä✧ï➊ëíì➈ä✥ï✑➌ró❨ç✙ï✖é❙ø➀é✗❶ä✥ï✖å❛ê✤ø➀é❼✂❨è✥ç✙ï✩➊å➈æ☎å✑➊ø➩è❅✘✛✠❀➌rè✧ç✙ï✖ä✧ï ø✓ê❭å❖è✧ä❿å➈ù✙ï❯✝⑥ì❄➘ ✡✎ï➊è④ó✇ï✖ï➊é❑è✧ç✙ï✺ù✂ï✒➊ä✧ï❞å➈ê✧ï✶ì➈ë✟❉❘❯❚✱❱▼❲❨❳ å➄é✎ù è✧ç☎ï✢ø➀é✦✝ ❶ä✥ï✖å❛ê✤ï✌ì➈ë❇è✥ç✙ï➉✂✐å➄æ✏➌☎ó❨ø➩è✥ç➲å➄é✫ì➈æ✙è✧ø➀î➻å➈û❀ú✠å➄û➀ÿ✙ï✌ì➈ë❯è✥ç✙ï➑✖å➄æ☎å➎❶ø➑è❅✘✜✠ è✧ç✎å✠è❭å➎❿ç✙ø➀ï➊ú➈ï❞ê➤è✧ç☎ï✶û➀ì✠ó✇ï❞ê④è➉✂❛ï➊é✙ï✖ä✥å➈û➑ø✠➽✖å➄è✧ø➀ì➈é➷ï➊ä✥ä✥ì➈ä ❉❙❘✂✁✢✄✿❘↔ð❺ö✫ì✐ê④è û➀ï✖å➄ä✥é✙ø➀é❼✂➽å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➉å✠è✤è✥ï➊î➻æ✂è♣è✧ì➽î➻ø➀é✙ø➑î➻ø✠➽➊ï❄❉❙❘❯❚❃❱✡❲❨❳✺å➈ê➉ó✇ï✖û➑û❯å➈ê ê✧ì➈î➻ï➔ï✖ê✤è✧ø➀î➓å✠è✥ï➔ì➄ë✟è✧ç☎ï✬✂❛å➈æ✝ð❨✕➘ëíì➈ä✥î➓å➄û☎ú❛ï➊ä❿ê✤ø➀ì➈é➻ì➈ë◆è✥ç✙ø✓ê➏ø✓ê➃✖å➄û➀û➑ï❞ù ê✤è✧ä✥ÿ✗↔è✥ÿ✙ä✥å➈û✟ä✧ø✓ê✤ô✶î❭ø➀é✙ø➀î➻ø✙➽❞å✠è✧ø➀ì➈é▲✞✓✠❖➌ ✞✔✆✠✶➌✙å➈é☎ù✶ø✓ê✔✡☎å❛ê✤ï❞ù➽ì➈é✺ù✂ï✄➞☎é✦✝ ø➀é❼✂❭å➻ê✧ï✒➍✐ÿ✙ï✖é✗❶ï➳ì➄ë❀û➀ï✖å➈ä✧é☎ø➑é❼✂❭î➓å✑❿ç✙ø➀é✙ï❞ê✇ì➈ë❀ø➀é✗❶ä✥ï✖å❛ê✤ø➀é❼✂➑✖å➄æ☎å➎❶ø➑è❅✘✑➌ ❶ì❛ä✧ä✥ï✖ê✧æ◆ì➈é☎ù✂ø➀é❼✂❖è✧ì å➷ê✤ï★➍❛ÿ☎ï➊é✗➊ï✢ì➈ë➳ê✧ÿ❼✡☎ê✧ï❶è✥ê➻ì➈ë➉è✥ç✙ï✫æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ê✧æ☎å✑➊ï➽ê✤ÿ✗❿ç➷è✥ç☎å✠è❙ï✖å✑❿ç ê✧ÿ❼✡☎ê✧ï❶è✒ø✓ê✒å❖ê✤ÿ✙æ◆ï➊ä❿ê✧ï❶è➞ì➄ë✛è✥ç✙ï➽æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê ê✧ÿ❼✡☎ê✧ï❶è✖ð➔➏➠é æ✙ä❿å✑↔è✥ø✠✖å➄û❫è✧ï➊ä✥î➓ê✄➌➏þ✐è✥ä✧ÿ✥↔è✧ÿ☎ä✥å➈û✎✍❨ø✓ê✧ô ö✫ø➀é✙ø➀î➻ø✙➽❞å✠è✧ø➀ì➈é ø✓ê✇ø➀î➻æ✙û➀ï➊î➻ï➊é✐è✧ï❞ù➣✡☛✘➻î➻ø➑é✙ø➀î➻ø✙➽✖ø➑é✗✂❄❉❙❘❯❚✱❱▼❲✍❳✤✣✦✥★✧✢➪❩❂➶✹➌✂ó❨ç✙ï✖ä✧ï➉è✧ç✙ï ëíÿ✙é✗❶è✧ø➀ì➈é✙✧➹➪✮❂➶❣ø✓ê➜✖å➄û➀û➑ï❞ù➓å✌ä✥ï✄✂➈ÿ☎û➀å➈ä✧ø✠➽✖å➄è✧ø➀ì➈é❙ëíÿ✙é✗↔è✥ø➑ì❛é✏➌➇å➄é☎ù✩✥✫ø✓ê å✆❶ì❛é☎ê✤è✥å➄é✐è❞ð✪✧✢➪❩❂➶➳ø➀ê➑❿ç✙ì❛ê✧ï➊é ê✧ÿ✗❿ç è✧ç☎å➄è✒ø➑è➞è✥å➈ô➈ï✖ê➞û➀å➈ä❺✂❛ï❭ú✠å➈û➟✝ ÿ✙ï❞ê✇ì❛é✶æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê❀❂ è✥ç☎å✠è✔✡✎ï✖û➑ì❛é❼✂❙è✧ì❭ç✙ø✙✂❛ç✦✝r➊å➄æ✎å✑❶ø➑è❅✘➻ê✤ÿ❼✡✎ê✤ï➊è✥ê ì➄ë❨è✥ç✙ï✶æ✎å➄ä❿å➄î➻ï❶è✥ï➊ä✒ê✧æ☎å✑➊ï➈ð❖ö➲ø➑é✙ø➀î➻ø✙➽✖ø➑é✗✂✫✧✢➪❩❂➶✌ø➀é ï❯➘✟ï✒❶è❙û➀ø➑î➚✝ ø➑è✥ê✌è✥ç✙ï ✖å➄æ☎å➎❶ø➑è❅✘➲ì➄ë✛è✧ç☎ï➽å✑✄➊ï✖ê✥ê✤ø✠✡✙û➀ï➽ê✤ÿ✗✡☎ê✤ï➊è➞ì➄ë✛è✧ç☎ï➽æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä ê✧æ☎å✑➊ï✑➌✇è✧ç✙ï✖ä✧ï✒✡❩✘⑨❶ì❛é❛è✥ä✧ì❛û➑û➀ø➀é❼✂ è✧ç✙ï✺è✥ä✥å❛ù✂ï➊ì✑➘❝✡✎ï➊è④ó✇ï✖ï➊é➘î➻ø➑é☎ø➑î➻ø✠➽❯✝ ø➀é❼✂❤è✥ç✙ï❺è✧ä❿å➄ø➀é✙ø➑é✗✂❤ï➊ä✥ä✥ì➈ä✢å➈é☎ù✻î➻ø➀é✙ø➀î❭ø✠➽➊ø➀é❼✂❑è✧ç✙ï ï❯↔✂æ◆ï✒❶è✧ï✖ù❝✂❛å➈æ ✡◆ï❶è④ó➵ï➊ï➊é✿è✥ç✙ï➤è✧ä❿å➄ø➀é✙ø➀é❼✂➻ï➊ä✥ä✧ì❛ä➵å➈é☎ù✢è✧ï✖ê✤è➔ï➊ä✥ä✥ì➈ä❞ð ✬✛ ➠✔➡❺➢ ✪✣ ➳✄➨✗➺❩✭✬ ➢♦➩❯➳✱✪ ✢ ➳❘➢❄➡✹➨✤✣●➨✦✥ ã✛ç✙ï➣✂➈ï✖é✙ï➊ä❿å➄û❣æ☎ä✧ì➎✡✙û➑ï✖î ì➄ë❨î➻ø➀é✙ø➑î➻ø✠➽➊ø➀é❼✂➲å✿ëíÿ✙é✗❶è✧ø➀ì➈é ó❨ø➑è✧ç ä✥ï❯✝ ê✧æ✎ï★↔è➏è✥ì➞å✌ê✤ï➊è❫ì➄ë✟æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê❦ø✓ê➏å✠è❣è✥ç✙ï➔ä✥ì✐ì➈è➏ì➄ë✟î➓å➄é☛✘✒ø✓ê✧ê✧ÿ✙ï❞ê❣ø➑é ❶ì❛î➻æ✙ÿ✂è✧ï✖ä✌ê❺➊ø➑ï✖é✗❶ï❛ð✌â➳ä❿å➈ù✙ø➑ï✖é❛è❺✝❖✚✛å❛ê✤ï❞ù✳✗❀ï✖å➈ä✧é✙ø➀é❼✂✿ù✂ä✥åró➔ê➉ì➈é❖è✧ç✙ï ë⑨å✑❶è➔è✧ç✎å✠è➉ø➑è➉ø✓ê✛✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘✶î✒ÿ✥❿ç✿ï❞å➈ê✧ø➑ï✖ä❨è✧ì➓î➻ø➀é✙ø➀î❭ø✠➽➊ï❭å➻ä✧ï❞å➈ê✧ì➈é✦✝ å❄✡☎û✙✘❖ê✧î➻ì➇ì➄è✧ç❀➌✏❶ì➈é✐è✥ø➑é➇ÿ✙ì❛ÿ☎ê♣ëíÿ✙é✗↔è✥ø➑ì❛é è✧ç☎å➈é å✿ù✂ø✓ê❺➊ä✧ï➊è✧ï✆➪ ➊ì➈î➉✡☎ø➟✝ é☎å➄è✧ì➈ä✥ø✓å➄û●➶➏ëíÿ☎é✗↔è✥ø➑ì❛é✝ð➵ã✛ç☎ï✌û➑ì✐ê✧ê➵ëíÿ✙é✥↔è✧ø➀ì➈é↕✖å➄é✫✡◆ï➞î➻ø➑é✙ø➀î➻ø✙➽✖ï✖ù✫✡☛✘ ï✖ê✤è✧ø➀î➓å✠è✥ø➑é❼✂❖è✧ç✙ï✿ø➀î❭æ✎å✑↔è❭ì➄ë➉ê✧î➓å➄û➀û✇ú✠å➄ä✥ø✓å✠è✧ø➀ì➈é✎ê➞ì➄ë❨è✥ç✙ï✿æ☎å➄ä❿å➄î➻ï❯✝ è✧ï✖ä❙ú✠å➈û➑ÿ✙ï❞ê✒ì❛é➷è✧ç✙ï✿û➀ì❛ê✥ê✌ëíÿ✙é✗❶è✧ø➀ì➈é✝ð➷ã✛ç✙ø✓ê✒ø✓ê❙î❭ï❞å➈ê✧ÿ✙ä✥ï✖ù ✡☛✘ è✧ç✙ï ✂➈ä❿å➈ù✙ø➑ï✖é❛è➓ì➈ë➉è✧ç☎ï✫û➑ì✐ê✧ê❭ëíÿ✙é✥↔è✧ø➀ì➈é➘ó❨ø➑è✧ç❍ä✥ï✖ê✧æ✎ï★↔è➻è✧ì➷è✥ç✙ï✫æ☎å➈ä✥å➈î➑✝ ï❶è✥ï➊ä❿ê➊ð✮✭✑✯➣❶ø➀ï➊é✐è❭û➀ï✖å➈ä✧é☎ø➑é❼✂➷å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➑➊å➈é➹✡◆ï✺ù✙ï➊ú➇ø➀ê✧ï✖ù❤ó❨ç✙ï➊é è✧ç☎ï➣✂➈ä❿å➈ù✂ø➀ï➊é✐è➞ú❛ï✒↔è✥ì➈ä➉✖å➄é ✡◆ï➙❶ì❛î➻æ✙ÿ✂è✧ï❞ù å➄é☎å➈û✙✘✐è✥ø✠✖å➄û➀û✙✘➹➪üå➈ê✒ì➈æ✦✝ æ◆ì❛ê✧ï✖ù❖è✧ì✫é➇ÿ✙î➻ï➊ä✥ø✠✖å➄û➀û✙✘✺è✥ç✙ä✥ì➈ÿ❼✂❛ç❺æ◆ï➊ä✧è✧ÿ✙ä❘✡☎å➄è✧ø➀ì➈é☎ê✴➶↔ð➓ã✛ç✙ø✓ê➤ø➀ê✌è✧ç✙ï ✡☎å❛ê✤ø✓ê❫ì➈ë❀é➇ÿ✙î➻ï➊ä✥ì➈ÿ☎ê➃✂➈ä❿å➈ù✙ø➑ï✖é❛è❺✝✶✡✎å➈ê✧ï✖ù➻û➀ï✖å➄ä✥é✙ø➀é❼✂❙å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î➓ê❫ó❨ø➑è✧ç ❶ì❛é✐è✧ø➀é✐ÿ☎ì➈ÿ☎ê❇✝♠ú✠å➈û➑ÿ✙ï❞ù➤æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê✖ð✏➏➠é➞è✥ç✙ï✇æ☎ä✧ì✦❶ï❞ù✂ÿ✙ä✥ï✖ê❇ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù ø➀é❙è✥ç✙ø✓ê➏å➄ä✧è✧ø✁❶û➀ï✑➌✠è✥ç✙ï❨ê✤ï➊è➏ì➄ë◆æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê ❂ ø✓ê❣å➳ä✧ï❞å➄û✙✝♠ú✠å➄û➀ÿ✙ï❞ù➞ú➈ï★✹✝ è✧ì❛ä✒➌✟ó❨ø➑è✧ç➲ä✥ï✖ê✧æ◆ï✒↔è➳è✧ì✢ó❨ç✙ø✁❿ç ❉➣➪✮❂➶➉ø✓ê④➊ì➈é✐è✧ø➀é➇ÿ✙ì➈ÿ☎ê✒➌✟å➈ê♣ó✇ï✖û➑û❦å➈ê ù✂ø✙➘◆ï✖ä✧ï✖é✐è✧ø✓å❄✡✙û➀ï➓å➄û➀î❭ì✐ê④è➤ï➊ú❛ï➊ä❘✘➇ó❨ç✙ï➊ä✥ï➈ð➞ã✛ç✙ï➓ê✧ø➑î➻æ✙û➀ï✖ê✤è➤î❭ø➀é✙ø➀î➻ø✙➽❞å♦✝ è✧ø➀ì➈é✲æ✙ä✧ì✦➊ï✖ù✂ÿ✙ä✥ï❖ø➀é✲ê✧ÿ✗❿ç✲å❑ê✤ï➊è✤è✧ø➀é❼✂❑ø➀ê✶è✧ç✙ï➛✂➈ä❿å➈ù✂ø➀ï➊é✐è✢ù✙ï✖ê❘❶ï➊é✐è å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î ó❨ç✙ï➊ä✥ï✜❂ ø✓ê❨ø➑è✧ï➊ä❿å✠è✥ø➑ú❛ï➊û✠✘➽å➈ù✆✓④ÿ✎ê④è✥ï✖ù✺å➈ê➵ëíì❛û➑û➀ì✠ó➔ê✱✰ ❂✳✲✹✺ ❂✳✲✒✴ ✜ ✆✶✵✸✷❉➣➪❩❂➶ ✷❂ ❭ ➪❩✑✑➶ ➏➠é✿è✧ç✙ï❙ê✤ø➀î➻æ✙û➑ï❞ê④è✬✖å➈ê✧ï✑➌ ✵ ø➀ê➉å❭ê❘➊å➈û➀å➈ä✛❶ì❛é☎ê④è❿å➄é✐è✖ð✇ö✫ì➈ä✥ï✌ê✤ì❛æ✙ç✙ø✓ê④è✥ø➟✝ ➊å➄è✧ï❞ù➽æ✙ä✥ì✦❶ï✖ù✙ÿ✙ä✧ï❞ê❫ÿ✎ê✤ï➳ú✠å➄ä✥ø➀å✑✡✙û➑ï ✵ ➌✐ì❛ä➵ê✧ÿ❼✡☎ê✤è✧ø➑è✧ÿ✙è✧ï➤ø➩è➵ëíì➈ä❨å❙ù✂ø➀å✑✂❄✝ ì➈é✎å➄û✝î➓å✠è✥ä✧ø✙↔❢➌✙ì➈ä➔ê✧ÿ❼✡☎ê✤è✧ø➑è✧ÿ✙è✧ï➞ø➑è➔ëíì➈ä♣å➄é✺ï✖ê✤è✧ø➀î➓å✠è✧ï✌ì➈ë❇è✥ç✙ï➞ø➀é➇ú➈ï➊ä❿ê✧ï õ➔ï❞ê✧ê✧ø✓å➄é❤î➓å➄è✧ä✥ø➟↔ å➈ê❭ø➀é❑ñ➉ï➊ó✛è✧ì❛é❤ì❛ä✚✹♣ÿ✎å➈ê✧ø➟✝➠ñ➔ï✖ó✛è✧ì➈é❤î❭ï➊è✧ç✙ì✂ù✙ê✖ð ã✛ç✙ï➹☞✇ì➈é✬✓④ÿ❼✂❛å➄è✧ï â➳ä✥å❛ù✂ø➀ï➊é✐è✫î❭ï➊è✧ç✙ì✂ù ✞✺✠➉✖å➄é å➄û✓ê✤ì⑨✡◆ï ÿ☎ê✤ï❞ù☛ð õ➔ì✠ó➵ï➊ú❛ï➊ä★➌➑✕➉æ✙æ✎ï✖é☎ù✂ø✙↔❫✚❅ê✧ç✙ì✠ó➔ê❖è✧ç☎å➄è➷ù✂ï❞ê✤æ☎ø➩è✥ï î➓å➈é❩✘②➊û➀å➈ø➑î➓ê è✧ì➷è✥ç✙ï➔➊ì➈é✐è✧ä❿å➄ä❘✘ ø➑é➘è✥ç✙ï✫û➀ø➩è✥ï➊ä❿å✠è✧ÿ☎ä✧ï➎➌✇è✥ç✙ï➲ÿ☎ê✧ï❶ëíÿ☎û➑é✙ï❞ê✧ê➽ì➄ë➳è✧ç✙ï❞ê✤ï ê✧ï✒❶ì❛é☎ù☛✝⑥ì➈ä❿ù✂ï➊ä➻î➻ï❶è✥ç✙ì✂ù✙ê➻è✥ì û➀å➈ä❺✂❛ï✿û➀ï✖å➈ä✧é☎ø➑é❼✂➷î➓å✑❿ç☎ø➑é✙ï❞ê➻ø➀ê➽ú➈ï➊ä❘✘ û➀ø➑î➻ø➑è✧ï✖ù✝ð ✕✾æ◆ì➈æ☎ÿ✙û➀å➈ä✇î➻ø➀é✙ø➀î❭ø✠➽✖å➄è✧ø➀ì➈é✿æ✙ä✥ì☛➊ï✖ù✂ÿ☎ä✧ï♣ø➀ê➵è✥ç✙ï➳ê✤è✧ì✦❿ç☎å❛ê④è✥ø✠④✂➈ä❿å♦✝ ù✂ø➀ï➊é✐è➉å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➐➌✎å➄û✓ê✤ì➵➊å➈û➑û➀ï✖ù✶è✧ç☎ï➞ì➈é✦✝⑥û➀ø➑é✙ï➤ÿ✙æ✟ù✙å➄è✧ï➈ð➃➏⑥è④❶ì❛é☎ê✧ø➀ê✤è✥ê ø➀é➽ÿ✙æ✟ù✙å✠è✥ø➑é❼✂❙è✧ç☎ï➉æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä❫ú➈ï★↔è✥ì➈ä➏ÿ☎ê✧ø➀é❼✂❙å✒é✙ì➈ø✓ê❇✘➎➌❛ì❛ä✇å➈æ✙æ✙ä✥ì✪↔❩✝ ø➀î➻å➄è✧ï❞ù❢➌☎ú❛ï➊ä❿ê✤ø➀ì➈é✿ì➄ë❦è✧ç☎ï✒årú➈ï✖ä✥å✑✂➈ï✞✂➈ä❿å➈ù✂ø➀ï➊é✐è✖ð➃➏➠é✺è✧ç✙ï✒î❭ì✐ê④è✞❶ì➈î➚✝ î➻ì➈é❺ø➑é☎ê✤è✥å➈é✗❶ï➻ì➈ë❫ø➩è★➌ ❂ ø✓ê➤ÿ✙æ✟ù✙å✠è✥ï✖ù❺ì➈é❖è✧ç✙ï➝✡☎å❛ê✤ø✓ê➳ì➄ë✇å✺ê✧ø➑é❼✂❛û➑ï ê✥å➄î➻æ✙û➀ï✻✰ ❂✲ ✺ ❂✲✒✴ ✜ ✆✳✵ ✷❉❊✸✽✼☛➪✮❂➶ ✷❂ ➪✮❜➎➶ ✎ø➑è✧ç è✧ç☎ø➀ê æ☎ä✧ì✦❶ï❞ù✂ÿ✙ä✥ï✴è✥ç✙ï æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä ú➈ï✒❶è✧ì❛ä✿✾✎ÿ✗↔è✥ÿ☎å✠è✥ï✖ê å➄ä✥ì➈ÿ☎é☎ù➞å➄é❙årú➈ï➊ä❿å❄✂❛ï❯è✧ä❿å✓④ï✒❶è✧ì❛ä❺✘➎➌★✡☎ÿ✂è❯ÿ✎ê✤ÿ☎å➈û➑û✠✘✌❶ì❛é✐ú❛ï➊ä❘✂➈ï❞ê✏❶ì❛é☎ê✤ø✓ù☛✝ ï➊ä❿å❄✡☎û✙✘➓ë⑨å❛ê④è✥ï➊ä❨è✥ç☎å➄é✫ä✥ï✄✂❛ÿ✙û✓å➄ä✩✂➈ä❿å➈ù✙ø➑ï✖é❛è➔ù✂ï❞ê❺➊ï➊é✐è➉å➄é✎ù✿ê✧ï✒➊ì➈é☎ù✿ì❛ä❇✝ ù✂ï✖ä➏î➻ï❶è✧ç☎ì➇ù☎ê➏ì➈é➓û✓å➄ä❘✂➈ï➵è✥ä✥å➈ø➑é✙ø➀é❼✂➞ê✧ï❶è❿ê➏ó❨ø➑è✧ç➓ä✥ï✖ù✂ÿ✙é✎ù✙å➄é✐è✇ê✥å➄î➻æ✙û➀ï✖ê ➪⑨ê✧ÿ✗❿ç✫å➈ê✇è✥ç✙ì❛ê✧ï✌ï➊é✗➊ì➈ÿ✙é✐è✥ï➊ä✥ï✖ù➽ø➀é✫ê✤æ◆ï➊ï★❿ç✺ì➈ä✩❿ç☎å➄ä❿å✑❶è✧ï✖ä➵ä✥ï✒➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é✈➶↔ð✶ã✛ç✙ï➓ä✥ï✖å❛ê✤ì❛é☎ê♣ëíì➈ä➤è✥ç✙ø➀ê✒å➄ä✥ï❭ï✄↔✂æ✙û➀å➈ø➑é☎ï✖ù❺ø➑é①✕➔æ☎æ✎ï✖é☎ù✂ø✙↔ ✚➳ð ã✛ç✙ï➳æ✙ä✥ì➈æ◆ï➊ä✧è✧ø➀ï✖ê➏ì➈ë❇ê✧ÿ✗❿ç✶å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê➵å➄æ☎æ✙û➑ø➀ï✖ù➓è✧ì❭û➑ï❞å➄ä✥é✙ø➑é✗✂➞ç☎årú➈ï ✡◆ï➊ï➊é❍ê✤è✧ÿ☎ù✙ø➑ï❞ù è✥ç✙ï➊ì❛ä✧ï➊è✧ø✁➊å➄û➀û✠✘❤ê✤ø➀é✗➊ï✢è✥ç✙ï ➾✽❀✗✓✻✘❂❁ ê✳✞❀✬✠✶➌✒✞➟➾✒✘✠❖➌✟✞✙➾✑➾❪✠✶➌ ✡✙ÿ✂è✌æ✙ä❿å✑❶è✧ø✁➊å➈û❦ê✧ÿ✗✒❶ï✖ê✥ê✧ï✖ê♣ëíì➈ä➤é✙ì❛é✦✝üè✥ä✧ø➀ú➇ø➀å➈û❯è✥å➈ê✧ô✂ê➤ù✂ø➀ù❺é✙ì➈è➤ì✦✒❶ÿ✙ä ÿ✙é✐è✧ø➀û☛è✥ç✙ï✌î➻ø➀ù✿ï➊ø✠✂➈ç✐è✥ø➑ï❞ê➊ð ❃✛ ➠✔➡❺➢ ✪✣ ➳✄➨✗➺ ✬ ➢✻❄✴➼✭✡❅✎➡❘➯❘➤✗➢❪✥❩➢♦➺❩✣ ➯♦➨ â➳ä✥å❛ù✂ø➑ï✖é✐è❇✝r✚➵å❛ê✤ï❞ù ✗✝ï❞å➄ä✥é✙ø➀é❼✂➶æ☎ä✧ì✦❶ï❞ù✂ÿ✙ä✥ï✖ê ç☎årú❛ï ✡◆ï➊ï✖éPÿ☎ê✧ï✖ù ê✧ø➑é✗➊ï➞è✥ç✙ï❙û✓å✠è✥ï ➾✒❀ ✙❆✘❇❁ ê✒➌✗✡☎ÿ✂è♣è✥ç✙ï✄✘✫ó➵ï➊ä✥ï✒î➻ì✐ê④è✥û✙✘✺û➀ø➑î➻ø➑è✧ï✖ù➲è✧ì✢û➑ø➀é✦✝ ï✖å➈ä✫ê❇✘✂ê✤è✧ï➊î➓ê▲✞✙➾✡✠⑥ð▼ã✛ç✙ï ê✧ÿ✙ä✥æ✙ä✧ø✓ê✧ø➑é❼✂❑ÿ☎ê✧ï❶ëíÿ✙û➀é✙ï✖ê✥ê✺ì➄ë➻ê✤ÿ✥❿ç✾ê✧ø➑î➚✝ æ✙û➀ï✩✂➈ä❿å➈ù✂ø➀ï➊é✐è➏ù✂ï❞ê❺➊ï➊é✐è❣è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✖ê❦ëíì❛ä➜❶ì❛î❭æ☎û➑ï✄↔✒î➓å✑❿ç✙ø➀é✙ï❨û➀ï✖å➄ä✥é✦✝ ø➀é❼✂➓è✥å❛ê✤ô✂ê➉ó➵å❛ê❨é✙ì➄è➳ó❨ø➀ù✙ï➊û✠✘✢ä✥ï✖å➈û➑ø✠➽➊ï❞ù✺ÿ✙é✐è✧ø➀û❀è✥ç✙ï➞ëíì❛û➑û➀ì✠ó❨ø➑é✗✂➻è✧ç✙ä✥ï➊ï ï➊ú❛ï➊é✐è✥ê➳ì✦✄➊ÿ✙ä✥ä✧ï❞ù☛ð❙ã✛ç✙ï➚➞☎ä❿ê✤è➤ï✖ú➈ï✖é❛è✌ó✛å➈ê♣è✧ç☎ï❭ä✥ï✖å➈û➑ø✠➽✖å➄è✧ø➀ì➈é❖è✧ç☎å➄è✒➌ ù✂ï❞ê✤æ✙ø➑è✧ï✺ï❞å➄ä✥û✙✘➷ó✛å➄ä✥é✙ø➑é✗✂❛ê✌è✥ì è✧ç✙ï✫❶ì❛é❛è✥ä✥å➈ä❺✘✩✞✙➾ ✑✆✠❖➌❣è✥ç✙ï✺æ✙ä✧ï❞ê✤ï✖é✗❶ï ì➄ë✢û➑ì✦➊å➈û➻î❭ø➀é✙ø➀î➓å✾ø➀éòè✥ç✙ï➘û➑ì✐ê✧ê❺ëíÿ✙é✥↔è✧ø➀ì➈é▲ù✂ì➇ï✖ê é✙ì➈è➷ê✧ï➊ï✖î è✧ì ✡◆ï❖å î➓å✓④ì❛ä➽æ✙ä✥ì✑✡✙û➀ï➊î ø➑é✻æ☎ä✥å➎↔è✧ø✁❶ï❛ð❲ã✛ç☎ø➀ê ✡✎ï★➊å➄î➻ï❖å➈æ✙æ☎å➈ä✧ï✖é❛è ó❨ç✙ï✖é✻ø➩è✿ó➵å❛ê➓é✙ì➈è✧ø✁❶ï✖ù❍è✧ç☎å➄è✢û➀ì☛✖å➄û♣î❭ø➀é✙ø➀î➓å ù✂ø✓ù❍é✙ì➄è✺ê✤ï✖ï➊î è✧ì ✡◆ï➻å✿î➓å✓④ì❛ä➤ø➑î➻æ◆ï✖ù✂ø➀î➻ï➊é✐è✌è✧ì✿è✧ç☎ï➻ê✧ÿ✗✒❶ï✖ê✥ê➳ì➄ë➵ï✖å➄ä✥û✠✘✫é✙ì➈é✦✝⑥û➀ø➑é✙ï❞å➄ä ✂➈ä❿å➈ù✙ø➑ï✖é❛è❺✝✶✡✎å➈ê✧ï✖ù❊✗✝ï✖å➈ä✧é☎ø➑é❼✂➔è✧ï✒❿ç☎é✙ø✠➍✐ÿ✙ï❞ê❀ê✧ÿ✗❿ç➞å➈ê❳✚➵ì➈û➑è❺➽➊î➓å➈é✙é➞î➓å♦✝ ❿ç✙ø➀é✙ï✖ê ✞➟➾ ❜✠❖➌ ✞✙➾❪❝✬✠⑥ð✛ã✛ç✙ï✒ê✧ï✒❶ì❛é☎ù✿ï➊ú❛ï➊é✐è➔ó➵å❛ê✛è✧ç☎ï➞æ✎ì❛æ✙ÿ✙û✓å➄ä✥ø✙➽❞å✠è✥ø➑ì❛é ✡☛✘①✍❨ÿ✙î➻ï➊û➀ç☎å➄ä✧è✒➌❣õ➉ø➑é✐è✧ì❛é❤å➄é✎ù ✎ø➑û➀û➀ø➀å➈î➻ê✳✞✙➾ ✙✆✠✛å➄é☎ù ì➄è✧ç☎ï➊ä❿ê➞ì➄ë➔å ê✧ø➑î➻æ✙û➀ï å➄é☎ù✲ï❈✯➣❶ø➀ï➊é✐è✫æ✙ä✥ì☛➊ï✖ù✂ÿ☎ä✧ï➎➌➔è✧ç☎ï➒✡✎å✑❿ô❩✝♠æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é❍å➈û➟✝ ✂➈ì❛ä✧ø➑è✧ç☎î➐➌❦è✥ì➛➊ì➈î➻æ✙ÿ✂è✥ï➽è✧ç☎ï➙✂❛ä✥å❛ù✂ø➑ï✖é✐è✒ø➑é❑å➲é✙ì❛é✦✝♠û➀ø➀é✙ï✖å➈ä❙ê❺✘➇ê✤è✧ï✖î ❶ì❛î➻æ✎ì✐ê✤ï❞ù✫ì➄ë✇ê✧ï➊ú❛ï➊ä❿å➄û❯û➀å✪✘❛ï➊ä❿ê➔ì➈ë❫æ✙ä✥ì☛➊ï✖ê✥ê✤ø➀é❼✂✎ð➳ã✛ç✙ï❙è✧ç☎ø➑ä❿ù❖ï✖ú➈ï✖é❛è ó✛å➈ê❙è✧ç✙ï➲ù✙ï➊î➻ì➈é☎ê✤è✧ä❿å✠è✥ø➑ì❛é❤è✥ç☎å✠è➻è✥ç✙ï✆✡☎å✑❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é æ✙ä✥ì☛➊ï❯✝ ù✂ÿ✙ä✥ï➉å➈æ✙æ✙û➀ø➑ï❞ù➻è✧ì➞î❙ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä➏é✙ï➊ÿ☎ä✥å➈û✙é✙ï❶è④ó➵ì➈ä✥ô✂ê❣ó❨ø➑è✧ç✶ê✧ø✙✂❛î➻ì➈ø✓ù✙å➄û ÿ✙é✙ø➑è✥ê✬➊å➄é✺ê✧ì➈û➀ú➈ï✌❶ì❛î❭æ☎û➑ø✁➊å➄è✧ï✖ù✢û➀ï✖å➄ä✥é✙ø➀é❼✂❙è✥å➈ê✧ô✂ê➊ð➏ã✛ç☎ï✞✡✎å➈ê✧ø✠➳ø✓ù✂ï✖å ì➄ë✇✡☎å➎❿ô➎✝⑥æ✙ä✥ì➈æ☎å✑✂❛å➄è✧ø➀ì➈é✌ø✓ê❯è✧ç☎å➄è❹✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê❨✖å➄é➚✡✎ï✛➊ì➈î➻æ✙ÿ✂è✥ï✖ù❭ï❈✯➝✝ ❶ø➀ï➊é✐è✥û✙✘➣✡☛✘➽æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é➽ëíä✧ì❛î è✧ç✙ï✌ì❛ÿ✂è✧æ☎ÿ✂è❨è✧ì❭è✧ç☎ï✌ø➑é✙æ☎ÿ✂è✖ð➏ã✛ç✙ø✓ê ø✓ù✂ï✖å✫ó✛å➈ê✌ù✙ï✖ê❘❶ä✥ø✙✡◆ï✖ù➷ø➀é➷è✧ç✙ï➙❶ì➈é✐è✥ä✧ì❛û❣è✧ç✙ï✖ì➈ä❘✘❖û➀ø➩è✥ï➊ä❿å✠è✥ÿ✙ä✧ï➓ì➈ë✛è✧ç✙ï ï✖å➈ä✧û✠✘➷ê✤ø✙↔➇è✧ø➀ï✖ê✳✞➟➾✒✓✠❖➌❜✡☎ÿ✂è❙ø➑è✥ê❭å➄æ✙æ☎û➑ø✁➊å➄è✧ø➀ì➈é è✥ì❖î➓å✑❿ç✙ø➀é✙ï➽û➀ï✖å➈ä✧é☎ø➑é❼✂ ó✛å➈ê➓é✙ì➈è ✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘❤ä✥ï✖å➄û➀ø✠➽➊ï✖ù❑è✥ç✙ï➊é✝ð ➏➠é✐è✥ï➊ä✥ï✖ê✤è✧ø➀é❼✂➈û✠✘✑➌✛è✥ç✙ï➲ï✖å➈ä✧û✠✘ ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ì➈é✎ê➽ì➄ë✌✡☎å➎❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é ø➑é✻è✧ç✙ï➒❶ì➈é✐è✥ï❯↔➇è✶ì➄ë➞é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô û➑ï❞å➄ä✥é✙ø➀é❼✂❖ù✂ø✓ù é✙ì➈è❭ÿ✎ê✤ï➙✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê✒➌❨✡✙ÿ✂è✏☛✤ú➇ø➀ä✤è✥ÿ☎å➄û❫è✥å➈ä❇✝ ✂➈ï➊è✥ê❃✌➞ëíì❛ä❨ÿ✙é✙ø➑è✥ê✛ø➀é✺ø➑é✐è✧ï✖ä✧î➻ï❞ù✂ø➀å➄è✧ï➳û➀å✪✘❛ï➊ä❿ê✒✞➟➾✍✔✠❖➌ ✞✙➾✽✺✬✠✶➌✂ì❛ä✛î❭ø➀é✙ø➀î➓å➄û ù✂ø✓ê④è✥ÿ✙ä❘✡☎å➄é✗➊ï➳å➈ä❺✂❛ÿ✙î➻ï➊é✐è✥ê✟✞➟➾✒❀✠⑥ð❣ã✛ç✙ï❊✗❇å❄✂❛ä✥å➈é❼✂➈ï❨ëíì❛ä✧î➓å➈û➑ø✓ê✤î▲ÿ☎ê✧ï✖ù ø➀é❖è✥ç✙ï➣❶ì❛é❛è✥ä✧ì❛û❀è✥ç✙ï➊ì❛ä❺✘✫û➀ø➑è✧ï➊ä❿å✠è✥ÿ✙ä✥ï❙æ✙ä✥ì✠ú➇ø➀ù✙ï✖ê➳æ✎ï✖ä✧ç✎å➄æ☎ê♣è✧ç☎ï➵✡✎ï❞ê④è ä✥ø✙✂❛ì➈ä✥ì➈ÿ☎ê☛î➻ï➊è✧ç✙ì✂ù➤ëíì❛ä❇ù✂ï✖ä✧ø➀ú➇ø➑é❼✂④✡☎å✑❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é ✞✑❆✘✬✠✶➌✠å➄é✎ù➤ëíì➈ä ù✂ï✖ä✧ø➀ú➇ø➑é❼✂➒✂➈ï✖é✙ï➊ä❿å➄û➀ø✠➽✖å✠è✥ø➑ì❛é☎ê❙ì➄ë④✡✎å✑❿ô❩✝♠æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ø➀ì➈é➷è✥ì➷ä✧ï★❶ÿ✙ä✥ä✧ï✖é✐è
PROC.OF THE IEEE,NOVEMBER 1998 networks 21,and networks of heterogeneous modules 22 ferentiable,and therefore lends itself to the use of Gradient- A simple derivation for generic multi-layer systems is given Based Learning methods.Section V introduces the use of in Section I-E. directed acyclic graphs whose arcs carry numerical infor- The fact that local minima do not seem to be a problem mation as a way to represent the alternative hypotheses, for multi-layer neural networks is somewhat of a theoretical and introduces the idea of GTN. mystery.It is conPectured that if the network is oversized The second solution described in Section VII is to elim- for the task (as is usually the case in practice),the presence inate segmentation altogether.The idea is to sweep the of "extra dimensions"in parameter space reduces the risk recognizer over every possible location on the input image, of unattainable regions.Back-propagation is by far the and to rely on the "character spotting"property of the rec- most widely used neural-network learning algorithm,and ognizer,i.e.its ability to correctly recognize a well-centered probably the most widely used learning algorithm of any character in its input field,even in the presence of other form. characters besides it,while relecting images containing no centered characters [26],[27].The sequence of recognizer D.Learn Vig Vi F eal s andurig F ecogn on Systems outputs obtained by sweeping the recognizer over the in- Isolated handwritten character recognition has been ex- put is then fed to a Graph Transformer Network that takes tensively studied in the literature(see [23],[24]for reviews). linguistic constraints into account and finally extracts the and was one of the early successful applications of neural most likely interpretation.This GTN is somewhat similar networks [25].Comparative experiments on recognition of to Hidden Markov Models (HMM),which makes the ap- individual handwritten digits are reported in Section III. proach reminiscent of the classical speech recognition [28], They show that neural networks trained with Gradient- [29].While this technique would be quite expensive in Based Learning perform better than all other methods the general case,the use of Convolutional Neural Networks tested here on the same data.The best neural networks, makes it particularly attractive because it allows significant called Convolutional Networks,are designed to learn to savings in computational cost extract relevant features directly from pixel images (see Section II). 7.Glonally Travanle Systems One of the most difficult problems in handwriting recog- As stated earlier,most practical pattern recognition sys- nition,however,is not only to recognize individual charac- tems are composed of multiple modules.For example,a ters,but also to separate out characters from their neigh- document recognition system is composed of a field locator, bors within the word or sentence,a process known as seg- which extracts regions of interest,a field segmenter,which mentation.The technique for doing this that has become cuts the input image into images of candidate characters,a the "standard"is called eurVt Over-Segmentatin.It recognizer,which classifies and scores each candidate char- consists in generating a large number of potential cuts acter,and a contextual post-processor,generally based on between characters using heuristic image processing tech- a stochastic grammar,which selects the best grammatically niques,and subsequently selecting the best combination of correct answer from the hypotheses generated by the recog- cuts based on scores given for each candidate character by nizer.In most cases,the information carried from module the recognizer.In such a model,the accuracy of the sys- to module is best represented as graphs with numerical in- tem depends upon the quality of the cuts generated by the formation attached to the arcs.For example,the output heuristics,and on the ability of the recognizer to distin-of the recognizer module can be represented as an acyclic guish correctly segmented characters from pieces of char-graph where each arc contains the label and the score of acters,multiple characters,or otherwise incorrectly seg- a candidate character,and where each path represent a mented characters.Training a recognizer to perform this alternative interpretation of the input string.Typically, task poses a mapor challenge because of the difficulty in cre- each module is manually optimized,or sometimes trained, ating a labeled database of incorrectly segmented charac- outside of its context.For example,the character recog- ters.The simplest solution consists in running the images nizer would be trained on labeled images of pre-segmented of character strings through the segmenter,and then man- characters.Then the complete system is assembled,and ually labeling all the character hypotheses.Unfortunately,a subset of the parameters of the modules is manually ad- not only is this an extremely tedious and costly task,it is Pusted to maximize the overall performance.This last step also difficult to do the labeling consistently.For example, is extremely tedious,time-consuming,and almost certainly should the right half of a cut up 4 be labeled as a 1 or as suboptimal. a non-character2 should the right half of a cut up 8 be A better alternative would be to somehow train the en- labeled as a 32 tire system so as to minimize a global error measure such as The first solution,described in Section V consists in the probability of character misclassifications at the docu- training the system at the level of whole strings of char-ment level.Ideally,we would want to find a good minimum acters,rather than at the character level.The notion of of this global loss function with respect to all the param- Gradient-Based Learning can be used for this purpose.The eters in the system.If the loss function d measuring the system is trained to minimize an overall loss function which performance can be made differentiable with respect to the measures the probability of an erroneous answer.Section V system's tunable parameters W,we can find a local min- explores various ways to ensure that the loss function is dif- imum of d using Gradient-Based Learning.However,at
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ é✙ï➊è④ó✇ì❛ä✧ô✂ê❅✞✑❼➾✡✠❖➌➈å➄é✎ù✒é✙ï➊è④ó✇ì❛ä✧ô✂ê❯ì➄ë✟ç✙ï➊è✧ï➊ä✥ì✑✂❛ï➊é✙ï✖ì➈ÿ☎ê❇î➻ì✂ù✂ÿ✙û➀ï✖ê ✞✑✫✑ ✠♠ð ✕✻ê✧ø➑î➻æ✙û➀ï➔ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ì➈é❭ëíì❛ä❷✂➈ï➊é☎ï➊ä✥ø✠✛î❙ÿ✙û➩è✥ø➟✝⑥û✓å✪✘➈ï➊ä❣ê❺✘✂ê④è✥ï➊î➓ê❣ø✓ê❷✂➈ø➀ú➈ï✖é ø➀é✫þ✂ï✒↔è✥ø➑ì❛é✫➏❖✝✭➉ð ã✛ç✙ï➉ë⑨å➎↔è✇è✧ç☎å➄è✇û➀ì☛✖å➄û◆î➻ø➑é✙ø➀î➓å❙ù✙ì✒é✙ì➈è➵ê✧ï➊ï➊î✭è✥ì➉✡◆ï➳å✒æ✙ä✧ì➎✡✙û➀ï➊î ëíì➈ä❯î✒ÿ☎û➩è✥ø➟✝⑥û➀å✪✘❛ï➊ä❇é✙ï➊ÿ✙ä❿å➄û✐é✙ï➊è④ó✇ì❛ä✧ô✂ê✝ø✓ê❦ê✤ì❛î➻ï➊ó❨ç☎å➄è❇ì➄ë◆å❨è✧ç☎ï➊ì➈ä✥ï❶è✥ø✠✖å➄û î➉✘✂ê✤è✧ï✖ä❺✘❛ð✬➏⑥è♣ø✓ê✞❶ì➈é✬✓④ï✒↔è✥ÿ✙ä✥ï✖ù✺è✥ç☎å✠è➳ø➩ë➏è✥ç✙ï❭é✙ï❶è④ó➵ì➈ä✥ô✶ø✓ê➉ì✠ú❛ï➊ä❿ê✤ø✠➽➊ï❞ù ëíì➈ä❯è✥ç✙ï❫è❿å➈ê✧ô➣➪⑨å❛ê❇ø✓ê❯ÿ☎ê✧ÿ☎å➄û➀û✙✘➳è✥ç✙ï✩➊å➈ê✧ï❫ø➀é✒æ☎ä✥å➎↔è✧ø✁❶ï✪➶✹➌✠è✥ç✙ï➵æ✙ä✧ï❞ê✤ï✖é✗❶ï ì➄ë✹☛✧ï❯↔➇è✧ä❿å➻ù✂ø➑î➻ï✖é☎ê✤ø➀ì➈é✎ê❴✌➻ø➀é✫æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä➔ê✧æ☎å➎❶ï✌ä✥ï✖ù✂ÿ✗➊ï✖ê✛è✥ç✙ï➞ä✥ø➀ê✧ô ì➄ë➞ÿ☎é☎å✠è✧è✥å➄ø➀é☎å✑✡✙û➑ï❺ä✧ï✒✂➈ø➀ì➈é☎ê✖ð②✚✛å✑❿ô❩✝⑥æ✙ä✧ì❛æ☎å❄✂✐å✠è✥ø➑ì❛é❤ø✓ê ✡☛✘❑ë⑨å➄ä✢è✧ç✙ï î➻ì❛ê✤è➞ó❨ø➀ù✙ï➊û✠✘❺ÿ☎ê✧ï✖ù é✙ï✖ÿ✙ä❿å➄û✙✝♠é✙ï➊è④ó✇ì❛ä✧ô✫û➀ï✖å➈ä✧é☎ø➑é❼✂✫å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î✫➌❀å➄é☎ù æ✙ä✥ì✑✡☎å✑✡✙û✠✘➲è✧ç☎ï➓î❭ì✐ê④è❙ó❨ø➀ù✂ï✖û✙✘❺ÿ☎ê✧ï✖ù➷û➀ï✖å➄ä✥é✙ø➀é❼✂✫å➄û✠✂➈ì❛ä✧ø➑è✧ç✙î❂ì➄ë❨å➄é☛✘ ëíì➈ä✥î✺ð ✧✛❊✢➳✴➢♦➡✹➨✤✣●➨✦✥❵✣●➨✂✁✬➳✴➢✲☎✄➑➢♦➨ ✪♦➻❷➡▼✣●➺❩✣●➨✵✥✆✁✬➳❄✴➯❴✥✑➨✤✣●➺❩✣ ➯♦➨✞✝☎✟✪➩✹➺❖➳✄➲➑➩ ➏④ê✤ì❛û➀å➄è✧ï✖ù✿ç☎å➈é☎ù✂ó❨ä✥ø➩è✧è✧ï✖é✫❿ç✎å➄ä❿å✑↔è✥ï➊ä✛ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✿ç☎å❛ê✔✡✎ï✖ï➊é✫ï❯↔☛✝ è✧ï✖é☎ê✧ø➑ú❛ï➊û✠✘♣ê✤è✧ÿ☎ù✂ø➀ï✖ù➤ø➀é➤è✧ç☎ï❫û➀ø➩è✥ï➊ä❿å✠è✧ÿ☎ä✧ï✩➪⑨ê✧ï➊ï✟✞✑✫❜✠❖➌✵✞✑❝✫✠➄ëíì➈ä✝ä✥ï➊ú➇ø➀ï➊ó➔ê✴➶✹➌ å➄é✎ù❖ó✛å➈ê♣ì➈é✙ï➻ì➈ë➏è✧ç✙ï➻ï❞å➄ä✥û✙✘➲ê✧ÿ✗✄➊ï✖ê✥ê④ëíÿ☎û❦å➈æ✙æ✙û➀ø✠✖å✠è✥ø➑ì❛é☎ê➳ì➄ë✇é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô✂ê✹✞✑✫✙✆✠⑥ð✎☞✇ì❛î❭æ✎å➄ä❿å✠è✧ø➀ú➈ï➳ï❯↔✂æ◆ï➊ä✥ø➑î➻ï✖é❛è❿ê❨ì➈é✺ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✢ì➈ë ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û✇ç✎å➄é☎ù✂ó❨ä✥ø➑è✤è✧ï✖é ù✂ø✠✂➈ø➑è✥ê✒å➈ä✧ï✶ä✧ï✖æ✎ì❛ä✤è✥ï✖ù➷ø➀é❤þ➇ï✒❶è✧ø➀ì➈é✢➏❺➏❇➏↔ð ã✛ç✙ï✒✘➘ê✧ç✙ì✠ó▼è✧ç✎å✠è✢é☎ï➊ÿ✙ä❿å➄û♣é✙ï❶è④ó➵ì➈ä✥ô✂ê➻è✧ä❿å➄ø➀é✙ï✖ù✻ó❨ø➩è✥ç✾â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝ ✚✛å➈ê✧ï✖ù✷✗❀ï✖å➈ä✧é✙ø➀é❼✂✾æ✎ï✖ä✤ëíì❛ä✧î ✡◆ï❶è✤è✥ï➊ä è✧ç✎å➄é➶å➈û➑û➻ì➈è✧ç✙ï✖ä❺î➻ï❶è✥ç✙ì✂ù✙ê è✧ï❞ê④è✥ï✖ù ç✙ï✖ä✧ï➻ì➈é❺è✧ç☎ï➽ê✧å➈î❭ï➓ù✙å➄è✥å☎ð➻ã✛ç☎ï➚✡◆ï✖ê✤è➞é✙ï✖ÿ✙ä✥å➈û❣é✙ï❶è④ó➵ì➈ä✥ô✂ê✄➌ ➊å➈û➑û➀ï✖ù✟☞✇ì➈é➇ú➈ì❛û➑ÿ✙è✧ø➀ì➈é☎å➈û♣ñ➔ï➊è④ó✇ì❛ä✧ô✂ê✒➌➉å➄ä✥ï❺ù✙ï✖ê✧ø✙✂❛é✙ï✖ù✻è✧ì û➑ï❞å➄ä✥é❍è✧ì ï❯↔➇è✥ä✥å➎↔è✢ä✥ï➊û➀ï➊ú✠å➈é❛è✢ëíï✖å➄è✧ÿ✙ä✥ï✖ê✺ù✂ø➀ä✧ï★↔è✥û✙✘ ëíä✥ì➈î æ✙ø✙↔✂ï➊û➤ø➑î➓å✑✂➈ï✖ê➛➪⑨ê✧ï➊ï þ➇ï★↔è✧ø➀ì➈é✫➏❺➏❇➶↔ð ý♣é✙ï➔ì➄ë✟è✥ç✙ï➉î➻ì❛ê✤è✇ù✂ø✯➣❶ÿ✙û➑è✇æ☎ä✧ì➎✡✙û➑ï✖î➓ê➏ø➀é➓ç☎å➄é☎ù✙ó❨ä✧ø➑è✧ø➀é❼✂➞ä✥ï✒➊ì✑✂❄✝ é✙ø➑è✧ø➀ì➈é✏➌➇ç✙ì✠ó➵ï➊ú❛ï➊ä★➌➈ø✓ê❫é☎ì➄è✛ì➈é✙û✠✘❭è✧ì✒ä✥ï✒➊ì✑✂❛é✙ø✙➽✖ï➔ø➀é☎ù✂ø➀ú✐ø✓ù✂ÿ☎å➈û✈❿ç✎å➄ä❿å✑✹✝ è✧ï✖ä✥ê✒➌✈✡✙ÿ✂è✌å➄û✓ê✤ì➻è✥ì✢ê✧ï➊æ☎å➈ä✥å➄è✧ï✌ì➈ÿ✙è✞❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê❨ëíä✧ì❛î✴è✥ç✙ï➊ø➀ä♣é☎ï➊ø✠✂➈ç✦✝ ✡◆ì➈ä❿ê❨ó❨ø➩è✥ç✙ø➀é✺è✧ç✙ï✌ó➵ì➈ä❿ù✶ì❛ä➔ê✤ï✖é✐è✧ï➊é✥❶ï✑➌◆å❭æ✙ä✧ì✦➊ï✖ê✥ê✛ô✐é☎ì✠ó❨é✺å❛ê❨ê✤ï✒✂❄✝ î➻ï➊é✐è✥å➄è✧ø➀ì➈é✝ð✌ã✛ç✙ï✒è✧ï★❿ç✙é✙ø✁➍❛ÿ☎ï✒ëíì❛ä➳ù✂ì❛ø➑é✗✂➽è✧ç☎ø➀ê♣è✧ç✎å✠è♣ç✎å➈ê④✡✎ï★❶ì➈î➻ï è✧ç☎ï ☛✧ê✤è✥å➈é☎ù✙å➈ä✥ù✌✿ø✓ê➉✖å➄û➀û➑ï❞ù✠✄➑➳☛✡✦➡▼✣✁➩✹➺❩✣✂❄✌☞✎✍♦➳✄➡▼✭✏✝✇➳✿✥❄➲➵➳✄➨✗➺r➢♦➺❩✣ ➯♦➨☎ð✫➏⑥è ❶ì❛é☎ê✧ø➀ê✤è✥ê❖ø➑é②✂❛ï➊é✙ï✖ä✥å➄è✧ø➀é❼✂✻å❍û✓å➄ä❘✂➈ï é➇ÿ✙î➑✡✎ï✖ä❖ì➈ë➽æ✎ì➈è✧ï➊é✐è✥ø➀å➈û➑❶ÿ✙è✥ê ✡◆ï❶è④ó➵ï➊ï➊é ❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê➉ÿ☎ê✧ø➑é❼✂✺ç✙ï✖ÿ✙ä✥ø➀ê✤è✧ø✁✒ø➀î➓å❄✂❛ï❙æ✙ä✥ì✦❶ï❞ê✧ê✧ø➑é✗✂✶è✥ï✒❿ç✦✝ é✙ø✁➍✐ÿ✙ï✖ê✒➌✂å➄é☎ù➓ê✧ÿ❼✡☎ê✧ï✒➍✐ÿ✙ï✖é❛è✥û✙✘➓ê✧ï➊û➀ï✒❶è✧ø➀é❼✂✌è✧ç☎ï④✡◆ï✖ê✤è✎➊ì➈î➉✡☎ø➑é☎å➄è✧ø➀ì➈é➓ì➈ë ❶ÿ✙è✥ê✔✡☎å❛ê✤ï❞ù➽ì➈é✿ê❺➊ì➈ä✥ï✖ê➃✂➈ø➀ú➈ï✖é➽ëíì➈ä✛ï❞å✑❿ç➙✖å➄é☎ù✙ø➀ù✙å➄è✧ït❿ç☎å➄ä❿å✑❶è✧ï✖ä➃✡☛✘ è✧ç☎ï➓ä✧ï★❶ì✑✂❛é✙ø✠➽➊ï➊ä❞ð➓➏➠é➷ê✤ÿ✥❿ç❺å✿î➻ì➇ù✙ï➊û✶➌☛è✧ç✙ï➽å➎✄➊ÿ✙ä✥å➎❯✘✺ì➄ë✇è✧ç✙ï➓ê❺✘✂ê❅✝ è✧ï✖î ù✂ï➊æ◆ï➊é☎ù☎ê✇ÿ☎æ✎ì❛é➓è✧ç✙ï✞➍✐ÿ☎å➄û➀ø➩è❅✘❭ì➄ë☛è✥ç✙ï④➊ÿ✂è✥ê➃✂➈ï✖é✙ï➊ä❿å✠è✥ï✖ù➚✡☛✘❭è✧ç✙ï ç✙ï✖ÿ✙ä✧ø✓ê✤è✧ø✁➊ê✒➌❫å➄é☎ù❤ì➈é❤è✧ç✙ï✺å❄✡✙ø➀û➀ø➩è❅✘ ì➈ë➔è✧ç✙ï✺ä✥ï✒➊ì✑✂➈é☎ø✙➽✖ï➊ä➞è✥ì ù✙ø➀ê✤è✧ø➀é✦✝ ✂➈ÿ☎ø➀ê✧ç①❶ì➈ä✥ä✥ï✒↔è✥û✙✘❺ê✤ï✒✂➈î➻ï➊é✐è✧ï❞ù➛❿ç✎å➄ä❿å✑↔è✥ï➊ä❿ê♣ëíä✧ì❛î❋æ✙ø➀ï✒➊ï✖ê✌ì➈ë✔❿ç☎å➈ä❇✝ å✑❶è✧ï✖ä✥ê✒➌➔î✒ÿ☎û➩è✥ø➑æ✙û➀ï➛❿ç✎å➄ä❿å✑↔è✥ï➊ä❿ê✄➌❨ì❛ä✶ì➈è✧ç✙ï✖ä✧ó❨ø✓ê✤ï❖ø➑é✗➊ì➈ä✥ä✧ï★↔è✥û✙✘➘ê✤ï✒✂❄✝ î➻ï➊é✐è✧ï❞ù ❿ç☎å➄ä❿å✑❶è✧ï➊ä❿ê✖ð❭ã❇ä✥å➈ø➑é☎ø➑é❼✂✫å✿ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï✖ä➳è✧ì✫æ◆ï➊ä✧ëíì➈ä✥î è✧ç✙ø✓ê è✥å❛ê✤ô➤æ◆ì❛ê✧ï✖ê❇å➔î➓å✓④ì❛ä❳❿ç☎å➈û➑û➀ï➊é❼✂❛ï❷✡✎ï★➊å➈ÿ☎ê✤ï✇ì➄ë✂è✥ç✙ï✇ù✂ø✯➣❶ÿ✙û➑è❅✘✌ø➀é➉❶ä✥ï❯✝ å✠è✥ø➑é✗✂✿å✿û✓å❄✡◆ï➊û➀ï✖ù ù✙å✠è❿å❄✡☎å❛ê✤ï❙ì➄ë✇ø➀é✗➊ì➈ä✥ä✧ï★↔è✧û✠✘✫ê✤ï✒✂➈î➻ï➊é✐è✥ï✖ù➔❿ç✎å➄ä❿å✑✹✝ è✧ï✖ä✥ê✖ð✛ã✛ç✙ï➞ê✤ø➀î➻æ✙û➀ï✖ê✤è♣ê✤ì❛û➑ÿ✙è✧ø➀ì➈é✆❶ì❛é☎ê✤ø✓ê✤è✥ê❨ø➀é✫ä✧ÿ☎é✙é✙ø➀é❼✂➻è✧ç✙ï✒ø➑î➓å❄✂❛ï✖ê ì➄ë❀❿ç✎å➄ä❿å✑↔è✥ï➊ä➵ê④è✥ä✧ø➀é❼✂❛ê➏è✧ç✙ä✥ì➈ÿ✗✂➈ç➻è✧ç☎ï➳ê✧ï✄✂❛î❭ï✖é✐è✧ï➊ä★➌➇å➄é☎ù➻è✥ç✙ï➊é✢î➓å➄é✦✝ ÿ☎å➈û➑û✠✘➓û➀å✑✡✎ï✖û➑ø➀é❼✂➓å➄û➀û◆è✧ç✙ï✌❿ç☎å➈ä✥å➎↔è✥ï➊ä❫ç☛✘➇æ◆ì➄è✧ç☎ï✖ê✧ï✖ê✖ð❨→➉é✂ëíì➈ä✧è✧ÿ☎é☎å✠è✥ï➊û✠✘✑➌ é✙ì➈è➉ì➈é✙û✠✘✶ø✓ê❨è✥ç✙ø✓ê➉å➄é✫ï✄↔➇è✧ä✥ï➊î➻ï➊û✠✘➽è✥ï✖ù✂ø➀ì➈ÿ☎ê♣å➄é☎ù✫➊ì❛ê✤è✧û✠✘➽è✥å❛ê✤ô✇➌✎ø➩è➉ø✓ê å➄û✓ê✧ì✶ù✂ø✯➵➊ÿ✙û➑è♣è✥ì✶ù✂ì➽è✥ç✙ï✒û✓å❄✡◆ï➊û➀ø➀é❼✂➙➊ì➈é☎ê✧ø➀ê✤è✧ï✖é✐è✧û✠✘➈ð✬✜☎ì➈ä➳ï❯↔✙å➄î➻æ✙û➀ï✑➌ ê✧ç✙ì➈ÿ✙û✓ù✺è✥ç✙ï✒ä✥ø✙✂❛ç✐è➉ç☎å➄û➑ë➏ì➄ë❫å ❶ÿ✂è➳ÿ✙æ ❝➣✡◆ï✒û✓å❄✡◆ï➊û➀ï✖ù❖å➈ê➉å➔➾➞ì➈ä➳å➈ê å é✙ì➈é❼✝❖❿ç☎å➈ä✥å➎↔è✥ï➊ä✒✑ ê✤ç✙ì❛ÿ✙û✓ù è✥ç✙ï✺ä✧ø✠✂➈ç✐è➻ç☎å➄û➑ë♣ì➈ë➳å➒❶ÿ✂è➓ÿ✙æ ✺➒✡✎ï û✓å❄✡◆ï➊û➀ï✖ù✺å➈ê❨å❑❜✓✑ ã✛ç✙ï①➞☎ä❿ê④è❖ê✧ì➈û➀ÿ✂è✥ø➑ì❛é✏➌➞ù✂ï✖ê❘❶ä✥ø✠✡✎ï❞ù✾ø➀é✹þ➇ï★↔è✧ø➀ì➈é✟✣ ➊ì➈é☎ê✧ø✓ê④è❿ê✫ø➑é è✧ä❿å➄ø➀é✙ø➀é❼✂✫è✥ç✙ï✢ê❺✘✂ê④è✥ï➊î å✠è❙è✧ç✙ï✢û➑ï✖ú➈ï➊û✇ì➄ë➔ó❨ç✙ì❛û➑ï✢ê④è✥ä✧ø➀é❼✂❛ê✒ì➄ë✬❿ç☎å➈ä❇✝ å✑❶è✧ï✖ä✥ê✒➌❦ä✥å➄è✧ç✙ï✖ä➞è✧ç✎å➄é❤å➄è➞è✧ç✙ï✫❿ç☎å➈ä✥å➎↔è✧ï✖ä➞û➀ï➊ú➈ï✖ûüð❺ã✛ç✙ï✶é✙ì➈è✧ø➀ì➈é ì➈ë â➳ä❿å➈ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù✒✗✝ï❞å➄ä✥é✙ø➀é❼✂✛➊å➈é➓✡✎ï✇ÿ☎ê✧ï✖ù➳ëíì➈ä❇è✧ç☎ø➀ê❀æ✙ÿ✙ä✥æ✎ì✐ê✤ï❛ð❇ã✛ç✙ï ê❺✘➇ê✤è✧ï✖î ø➀ê❀è✧ä❿å➄ø➀é✙ï❞ù➤è✧ì➳î➻ø➑é✙ø➀î➻ø✙➽✖ï➵å➈é✒ì✠ú❛ï➊ä❿å➄û➀û➄û➑ì✐ê✧ê❀ëíÿ✙é✗❶è✧ø➀ì➈é❙ó❨ç✙ø✠❿ç î➻ï✖å❛ê✤ÿ✙ä✥ï✖ê✝è✧ç✙ï✇æ✙ä✥ì✑✡☎å✑✡✙ø➑û➀ø➑è❅✘♣ì➈ë☎å➄é➞ï✖ä✧ä✥ì➈é✙ï✖ì➈ÿ☎ê❀å➄é☎ê✧ó✇ï✖ä✖ð❯þ➇ï✒❶è✧ø➀ì➈é➓✣ ï❯↔✂æ✙û➀ì➈ä✥ï✖ê❇ú✠å➄ä✥ø➑ì❛ÿ☎ê❇ó➵å✪✘✂ê☛è✥ì♣ï✖é☎ê✧ÿ✙ä✧ï❫è✧ç☎å➄è❦è✧ç✙ï➵û➑ì✐ê✧ê❀ëíÿ✙é✗❶è✧ø➀ì➈é❭ø➀ê❦ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➎➌✠å➄é☎ù➳è✥ç✙ï➊ä✥ï❶ëíì❛ä✧ï➏û➀ï➊é✎ù✙ê✝ø➑è✥ê✧ï➊û➑ë✂è✧ì❨è✥ç✙ï❫ÿ✎ê✤ï✇ì➄ë☎â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝ ✚✛å➈ê✧ï✖ù❖✗❀ï✖å➄ä✥é✙ø➀é❼✂➓î➻ï❶è✥ç✙ì✂ù✙ê➊ð➳þ➇ï✒❶è✧ø➀ì➈é✆✣✹ø➀é✐è✧ä✥ì✂ù✂ÿ✗❶ï❞ê❨è✥ç✙ï✒ÿ☎ê✧ï➞ì➈ë ù✂ø➀ä✧ï★↔è✥ï✖ù å✑✄✘✦❶û➀ø✠➵✂❛ä✥å➈æ✙ç☎ê✌ó❨ç☎ì❛ê✧ï✶å➄ä✴➊ê➓✖å➄ä✥ä❺✘❺é➇ÿ✙î➻ï➊ä✥ø✠✖å➄û❫ø➀é✂ëíì❛ä❇✝ î➓å✠è✥ø➑ì❛é å➈ê✒å➲ó➵å✪✘❺è✧ì➲ä✥ï➊æ☎ä✧ï❞ê✤ï✖é❛è✌è✥ç✙ï✢å➈û➩è✥ï➊ä✥é☎å✠è✥ø➑ú❛ï➽ç☛✘➇æ✎ì➈è✧ç✙ï❞ê✤ï❞ê✄➌ å➄é✎ù✶ø➀é✐è✧ä✥ì➇ù✙ÿ✗❶ï❞ê➵è✧ç✙ï✌ø✓ù✂ï✖å➻ì➈ë❦â➤ã❨ñ✒ð ã✛ç✙ï➞ê✤ï★❶ì❛é☎ù✺ê✤ì❛û➑ÿ✂è✥ø➑ì❛é✫ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù✿ø➀é➲þ➇ï✒❶è✧ø➀ì➈é✫✣✬➏❺➏➵ø✓ê✛è✥ì➓ï➊û➀ø➑î➚✝ ø➀é☎å✠è✥ï✺ê✧ï✄✂❛î➻ï➊é✐è✥å➄è✧ø➀ì➈é❤å➈û➩è✥ì✑✂➈ï➊è✧ç✙ï✖ä✖ð❑ã✛ç✙ï✺ø➀ù✂ï❞å❖ø✓ê❭è✧ì ê✧ó✇ï✖ï➊æ❤è✧ç✙ï ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï✖ä➏ì✠ú➈ï✖ä➏ï➊ú❛ï➊ä❘✘❙æ◆ì❛ê✥ê✧ø✙✡✙û➀ï➉û➀ì✦➊å➄è✧ø➀ì➈é➓ì➈é➽è✧ç✙ï➳ø➑é☎æ✙ÿ✂è✛ø➑î➓å❄✂❛ï✑➌ å➄é✎ù➞è✥ì➳ä✥ï➊û✠✘➞ì❛é➞è✧ç☎ï ☛❺❿ç☎å➈ä✥å➎↔è✧ï✖ä❯ê✧æ◆ì➄è✤è✥ø➑é✗✂✌♣æ✙ä✧ì❛æ✎ï✖ä✤è❅✘✌ì➈ë☎è✥ç✙ï❨ä✧ï★✹✝ ì✑✂❛é✙ø✠➽➊ï➊ä★➌❞øüð ï➈ð❇ø➩è❿ê❀å❄✡☎ø➑û➀ø➩è❅✘➳è✥ì✬❶ì❛ä✧ä✥ï✒❶è✧û✠✘➉ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï❣å❨ó➵ï➊û➀û✙✝❖➊ï➊é✐è✧ï✖ä✧ï❞ù ❿ç☎å➈ä✥å➎↔è✧ï✖ä✌ø➑é❤ø➩è❿ê➞ø➀é✙æ✙ÿ✂è➑➞☎ï➊û✓ù❢➌❦ï➊ú❛ï➊é ø➀é➷è✧ç✙ï✢æ✙ä✧ï❞ê✤ï✖é✗❶ï➓ì➈ë➔ì➄è✥ç✙ï➊ä ❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê✎✡◆ï✖ê✧ø➀ù✙ï✖ê➔ø➑è✒➌✎ó❨ç✙ø➀û➑ï➞ä✥ï✓④ï★↔è✥ø➑é❼✂➓ø➀î➓å❄✂❛ï✖ê✛➊ì➈é✐è✥å➈ø➑é✙ø➀é❼✂➻é✙ì ❶ï✖é✐è✧ï➊ä✥ï✖ù➒❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê❄✞✑❆✓✠❖➌ ✞✑✗✔✆✠♠ð➻ã✛ç✙ï➽ê✧ï✒➍✐ÿ✙ï➊é✥❶ï➻ì➄ë➵ä✧ï★❶ì✑✂❛é✙ø✠➽➊ï➊ä ì➈ÿ✙è✧æ✙ÿ✂è❿ê➞ì➎✡✂è✥å➈ø➑é✙ï❞ù ✡☛✘❺ê✧ó✇ï✖ï➊æ✙ø➀é❼✂✺è✧ç☎ï➽ä✧ï★❶ì➎✂➈é✙ø✠➽➊ï✖ä✌ì✠ú➈ï➊ä➤è✥ç✙ï➽ø➀é✦✝ æ✙ÿ✂è✇ø➀ê❯è✥ç✙ï➊é➻ëíï❞ù✒è✥ì➞å➞â➳ä✥å➈æ✙ç➻ã❀ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä❣ñ➔ï➊è④ó✇ì❛ä✧ô➞è✧ç☎å➄è❣è✥å➄ô❛ï✖ê û➀ø➑é❼✂❛ÿ✙ø✓ê④è✥ø✠➑❶ì➈é✎ê④è✥ä✥å➈ø➑é✐è✥ê➔ø➑é✐è✧ì✢å✑✄➊ì➈ÿ✙é✐è♣å➈é☎ù✫➞☎é✎å➄û➀û✙✘✿ï❯↔➇è✧ä❿å✑❶è✥ê➔è✧ç✙ï î➻ì❛ê✤è➔û➑ø➀ô➈ï✖û✙✘✢ø➑é✐è✥ï➊ä✥æ✙ä✧ï➊è✥å➄è✧ø➀ì➈é✝ð➵ã✛ç✙ø➀ê➳â➤ã❨ñ ø✓ê➔ê✤ì❛î➻ï➊ó❨ç☎å➄è➉ê✤ø➀î➻ø➑û✓å➄ä è✧ì õ➔ø✓ù✙ù✂ï✖é❑ö➲å➈ä✧ô❛ì✠ú❖ö✫ì✂ù✂ï➊û✓ê➐➪⑨õ➉ö❖ö➔➶✹➌❦ó❨ç✙ø✁❿ç î➓å➄ô❛ï✖ê✌è✥ç✙ï✺å➄æ✦✝ æ✙ä✥ì❛å➎❿ç✶ä✥ï➊î➻ø➑é☎ø➀ê❘❶ï✖é❛è❨ì➈ë❇è✥ç✙ï➓❶û✓å➈ê✥ê✤ø✁➊å➈û☛ê✤æ◆ï➊ï★❿ç✿ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é ✞✑❆✺✬✠✶➌ ✞✑❆❀✠⑥ð ✎ç✙ø➀û➀ï❖è✥ç✙ø➀ê✿è✥ï✒❿ç✙é☎ø✠➍✐ÿ✙ï➷ó➵ì➈ÿ✙û✓ù❭✡◆ï ➍✐ÿ✙ø➑è✧ï➷ï✄↔➇æ◆ï➊é✎ê✤ø➀ú➈ï ø➑é è✧ç☎ï✩✂➈ï➊é☎ï➊ä❿å➄û❼➊å❛ê✤ï➎➌✠è✧ç☎ï❨ÿ☎ê✤ï❨ì➈ë✏☞✇ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✂ñ➔ï✖ÿ✙ä❿å➄û☎ñ➉ï❶è④ó➵ì➈ä✥ô➇ê î➓å➄ô❛ï✖ê✝ø➑è❇æ✎å➄ä✧è✧ø✁❶ÿ✙û✓å➄ä✥û✙✘➤å✠è✧è✧ä❿å✑❶è✧ø➀ú➈ï❹✡◆ï✒✖å➄ÿ☎ê✧ï❫ø➑è❯å➈û➑û➀ì✠ó➔ê❀ê✧ø✙✂❛é✙ø✙➞✥➊å➈é❛è ê✥årú✐ø➀é❼✂✐ê✇ø➀é✆❶ì❛î❭æ☎ÿ✂è✥å➄è✧ø➀ì➈é☎å➈û✏❶ì❛ê✤è✖ð ✔✛ ➠❅✲✙➯✫✯✴➢✲ ✲✕✟ ➦❼➡❺➢✣●➨✇➢✫✯❪✲✙➳✖✝☎✟✪➩✹➺❖➳✄➲➑➩ ✕➉ê❫ê✤è✥å➄è✧ï❞ù➻ï✖å➄ä✥û➀ø➑ï✖ä✒➌➄î➻ì✐ê④è➏æ✙ä❿å✑❶è✧ø✁➊å➈û✙æ☎å✠è✧è✧ï✖ä✧é➓ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é❭ê❺✘✂ê❅✝ è✧ï✖î➓ê❙å➈ä✧ï➐❶ì➈î➻æ◆ì❛ê✧ï✖ù➷ì➈ë➔î✒ÿ✙û➑è✧ø➀æ✙û➀ï✢î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð➒✜☎ì➈ä❭ï❯↔✙å➄î➻æ✙û➀ï✑➌❣å ù✂ì✦❶ÿ☎î❭ï✖é✐è❯ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é➞ê❇✘✂ê✤è✧ï➊î ø✓ê❳➊ì➈î➻æ◆ì❛ê✧ï✖ù✌ì➈ë✎å✩➞☎ï✖û➀ù➞û➑ì✦➊å➄è✧ì❛ä✒➌ ó❨ç✙ø✁❿ç✶ï✄↔✐è✥ä✥å➎↔è❿ê❫ä✥ï✄✂❛ø➑ì❛é☎ê❫ì➈ë❀ø➀é❛è✥ï➊ä✥ï✖ê✤è✒➌✂å✌➞✎ï➊û✓ù✢ê✧ï✄✂❛î❭ï✖é✐è✧ï➊ä★➌✐ó❨ç✙ø✠❿ç ❶ÿ✙è✥ê❦è✥ç✙ï➔ø➀é✙æ✙ÿ✂è➏ø➀î➓å❄✂➈ï❨ø➀é✐è✧ì➤ø➀î➻å✑✂➈ï❞ê❦ì➄ë❢➊å➄é✎ù✂ø➀ù☎å✠è✧ï✛❿ç✎å➄ä❿å✑↔è✥ï➊ä❿ê✄➌✠å ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï✖ä✒➌➈ó❨ç✙ø✠❿ç➣➊û➀å❛ê✧ê✧ø✙➞☎ï✖ê➏å➈é☎ù➓ê❘❶ì➈ä✥ï✖ê❦ï❞å✑❿ç➣✖å➄é☎ù✂ø✓ù✙å➄è✧ï✬❿ç☎å➈ä❇✝ å✑❶è✧ï✖ä✒➌☎å➈é☎ù✺å➝❶ì➈é✐è✥ï❯↔➇è✧ÿ☎å➈û✝æ◆ì❛ê✤è❇✝⑥æ✙ä✧ì✦➊ï✖ê✥ê✤ì❛ä✒➌☛✂➈ï✖é✙ï➊ä❿å➄û➀û✠✘➣✡☎å➈ê✧ï✖ù✢ì➈é å➔ê✤è✧ì✦❿ç☎å❛ê④è✥ø✠❷✂❛ä✥å➈î➻î➻å➈ä✒➌✖ó❨ç✙ø✁❿ç➞ê✤ï✖û➑ï★↔è✥ê✝è✧ç✙ï➃✡✎ï❞ê④è❜✂➈ä❿å➄î➻î➓å✠è✥ø✠✖å➄û➀û✙✘ ❶ì❛ä✧ä✥ï✒❶è❇å➄é✎ê✤ó➵ï➊ä❇ëíä✧ì❛î❲è✧ç✙ï➵ç☛✘✐æ◆ì➄è✥ç✙ï✖ê✧ï✖ê❳✂➈ï✖é✙ï➊ä❿å✠è✥ï✖ù✌✡☛✘➳è✧ç☎ï✇ä✥ï✒➊ì✑✂❄✝ é✙ø✠➽➊ï✖ä✖ð➃➏➠é✿î➻ì✐ê④è✬✖å➈ê✧ï✖ê✒➌✂è✧ç✙ï✌ø➀é✂ëíì❛ä✧î➓å✠è✥ø➑ì❛é✆➊å➄ä✥ä✥ø➑ï❞ù➽ëíä✥ì➈îPî❭ì✂ù✂ÿ☎û➑ï è✧ì❙î➻ì➇ù✙ÿ✙û➑ï➳ø✓ê➜✡◆ï✖ê✤è✇ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✥ï✖ù➽å❛ê❹✂➈ä❿å➄æ☎ç☎ê➏ó❨ø➑è✧ç✢é✐ÿ☎î❭ï✖ä✧ø✁➊å➈û☎ø➀é✦✝ ëíì➈ä✥î➓å✠è✥ø➑ì❛é➷å➄è✤è❿å✑❿ç✙ï❞ù❖è✥ì✺è✧ç✙ï✢å➄ä✴➊ê✖ð ✜✙ì❛ä➞ï❯↔✙å➄î➻æ✙û➀ï✑➌❇è✧ç☎ï➽ì➈ÿ✂è✥æ✙ÿ✂è ì➄ë❣è✥ç✙ï❭ä✧ï★❶ì✑✂❛é✙ø✠➽➊ï➊ä➉î❭ì✂ù✂ÿ☎û➑ï➝➊å➈é↕✡✎ï❙ä✧ï✖æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù✫å➈ê➳å➄é❖å➎❯✘✦❶û➀ø✠ ✂➈ä❿å➄æ☎ç ó❨ç☎ï➊ä✥ï➓ï✖å✑❿ç å➈ä❘➵➊ì➈é✐è✥å➈ø➑é☎ê✌è✧ç✙ï✢û➀å✑✡✎ï✖û❫å➄é☎ù è✥ç✙ï✢ê❘❶ì❛ä✧ï➻ì➈ë å✢✖å➄é☎ù✂ø✓ù✙å➄è✧ï➔❿ç☎å➈ä✥å➎↔è✥ï➊ä★➌✛å➈é☎ù➘ó❨ç✙ï✖ä✧ï➲ï❞å✑❿ç❍æ☎å➄è✧ç❍ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✶å å➄û➑è✧ï✖ä✧é✎å✠è✧ø➀ú➈ï✺ø➀é✐è✧ï➊ä✥æ✙ä✥ï❶è❿å✠è✧ø➀ì➈é➘ì➈ë♣è✥ç✙ï➲ø➀é✙æ✙ÿ✂è✢ê✤è✧ä✥ø➀é❼✂☎ð❲ã➜✘➇æ✙ø✁➊å➈û➑û✠✘✑➌ ï✖å➎❿ç✢î➻ì✂ù✂ÿ✙û➀ï➤ø➀ê➔î➻å➈é➇ÿ☎å➄û➀û✙✘➓ì❛æ✂è✧ø➀î➻ø✙➽✖ï✖ù❢➌☎ì➈ä❨ê✤ì❛î➻ï❶è✧ø➀î➻ï✖ê➵è✥ä✥å➈ø➑é✙ï❞ù❢➌ ì➈ÿ✙è✥ê✧ø➀ù✂ï✶ì➄ë❨ø➩è❿ê➑➊ì➈é✐è✧ï✄↔➇è✖ð↕✜✙ì❛ä➞ï❯↔✙å➈î❭æ☎û➑ï➎➌❇è✥ç✙ï➙❿ç✎å➄ä❿å✑↔è✥ï➊ä✌ä✥ï✒➊ì✑✂❄✝ é✙ø✠➽➊ï✖ä✇ó➵ì➈ÿ✙û✓ù➵✡◆ï➉è✥ä✥å➈ø➑é☎ï✖ù➓ì❛é➽û➀å✑✡✎ï✖û➑ï❞ù➽ø➀î➓å❄✂➈ï❞ê➏ì➈ë✝æ✙ä✧ï✄✝⑥ê✧ï✄✂❛î➻ï➊é✐è✧ï❞ù ❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê✖ð➲ã✛ç✙ï✖é è✧ç✙ï✫❶ì❛î➻æ✙û➑ï➊è✧ï✶ê❺✘✂ê④è✥ï➊î÷ø✓ê❙å❛ê✧ê✧ï➊î➑✡✙û➑ï❞ù❢➌❣å➄é☎ù å➻ê✤ÿ✗✡☎ê✤ï➊è➔ì➄ë❇è✧ç✙ï✌æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê➵ì➈ë❇è✥ç✙ï✌î➻ì➇ù✙ÿ✙û➑ï❞ê✛ø➀ê➔î➻å➈é➇ÿ☎å➄û➀û✙✘✢å➈ù☛✝ ✓④ÿ☎ê✤è✧ï❞ù➓è✧ì❙î➓å♦↔✂ø➑î➻ø✠➽➊ï➉è✧ç✙ï♣ì✠ú➈ï➊ä❿å➄û➀û✙æ◆ï➊ä✧ëíì➈ä✥î➓å➄é✗➊ï➈ð❦ã✛ç✙ø➀ê✇û✓å➈ê✤è➵ê✤è✧ï➊æ ø✓ê❦ï❯↔➇è✧ä✥ï➊î➻ï➊û✠✘➤è✧ï❞ù✂ø➀ì➈ÿ☎ê✒➌✠è✧ø➀î➻ï❯✝r❶ì❛é☎ê✤ÿ☎î❭ø➀é❼✂✥➌➈å➄é☎ù❭å➄û➀î➻ì❛ê✤è❨➊ï➊ä✧è✥å➈ø➑é✙û✠✘ ê✧ÿ❼✡✎ì❛æ✂è✧ø➀î➓å➄û♠ð ✕ ✡✎ï➊è✤è✥ï➊ä♣å➈û➩è✥ï➊ä✥é☎å✠è✥ø➑ú❛ï✌ó✇ì❛ÿ✙û✓ù➙✡◆ï➞è✥ì➽ê✤ì❛î➻ï➊ç✙ì✠ó✾è✧ä❿å➄ø➀é✿è✧ç✙ï➞ï✖é✦✝ è✧ø➀ä✥ï✇ê❺✘➇ê✤è✧ï✖îòê✤ì➳å➈ê✝è✧ì➳î❭ø➀é✙ø➀î➻ø✙➽✖ï➵å✬✂➈û➀ì✑✡☎å➈û➈ï➊ä✥ä✥ì➈ä❀î➻ï❞å➈ê✧ÿ✙ä✧ï➵ê✤ÿ✥❿ç✒å➈ê è✧ç☎ï✌æ✙ä✧ì➎✡☎å❄✡☎ø➑û➀ø➩è❅✘✶ì➄ë❜❿ç✎å➄ä❿å✑↔è✥ï➊ä✛î➻ø➀ê❘❶û✓å➈ê✥ê✧ø➟➞✥✖å✠è✥ø➑ì❛é☎ê✛å✠è✛è✧ç✙ï➞ù✙ì☛➊ÿ✦✝ î➻ï➊é✐è❣û➑ï✖ú➈ï✖ûüð❳➏④ù✂ï✖å➈û➑û✠✘✑➌✠ó➵ï✇ó➵ì➈ÿ☎û➀ù✒ó➵å➈é❛è❯è✧ì④➞☎é☎ù❭å④✂➈ì➇ì✂ù✌î➻ø➑é✙ø➀î✒ÿ☎î ì➄ë❫è✧ç✙ø✓êt✂➈û➀ì✑✡☎å➈û❇û➀ì❛ê✥ê♣ëíÿ✙é✗❶è✧ø➀ì➈é ó❨ø➩è✥ç❖ä✥ï✖ê✧æ✎ï★↔è➳è✧ì✫å➄û➀û❯è✧ç✙ï❭æ☎å➄ä❿å➄î➚✝ ï❶è✥ï➊ä❿ê➳ø➑é❺è✧ç✙ï➽ê❺✘✂ê④è✥ï➊î✺ð➉➏⑥ë✇è✧ç✙ï➓û➀ì❛ê✥ê♣ëíÿ✙é✗❶è✧ø➀ì➈é▲❉✴î➻ï✖å❛ê✤ÿ☎ä✧ø➀é❼✂✢è✧ç✙ï æ◆ï➊ä✧ëíì➈ä✥î➻å➈é✗❶ï✩✖å➄é➚✡✎ï➔î➓å❛ù✂ï❨ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï❨ó❨ø➑è✧ç➻ä✥ï✖ê✧æ✎ï★↔è❣è✥ì➤è✧ç✙ï ê❺✘➇ê✤è✧ï✖î❁ ê♣è✧ÿ☎é☎å❄✡✙û➀ï➓æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê✒❂➌✝ó➵ï➚✖å➄é➛➞☎é☎ù å✢û➀ì✦➊å➈û❯î➻ø➀é✦✝ ø➀î✒ÿ✙î ì➈ë❇❉ ÿ☎ê✧ø➑é❼✂➷â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù▲✗✝ï❞å➄ä✥é✙ø➑é✗✂☎ð❺õ➉ì✠ó✇ï✖ú➈ï✖ä✒➌❦å✠è
2Fs-s 4 OvE2EEE7Xs LEO AEF 1ii8 first glance,it appears that the sheer size and complexity tion system is best represented by graphs with numerical of the system would make this intractable. information attached to the arcs.In this case,each module, To ensure that the global loss function EP(ZP'W)is dif- called a Graph Transformer,takes one or more graphs as ferentiable,the overall system is built as a feed-forward net- input,and produces a graph as output.Networks of such work of differentiable modules.The function implemented modules are called Graph Transformer Networks (GTN). by each module must be continuous and differentiable alE Sections IV,VI and VIII develop the concept of GTNs, most ererywhere with respect to the internal parameters of and show that Gradient-Based Learning can be used to the module (e.g.the weights of a Neural Net character rec- train all the parameters in all the modules so as to mini- ognizer in the case of a character recognition module),and mize a global loss function.It may seem paradoxical that with respect to the module's inputs.If this is the case,a gradients can be computed when the state information is simple generalization of the well-known back-propagation represented by essentially discrete obfects such as graphs, procedure can be used to efficiently compute the gradients but that difficulty can be circumvented,as shown later. of the loss function with respect to all the parameters in the system [22].For example,let us consider a system II.Y ON VOLUTIONAL NEURAL NETWORKVFOR built as a cascade of modules,each of which implements a IVOLATED Y HARACTER RECOGNITION function Xn 8 Fn(Wn'Xn-1),where Xn is a vector rep- The ability of multi-layer networks trained with gradi- resenting the output of the module,Wn is the vector of ent descent to learn complex,high-dimensional,non-linear tunable parameters in the module (a subset of W),and mappings from large collections of examples makes them Xn-1 is the module's input vector (as well as the previous obvious candidates for image recognition tasks.In the tra- module's output vector).The input X-to the first module ditional model of pattern recognition,a hand-designed fea- is the input pattern Zp.If the partial derivative of Ep with ture extractor gathers relevant information from the input respect to Xn is known,then the partial derivatives of Ep and eliminates irrelevant variabilities.A trainable classifier with respect to Wn and Xn-1 can be computed using the then categorizes the resulting feature vectors into classes. backward recurrence In this scheme,standard,fully-connected multi-layer net- 8EP 8 8 W w(W'Xn-1) EP works can be used as classifiers.A potentially more inter- esting scheme is to rely on as much as possible on learning 8EP 8F EP in the feature extractor itself.In the case of character -8 8Xn-1 (W'X-1) Xn (4) recognition,a network could be fed with almost raw in- puts (e.g.size-normalized images).While this can be done where(W)is the Jacobian of F with respect to with an ordinary fully connected feed-for ward network with W evaluated at the point (Wn'Xn-1),andWXn-) some success for tasks such as character recognition,there is the Jacobian of F with respect to X.The Jacobian of are problems. a vector function is a matrix containing the partial deriva- Firstly,typical images are large,often with several hun- tives of all the outputs with respect to all the inputs. dred variables (pixels).A fully-connected first layer with, The first equation computes some terms of the gradient say one hundred hidden units in the first layer,would al- of EP(W),while the second equation generates a back-ready contain several tens of thousands of weights.Such ward recurrence,as in the well-known back-propagation a large number of parameters increases the capacity of the procedure for neural networks.We can average the gradi- system and therefore requires a larger training set.In ad- ents over the training patterns to obtain the full gradient.dition,the memory requirement to store so many weights It is interesting to note that in many instances there is may rule out certain hardware implementations.But,the no need to explicitly compute the Jacobian matrix.The main deficiency of unstructured nets for image or speech above formula uses the product of the Jacobian with a vec- applications is that they have no built-in invariance with tor of partial derivatives,and it is often easier to compute respect to translations,or local distortions of the inputs. this product directly without computing the Jacobian be- Before being sent to the fixed-size input layer of a neural forehand.In By analogy with ordinary multi-layer neural net,character images,or other 2D or 1D signals,must be networks,all but the last module are called hidden layers approximately size-normalized and centered in the input because their outputs are not observable from the outside. field.Unfortunately,no such preprocessing can be perfect: more complex situations than the simple cascade of mod- handwriting is often normalized at the word level,which ules described above,the partial derivative notation be- can cause size,slant,and position variations for individual comes somewhat ambiguous and awkward.A completely characters.This,combined with variability in writing style, rigorous derivation in more general cases can be done using will cause variations in the position of distinctive features Lagrange functions [20],[21],[22]. in input obbects.In principle,a fully-connected network of Traditional multi-layer neural networks are a special case sufficient size could learn to produce outputs that are in- of the above where the state information xn is represented variant with respect to such variations.However,learning with fixed-sized vectors,and where the modules are al- such a task would probably result in multiple units with ternated lavers of matrix multiplications (the weights)and similar weight patterns positioned at various locations in component-wise sigmoid functions (the neurons).However, the input so as to detect distinctive features wherever they as stated ear lier,the state information in complex recogni- appear on the input.Learning these weight configurations
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ➞☎ä❿ê④è✞✂➈û✓å➄é✗➊ï✑➌◆ø➩è✌å➄æ☎æ✎ï❞å➄ä❿ê❨è✧ç✎å✠è➳è✧ç✙ï➻ê✤ç☎ï➊ï➊ä➤ê✧ø✙➽✖ï❙å➄é✎ù↕❶ì❛î❭æ☎û➑ï✄↔➇ø➑è❅✘ ì➄ë❯è✧ç☎ï➞ê❇✘✂ê✤è✧ï➊î✴ó✇ì❛ÿ✙û➀ù✢î➓å➄ô❛ï♣è✥ç✙ø✓ê❨ø➑é✐è✥ä✥å➎↔è✥å✑✡✙û➀ï➈ð ã❀ì➞ï✖é☎ê✧ÿ✙ä✧ï✛è✥ç☎å✠è✇è✧ç✙ï✬✂➈û➀ì✑✡☎å➈û✙û➑ì✐ê✧ê❣ëíÿ✙é✗↔è✥ø➑ì❛é✘❉❊✸❼➪❩✾❅✸ ❁❃❂➶❦ø✓ê❫ù✂ø➑ë●✝ ëíï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➎➌❞è✧ç✙ï➏ì✠ú❛ï➊ä❿å➄û➀û➄ê❇✘✂ê✤è✧ï✖î✾ø✓ê✏✡✙ÿ☎ø➑û➑è❯å❛ê❀å✛ëíï✖ï✖ù☛✝♠ëíì➈ä✥ó➵å➈ä✥ù♣é✙ï➊è❇✝ ó➵ì➈ä✥ô➓ì➄ë❣ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï➤î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð➏ã✛ç✙ï➤ëíÿ✙é✗❶è✧ø➀ì➈é✺ø➀î❭æ☎û➑ï✖î❭ï✖é✐è✧ï✖ù ✡☛✘✫ï✖å➎❿ç❖î➻ì✂ù✂ÿ✙û➀ï❭î❙ÿ☎ê✤è✞✡◆ï➵❶ì➈é✐è✥ø➑é➇ÿ✙ì❛ÿ☎ê♣å➈é☎ù ù✂ø➟➘✟ï➊ä✥ï➊é✐è✥ø➀å✑✡✙û➑ï✆➢✲✭ ➲➵➯✪➩✹➺❹➳☛✍♦➳❯➡✒✟♦➻❳➥✗➳❯➡❘➳➏ó❨ø➑è✧ç➓ä✥ï✖ê✧æ✎ï★↔è❣è✥ì✌è✧ç✙ï➉ø➑é✐è✧ï✖ä✧é✎å➄û✙æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê❯ì➈ë è✧ç☎ï❨î❭ì✂ù✂ÿ☎û➑ï➓➪⑨ï➈ð ✂☎ð❯è✧ç✙ï❨ó➵ï➊ø✠✂➈ç✐è❿ê❦ì➄ë◆å✌ñ➉ï➊ÿ✙ä❿å➄û✙ñ➉ï❶è➜❿ç☎å➈ä✥å➎↔è✥ï➊ä❦ä✧ï★✹✝ ì✑✂❛é✙ø✠➽➊ï➊ä❫ø➑é➓è✥ç✙ï④✖å➈ê✧ï➔ì➄ë❀å➉❿ç☎å➄ä❿å✑❶è✧ï➊ä❣ä✥ï✒➊ì✑✂❛é✙ø➩è✥ø➑ì❛é➓î➻ì➇ù✙ÿ✙û➑ï✪➶✹➌➇å➄é☎ù ó❨ø➑è✧ç ä✧ï❞ê✤æ◆ï✒❶è♣è✥ì✶è✥ç✙ï➻î❭ì✂ù✂ÿ☎û➑ï✗❁ ê➳ø➀é✙æ✙ÿ✂è❿ê➊ð➓➏⑥ë✇è✧ç✙ø✓ê➤ø✓ê➉è✥ç✙ï➝➊å➈ê✧ï✑➌✝å ê✧ø➑î➻æ✙û➀ï➵✂➈ï✖é✙ï➊ä❿å➄û➀ø✙➽❞å✠è✥ø➑ì❛é❖ì➄ë➵è✧ç☎ï➽ó✇ï✖û➑û✙✝⑥ô✐é☎ì✠ó❨é➛✡☎å➎❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é æ✙ä✥ì✦❶ï✖ù✙ÿ✙ä✧ï✞➊å➈é ✡◆ï♣ÿ☎ê✧ï✖ù➽è✧ì❭ï❈✯➣❶ø➀ï➊é✐è✥û✙✘➣➊ì➈î➻æ✙ÿ✂è✥ï➉è✥ç✙ït✂➈ä❿å➈ù✂ø➀ï➊é✐è✥ê ì➄ë✛è✥ç✙ï➽û➀ì❛ê✥ê➤ëíÿ✙é✗❶è✧ø➀ì➈é ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★↔è✌è✧ì❺å➄û➀û❣è✧ç✙ï✶æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê➤ø➑é è✧ç☎ï ê❺✘✂ê④è✥ï➊î ✞✑ ✑✆✠⑥ð➧✜✙ì❛ä✢ï✄↔✙å➄î➻æ✙û➀ï✑➌➔û➀ï❶è✿ÿ☎ê✫❶ì➈é✎ê✤ø✓ù✂ï➊ä✿å ê❺✘➇ê✤è✧ï✖î ✡✙ÿ✙ø➀û➑è➉å➈ê❨å➝➊å❛ê❺✖å➈ù✂ï➳ì➄ë❦î➻ì➇ù✙ÿ✙û➑ï❞ê✄➌✂ï❞å✑❿ç✿ì➄ë❯ó❨ç✙ø✠❿ç✺ø➀î➻æ✙û➑ï✖î➻ï➊é✐è✥ê➔å ëíÿ✙é✗❶è✧ø➀ì➈é✂✁❳✏✺ ✼❳❢➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❯➌❯ó❨ç☎ï➊ä✥ï☎✁❑❳❺ø✓ê✒å✺ú➈ï★↔è✥ì➈ä✌ä✥ï➊æ✦✝ ä✥ï✖ê✧ï➊é✐è✧ø➀é❼✂❺è✧ç✙ï✫ì❛ÿ✂è✧æ✙ÿ✙è➻ì➈ë➉è✥ç✙ï✫î➻ì➇ù✙ÿ✙û➑ï➎➌❀❂❳❤ø✓ê❭è✧ç☎ï✫ú➈ï✒❶è✧ì❛ä❙ì➈ë è✧ÿ☎é☎å❄✡✙û➀ï✫æ☎å➈ä✥å➈î❭ï➊è✧ï✖ä✥ê❭ø➀é❑è✥ç✙ï✫î➻ì✂ù✂ÿ✙û➀ï✢➪⑨å ê✧ÿ❼✡☎ê✧ï❶è➓ì➄ë✜❂➶✹➌✛å➄é☎ù ✁❳✸✴ ✜➉ø✓ê✇è✧ç✙ï➤î➻ì✂ù✂ÿ✙û➀ï✻❁ ê✛ø➑é✙æ☎ÿ✂è❨ú➈ï✒❶è✧ì❛ät➪üå➈ê➵ó➵ï➊û➀û✟å❛ê✇è✧ç✙ï➤æ✙ä✥ï➊ú➇ø➀ì➈ÿ☎ê î➻ì✂ù✂ÿ✙û➀ï✻❁ ê➏ì➈ÿ✙è✧æ✙ÿ✂è➵ú➈ï★↔è✧ì❛ä✴➶❶ð❯ã✛ç☎ï➔ø➑é☎æ✙ÿ✂è✆✁✞✝❨è✥ì✌è✧ç☎ï✛➞☎ä❿ê④è✇î❭ì✂ù✂ÿ☎û➑ï ø✓ê❯è✧ç✙ï❨ø➀é✙æ✙ÿ✂è➏æ✎å✠è✤è✥ï➊ä✥é✘✾❀✸✂ð❳➏⑥ë◆è✧ç☎ï➵æ✎å➄ä✧è✧ø✓å➄û✙ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï➵ì➄ë ❉❊✸♣ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★↔è❨è✧ì☎✁❳✢ø➀ê➔ô✐é☎ì✠ó❨é✏➌➇è✧ç✙ï✖é✿è✧ç✙ï➞æ✎å➄ä✧è✧ø✓å➄û✝ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï❞ê✛ì➄ë ❉❊✸ ó❨ø➑è✧ç➲ä✥ï✖ê✧æ✎ï★↔è➔è✥ì✳❂❳ å➄é☎ù✟✁❳✸✴ ✜t➊å➈é✆✡✎ï➚❶ì❛î❭æ☎ÿ✂è✧ï❞ù✫ÿ☎ê✧ø➑é❼✂➽è✧ç✙ï ✡☎å➎❿ô✐ó✛å➄ä❿ù➽ä✥ï✒➊ÿ✙ä✥ä✧ï✖é✗❶ï ✷❉❊✸ ✷❂❳ ✺ ✷✼ ✷❂ ➪❩❂❳ ❁✄✁❳✸✴ ✜ ➶ ✷❉❊✸ ✷✁❳ ✷❉❊✸ ✷✁❳✸✴ ✜ ✺ ✷✼ ✷✁ ➪❩❂❳ ❁✄✁❳✸✴ ✜✄➶ ✷❉❊✸ ✷✁❳ ➪❏❝❩➶ ó❨ç✙ï✖ä✧ï✡✠☞☛ ✠✍✌ ➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❫ø➀ê✇è✧ç☎ï✏✎✐å✑➊ì✑✡✙ø✓å➄é➓ì➈ë ✼ ó❨ø➑è✧ç✶ä✥ï✖ê✧æ◆ï✒↔è✇è✧ì ❂ ï✖úrå➈û➑ÿ✎å✠è✧ï❞ù➽å✠è✇è✧ç✙ï➳æ◆ì➈ø➀é❛èt➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶❯➌➇å➈é☎ù✑✠✒☛ ✠✒✓ ➪✮❂❳ ❁✄✁❳✸✴ ✜ ➶ ø✓ê➳è✧ç✙ï☎✎✐å✑❶ì➎✡✙ø✓å➄é❖ì➈ë❀✼✴ó❨ø➩è✥ç❺ä✥ï✖ê✧æ✎ï★↔è➤è✧ì✟✁ ð➻ã✛ç✙ï☎✎✐å✑❶ì➎✡✙ø✓å➄é➲ì➈ë å❙ú➈ï✒❶è✧ì❛ä➏ëíÿ✙é✗❶è✧ø➀ì➈é✿ø➀ê✛å✒î➻å➄è✧ä✥ø➟↔ ➊ì➈é✐è✥å➈ø➑é☎ø➑é❼✂✒è✧ç✙ï➳æ☎å➈ä✤è✥ø➀å➈û◆ù✂ï✖ä✧ø➀ú✠å♦✝ è✧ø➀ú➈ï❞ê❖ì➈ë✢å➄û➀û❙è✧ç✙ï ì➈ÿ✂è✥æ✙ÿ✂è❿ê❖ó❨ø➑è✧ç➶ä✧ï❞ê✤æ◆ï✒❶è❖è✧ì å➄û➀û❙è✧ç✙ï ø➑é☎æ✙ÿ✂è✥ê✖ð ã✛ç✙ï✆➞✎ä✥ê✤è➽ï✒➍✐ÿ☎å➄è✧ø➀ì➈é❭❶ì❛î❭æ☎ÿ✂è✧ï❞ê✶ê✧ì➈î➻ï✺è✧ï✖ä✧î➓ê➓ì➈ë♣è✥ç✙ï↕✂➈ä❿å➈ù✙ø➑ï✖é❛è ì➄ë❑❉❊✸❼➪✮❂➶✹➌➤ó❨ç✙ø➀û➀ï❖è✥ç✙ï ê✤ï★❶ì➈é✎ù✻ï✒➍✐ÿ☎å➄è✧ø➀ì➈é ✂➈ï➊é☎ï➊ä❿å✠è✧ï❞ê✶å⑧✡☎å✑❿ô❩✝ ó✛å➄ä❿ù❍ä✧ï★❶ÿ✙ä✥ä✥ï➊é✗➊ï✑➌➔å➈ê✢ø➀é✻è✧ç☎ï❺ó➵ï➊û➀û➟✝⑥ô➇é✙ì✠ó❨é ✡☎å➎❿ô❩✝♠æ✙ä✥ì➈æ✎å❄✂❛å➄è✧ø➀ì➈é æ✙ä✥ì✦❶ï✖ù✙ÿ✙ä✧ï➤ëíì❛ä➉é✙ï✖ÿ✙ä❿å➄û❀é☎ï❶è④ó➵ì➈ä✥ô➇ê✖ð ✎ï➚➊å➄é➲årú❛ï➊ä❿å❄✂❛ï➉è✥ç✙ï➉✂❛ä✥å❛ù✂ø➟✝ ï➊é✐è❿ê❨ì✠ú➈ï✖ä✛è✧ç✙ï✌è✥ä✥å➈ø➑é☎ø➑é❼✂➓æ✎å✠è✤è✥ï➊ä✥é☎ê✛è✥ì➓ì✑✡✂è❿å➄ø➀é✢è✥ç✙ï✌ëíÿ✙û➀û❀✂❛ä✥å❛ù✂ø➀ï➊é✐è✖ð ➏⑥è✢ø✓ê➽ø➀é✐è✧ï✖ä✧ï❞ê④è✥ø➑é❼✂ è✧ì é✙ì➈è✧ï✫è✥ç☎å✠è✿ø➑é✲î➻å➈é☛✘❤ø➀é☎ê④è❿å➄é✗➊ï✖ê➽è✧ç✙ï✖ä✧ï➲ø✓ê é✙ì❺é✙ï➊ï❞ù è✧ì❖ï✄↔✂æ✙û➑ø✁❶ø➑è✧û✠✘①❶ì❛î❭æ☎ÿ✂è✧ï➓è✥ç✙ï✔✎✐å✑➊ì✑✡✙ø✓å➄é î➓å➄è✧ä✥ø➟↔☛ð➲ã✛ç✙ï å❄✡◆ì✠ú➈ï➵ëíì❛ä✧î❙ÿ✙û➀å➳ÿ☎ê✧ï✖ê❣è✧ç✙ï➔æ☎ä✧ì✂ù✂ÿ✗❶è➏ì➄ë◆è✧ç✙ï✕✎❛å➎❶ì✑✡☎ø➀å➈é❙ó❨ø➑è✧ç➓å➤ú➈ï★✹✝ è✧ì❛ä❨ì➄ë❦æ☎å➄ä✧è✧ø✓å➄û✝ù✙ï➊ä✥ø➑ú✠å✠è✥ø➑ú❛ï✖ê✒➌✂å➄é☎ù✿ø➑è➔ø➀ê✛ì➈ë➺è✧ï➊é✺ï❞å➈ê✧ø➑ï✖ä✛è✧ì➵❶ì❛î❭æ☎ÿ✂è✧ï è✧ç☎ø➀ê♣æ✙ä✥ì➇ù✙ÿ✗↔è➤ù✂ø➑ä✥ï✒❶è✧û✠✘✢ó❨ø➑è✧ç☎ì➈ÿ✂èt❶ì➈î➻æ✙ÿ✙è✧ø➀é❼✂➽è✥ç✙ï✖✎✐å✑➊ì✑✡✙ø✓å➄é✫✡◆ï❯✝ ëíì➈ä✥ï➊ç✎å➄é☎ù☛ð✞➏➠é➛✚✎✘✫å➄é✎å➄û➀ì✑✂✑✘✢ó❨ø➩è✥ç❖ì❛ä✥ù✂ø➀é☎å➈ä❺✘✿î✒ÿ✙û➑è✧ø✙✝⑥û➀å✪✘❛ï➊ä➉é☎ï➊ÿ✙ä❿å➄û é✙ï➊è④ó✇ì❛ä✧ô✂ê✒➌◆å➈û➑û❜✡✙ÿ✂è➤è✧ç☎ï❙û✓å➈ê✤è♣î➻ì✂ù✂ÿ✙û➀ï❭å➈ä✧ï➚➊å➈û➑û➀ï✖ù✫ç☎ø➀ù✙ù✙ï➊é❖û✓å✪✘➈ï✖ä✥ê ✡◆ï✒➊å➈ÿ☎ê✧ï➉è✥ç✙ï➊ø➀ä❨ì➈ÿ✂è✥æ✙ÿ✂è❿ê➵å➈ä✧ï➳é✙ì➈è➵ì➎✡☎ê✤ï✖ä✧ú✠å✑✡✙û➑ï➉ëíä✧ì❛î▲è✧ç✙ï➤ì❛ÿ✂è✥ê✧ø➀ù✙ï➈ð î➻ì➈ä✥ï➑➊ì➈î➻æ✙û➀ï❯↔➲ê✤ø➑è✧ÿ✎å✠è✧ø➀ì➈é✎ê➔è✥ç☎å➄é❖è✧ç✙ï➻ê✤ø➀î➻æ✙û➀ï➚✖å➈ê❘➊å➈ù✙ï➞ì➄ë➏î➻ì✂ù☛✝ ÿ✙û➀ï✖ê✢ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù❍å✑✡✎ì✠ú❛ï✑➌➵è✧ç☎ï❖æ☎å➈ä✤è✥ø➀å➈û➉ù✂ï➊ä✥ø➀úrå➄è✧ø➀ú➈ï✫é✙ì➈è✥å➄è✧ø➀ì➈é❭✡◆ï❯✝ ❶ì❛î➻ï✖ê➞ê✧ì➈î➻ï➊ó❨ç✎å✠è➞å➄î➑✡✙ø✙✂❛ÿ✙ì➈ÿ✎ê✌å➄é☎ù➷åró❨ô➇ó➵å➈ä✥ù✝ð➣✕ ❶ì➈î➻æ✙û➀ï❶è✥ï➊û✠✘ ä✥ø✙✂❛ì➈ä✥ì➈ÿ☎ê❯ù✂ï➊ä✥ø➀úrå➄è✧ø➀ì➈é❭ø➑é❭î➻ì➈ä✥ï➃✂❛ï➊é✙ï✖ä✥å➈û✦➊å➈ê✧ï✖ê❨✖å➄é➚✡✎ï➔ù✂ì❛é✙ï✛ÿ☎ê✧ø➑é❼✂ ✗❀å✑✂➈ä❿å➄é❼✂❛ï➔ëíÿ✙é✥↔è✧ø➀ì➈é✎ê✜✞✑✻✘✠❖➌ ✞✑❼➾✡✠❖➌ ✞✑ ✑✆✠⑥ð ã❀ä❿å➈ù✂ø➑è✧ø➀ì➈é✎å➄û➈î❙ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä❀é✙ï✖ÿ✙ä✥å➈û❛é☎ï❶è④ó➵ì➈ä✥ô➇ê❇å➄ä✥ï❫å➉ê✧æ◆ï✒❶ø✓å➄û☛➊å❛ê✤ï ì➄ë✟è✥ç✙ï➉å❄✡◆ì✠ú➈ï✛ó❨ç✙ï✖ä✧ï✛è✥ç✙ï♣ê✤è✥å➄è✧ï❨ø➀é✂ëíì➈ä✥î➓å✠è✥ø➑ì❛é✗✁❳ ø✓ê❣ä✧ï✖æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù ó❨ø➑è✧ç ➞❼↔✂ï✖ù☛✝➠ê✤ø✠➽➊ï❞ù✻ú❛ï✒↔è✥ì➈ä❿ê✄➌♣å➄é☎ù✲ó❨ç✙ï➊ä✥ï❖è✥ç✙ï➷î➻ì➇ù✙ÿ✙û➑ï❞ê✫å➄ä✥ï❺å➈û➟✝ è✧ï✖ä✧é✎å✠è✧ï❞ù➓û✓å✪✘➈ï➊ä❿ê❦ì➄ë✝î➓å➄è✧ä✥ø➟↔➻î✒ÿ✙û➑è✧ø➀æ✙û➀ø✠✖å✠è✥ø➑ì❛é☎ê④➪íè✧ç✙ï♣ó✇ï✖ø✙✂❛ç✐è✥ê✴➶❣å➄é☎ù ❶ì❛î➻æ✎ì❛é✙ï➊é✐è❇✝⑥ó❨ø✓ê✤ï❫ê✧ø✠✂➈î➻ì➈ø✓ù➳ëíÿ✙é✗❶è✧ø➀ì➈é☎ê✎➪➺è✥ç✙ï❫é✙ï✖ÿ✙ä✥ì➈é☎ê✴➶↔ð❯õ➔ì✠ó➵ï➊ú➈ï✖ä✒➌ å➈ê✇ê④è❿å✠è✥ï✖ù➓ï❞å➄ä✥û➑ø➀ï➊ä★➌➈è✥ç✙ï♣ê✤è✥å➄è✧ï♣ø➑é✂ëíì❛ä✧î➓å➄è✧ø➀ì➈é➓ø➀é ❶ì❛î❭æ☎û➑ï✄↔➻ä✥ï✒➊ì✑✂❛é✙ø➟✝ è✧ø➀ì➈é➷ê❺✘➇ê✤è✧ï✖î❋ø✓êt✡✎ï❞ê④è➞ä✥ï➊æ☎ä✧ï❞ê✤ï✖é❛è✥ï✖ù➔✡❩✘➔✂➈ä❿å➄æ✙ç☎ê➳ó❨ø➑è✧ç➷é➇ÿ✙î➻ï➊ä✥ø✠✖å➄û ø➀é✂ëíì➈ä✥î➓å✠è✧ø➀ì➈é➞å➄è✤è❿å✑❿ç✙ï❞ù➳è✧ì➔è✥ç✙ï✇å➈ä❘✖ê➊ð❀➏➠é✌è✧ç✙ø✓ê❜➊å➈ê✧ï✑➌rï❞å✑❿ç✌î➻ì✂ù✂ÿ✙û➀ï✑➌ ➊å➈û➑û➀ï✖ù❺å✫â➳ä✥å➈æ✙ç❺ã❇ä✥å➈é☎ê✤ëíì➈ä✥î❭ï✖ä✒➌◆è❿å➄ô➈ï❞ê➉ì❛é✙ï➻ì➈ä➤î➻ì➈ä✥ï➉✂❛ä✥å➈æ✙ç☎ê➤å➈ê ø➀é✙æ✙ÿ✂è★➌✝å➄é☎ù✺æ✙ä✥ì✂ù✂ÿ✗➊ï✖ê➳å➣✂➈ä❿å➄æ☎ç➲å➈ê➔ì❛ÿ✂è✧æ☎ÿ✂è✖ð➤ñ➔ï➊è④ó✇ì❛ä✧ô✂ê❨ì➈ë❫ê✤ÿ✗❿ç î➻ì✂ù✂ÿ✙û➀ï✖ê❭å➄ä✥ï➣➊å➄û➀û➀ï✖ù❤â➳ä❿å➄æ✙ç ã❇ä❿å➄é☎ê✤ëíì➈ä✥î➻ï➊ä✒ñ➉ï❶è④ó➵ì➈ä✥ô➇ê ➪üâ➤ã❨ñ④➶❶ð þ➇ï★↔è✧ø➀ì➈é✎ê➵➏r✣➉➌✔✣✬➏➻å➄é✎ù⑧✣④➏❇➏❺➏➻ù✙ï➊ú➈ï✖û➑ì❛æ❤è✥ç✙ï↕❶ì❛é✗❶ï✖æ✂è➽ì➄ë➞â➤ã❨ñ➉ê✒➌ å➄é✎ù✻ê✤ç☎ì✠ó✴è✥ç☎å✠è➲â➳ä✥å❛ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù ✗✝ï✖å➈ä✧é☎ø➑é❼✂⑧➊å➄é❭✡◆ï❖ÿ✎ê✤ï❞ù➘è✧ì è✧ä❿å➄ø➀é❺å➈û➑û❦è✥ç✙ï➻æ☎å➄ä❿å➄î➻ï➊è✧ï➊ä❿ê♣ø➑é➷å➄û➀û❦è✧ç✙ï➻î➻ì✂ù✂ÿ✙û➀ï✖ê✌ê✧ì✿å❛ê♣è✧ì✺î➻ø➑é✙ø✙✝ î➻ø✙➽✖ï❙å➵✂❛û➑ì➎✡☎å➄û❀û➑ì✐ê✧ê➔ëíÿ✙é✗↔è✥ø➑ì❛é✝ð✬➏⑥è♣î➻å✪✘✺ê✧ï➊ï➊î æ☎å➈ä✥å❛ù✂ì✪↔✂ø✠✖å➄û✟è✥ç☎å✠è ✂➈ä❿å➈ù✙ø➑ï✖é❛è❿êt➊å➄é➒✡◆ï➣❶ì➈î➻æ✙ÿ✙è✧ï✖ù❺ó❨ç✙ï✖é❺è✥ç✙ï➓ê④è❿å✠è✥ï➓ø➑é✂ëíì❛ä✧î➓å➄è✧ø➀ì➈é❖ø✓ê ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù✫✡☛✘✢ï❞ê✧ê✧ï➊é✐è✧ø✓å➄û➀û✠✘✿ù✂ø✓ê❘❶ä✥ï❶è✧ï❙ì✑✡✔✓④ï★↔è❿ê➉ê✧ÿ✗❿ç❖å➈ê✬✂➈ä❿å➄æ✙ç✎ê✄➌ ✡✙ÿ✂è❨è✥ç☎å✠è➉ù✙ø✯➣❶ÿ☎û➩è❅✘➐➊å➄é➐✡◆ï➓❶ø➀ä❘➊ÿ✙î✒ú❛ï➊é✐è✧ï❞ù❢➌✙å➈ê❨ê✧ç✙ì✠ó❨é✿û➀å➄è✧ï➊ä❞ð ➇✄➇★➈☎✘➳×❀Ú✚✙✝×❇Ü✐ß☛Þ❢➋í×❀Ú☛Ý❇Ü✢Ö❭Ù✙ß✝à✟Ý❀Ü✺Ö❭Ù✂Þ✜✛➻×❇à✚✢✤✣✗✥✐×❀à ➇✦✣✖×❇Ü❛Ý✂Þ✟Ù✗➊✧✘✕★☛Ý❇à✟Ý☛Û✎Þ☛Ù✙à✪✩➞Ù☎Û✟×✬✫❇Ú✏➋✓Þ❢➋í×❇Ú ã✛ç✙ï✢å✑✡✙ø➑û➀ø➑è❅✘❺ì➈ë➔î✒ÿ✙û➑è✧ø✙✝♠û✓å✪✘➈ï✖ä➞é✙ï➊è④ó✇ì❛ä✧ô✂ê➤è✥ä✥å➈ø➑é✙ï❞ù➷ó❨ø➩è✥ç➹✂❛ä✥å❛ù✂ø➟✝ ï➊é✐è❨ù✂ï✖ê❘❶ï✖é✐è✇è✥ì❙û➀ï✖å➈ä✧é➐❶ì➈î➻æ✙û➀ï❯↔❢➌➇ç✙ø✠✂➈ç✦✝➠ù✂ø➀î➻ï➊é☎ê✧ø➑ì❛é☎å➄û✶➌✐é✙ì➈é✦✝⑥û➀ø➑é✙ï❞å➄ä î➓å➄æ✙æ☎ø➑é❼✂✐ê✌ëíä✥ì➈î÷û✓å➄ä❘✂➈ï➣❶ì❛û➑û➀ï✒❶è✧ø➀ì➈é☎ê❙ì➄ë❨ï✄↔✂å➈î➻æ✙û➑ï❞ê✒î➓å➈ô➈ï✖ê✌è✧ç✙ï✖î ì✑✡➇ú➇ø➀ì➈ÿ☎ê➃➊å➄é✎ù✂ø➀ù☎å✠è✧ï❞ê❣ëíì❛ä✇ø➀î➓å❄✂➈ï➉ä✧ï★❶ì✑✂❛é✙ø➑è✧ø➀ì➈é➻è✥å❛ê✤ô✂ê✖ð❳➏➠é➽è✧ç✙ï♣è✧ä❿å♦✝ ù✂ø➑è✧ø➀ì➈é☎å➈û☎î➻ì✂ù✂ï➊û✎ì➄ë✟æ☎å✠è✧è✧ï✖ä✧é➓ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✏➌➈å✌ç☎å➈é☎ù☛✝➠ù✂ï✖ê✧ø✠✂➈é✙ï❞ù✒ëíï❞å♦✝ è✧ÿ☎ä✧ï➳ï❯↔➇è✧ä❿å✑❶è✧ì❛ä➃✂✐å✠è✧ç☎ï➊ä❿ê❫ä✥ï➊û➀ï➊ú✠å➄é✐è✛ø➑é✙ëíì➈ä✥î➻å➄è✧ø➀ì➈é➓ëíä✥ì➈î▲è✥ç✙ï➤ø➑é☎æ✙ÿ✂è å➄é✎ù➞ï➊û➀ø➀î❭ø➀é☎å➄è✧ï✖ê❯ø➀ä✧ä✥ï➊û➀ï➊ú✠å➈é❛è❯ú✠å➄ä✥ø➀å✑✡✙ø➑û➀ø➑è✧ø➀ï✖ê✖ð❳✕❤è✧ä❿å➄ø➀é☎å❄✡✙û➀ï➃➊û➀å❛ê✧ê✧ø✙➞☎ï➊ä è✧ç☎ï➊é ➊å✠è✥ï✄✂❛ì➈ä✥ø✙➽✖ï✖ê➉è✧ç✙ï➻ä✥ï✖ê✧ÿ✙û➩è✥ø➑é✗✂✶ëíï❞å✠è✧ÿ☎ä✧ï➻ú➈ï★↔è✥ì➈ä❿ê➉ø➀é✐è✧ì✫❶û✓å➈ê✥ê✤ï❞ê➊ð ➏➠é è✧ç✙ø✓ê✌ê❘❿ç✙ï➊î➻ï✑➌❇ê④è❿å➄é☎ù✙å➈ä✥ù✏➌☛ëíÿ✙û➀û✙✘❩✝r❶ì➈é☎é✙ï✒❶è✧ï✖ù❺î✒ÿ☎û➩è✥ø➟✝⑥û➀å✪✘❛ï➊ä➤é✙ï➊è❇✝ ó➵ì➈ä✥ô➇ê✩✖å➄é✫✡◆ï➞ÿ☎ê✧ï✖ù✫å➈ê✩➊û➀å❛ê✧ê✧ø✙➞☎ï➊ä❿ê➊ð✎✕ æ◆ì➄è✧ï✖é✐è✧ø✓å➄û➀û✙✘✢î❭ì❛ä✧ï✌ø➀é✐è✧ï✖ä❇✝ ï✖ê✤è✧ø➀é❼✂➻ê❺❿ç☎ï➊î➻ï➉ø✓ê✇è✧ì✒ä✥ï➊û✠✘➓ì➈é✢å➈ê✇î✒ÿ✗❿ç✿å➈ê✇æ✎ì✐ê✧ê✧ø✙✡☎û➑ï➳ì➈é➽û➀ï✖å➈ä✧é☎ø➑é❼✂ ø➀é✲è✥ç✙ï❖ëíï❞å✠è✥ÿ✙ä✧ï ï❯↔➇è✥ä✥å➎↔è✧ì❛ä✢ø➑è✥ê✧ï➊û➑ë④ð ➏➠é✾è✧ç✙ï ➊å❛ê✤ï ì➄ë➚❿ç☎å➄ä❿å✑❶è✧ï➊ä ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❀➌✇å é✙ï❶è④ó➵ì➈ä✥ô✢❶ì❛ÿ✙û✓ù➹✡◆ï✺ëíï✖ù ó❨ø➑è✧ç❍å➄û➀î➻ì❛ê✤è➻ä❿åró ø➀é✦✝ æ✙ÿ✂è❿ê✬➪⑨ï➈ð ✂☎ð❣ê✤ø✠➽➊ï✄✝♠é☎ì➈ä✥î➻å➈û➑ø✠➽➊ï❞ù✒ø➀î➓å❄✂❛ï✖ê✴➶↔ð ✎ç✙ø➑û➀ï❨è✥ç✙ø➀ê✎➊å➈é➝✡✎ï➉ù✙ì➈é✙ï ó❨ø➑è✧ç✒å➈é✌ì➈ä❿ù✂ø➑é✎å➄ä❘✘❨ëíÿ✙û➀û✙✘✌❶ì❛é✙é✙ï★↔è✧ï❞ù➳ëíï➊ï❞ù☛✝üëíì❛ä✧ó✛å➄ä❿ù➳é✙ï❶è④ó➵ì➈ä✥ô➉ó❨ø➑è✧ç ê✧ì➈î➻ï➳ê✧ÿ✗✒❶ï✖ê✥ê➵ëíì➈ä➵è❿å➈ê✧ô➇ê✛ê✧ÿ✗❿ç✺å➈ê✔❿ç☎å➄ä❿å✑❶è✧ï✖ä✇ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✏➌✐è✧ç✙ï✖ä✧ï å➄ä✥ï➤æ✙ä✥ì✑✡✙û➀ï➊î➓ê✖ð ✜❇ø➀ä❿ê④è✥û✙✘➎➌➇è❅✘✐æ☎ø✠✖å➄û✝ø➀î➓å❄✂➈ï❞ê✛å➄ä✥ï➳û✓å➄ä❘✂➈ï➎➌➇ì➄ë➺è✧ï✖é✢ó❨ø➑è✧ç➲ê✧ï➊ú❛ï➊ä❿å➄û◆ç➇ÿ✙é✦✝ ù✂ä✥ï✖ù✫ú✠å➈ä✧ø✓å❄✡✙û➀ï✖ê➑➪íæ✙ø✙↔✂ï➊û✓ê❘➶❶ð✌✕òëíÿ✙û➀û✠✘➎✝r❶ì❛é✙é✙ï★↔è✧ï❞ù✆➞☎ä❿ê✤è♣û✓å✪✘➈ï➊ä➉ó❨ø➩è✥ç✏➌ ê✥å✪✘✫ì➈é✙ï❭ç➇ÿ✙é☎ù✂ä✥ï✖ù❖ç✙ø➀ù☎ù✂ï➊é➷ÿ✙é✙ø➑è✥ê➤ø➀é❺è✧ç✙ï➚➞☎ä❿ê④è➤û✓å✪✘➈ï➊ä★➌✟ó✇ì❛ÿ✙û✓ù❺å➈û➟✝ ä✥ï✖å➈ù❼✘➛❶ì❛é✐è✥å➄ø➀é➷ê✤ï✖ú➈ï➊ä❿å➄û❦è✥ï➊é☎ê✒ì➄ë✛è✧ç☎ì➈ÿ☎ê✥å➄é☎ù☎ê✌ì➄ë✛ó✇ï✖ø✙✂❛ç✐è✥ê✖ð✿þ➇ÿ✗❿ç å❙û➀å➈ä❺✂❛ï➔é➇ÿ✙î➉✡◆ï➊ä✛ì➄ë❇æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê❫ø➀é✗➊ä✧ï❞å➈ê✧ï✖ê❣è✥ç✙ï✌➊å➈æ☎å✑➊ø➩è❅✘➻ì➄ë❀è✧ç✙ï ê❺✘➇ê✤è✧ï✖î▼å➈é☎ù✿è✧ç☎ï➊ä✥ï❶ëíì➈ä✥ï✌ä✥ï✒➍✐ÿ✙ø➀ä✧ï❞ê❨å➓û➀å➈ä❺✂❛ï➊ä✛è✥ä✥å➈ø➑é✙ø➀é❼✂➽ê✧ï❶è❞ð✎➏➠é❺å➈ù☛✝ ù✂ø➑è✧ø➀ì➈é✏➌◆è✥ç✙ï❙î➻ï✖î❭ì❛ä❺✘✺ä✥ï✒➍✐ÿ✙ø➀ä✧ï✖î➻ï➊é✐è➉è✥ì✢ê✤è✧ì➈ä✥ï✒ê✧ì✶î➓å➄é☛✘✺ó➵ï➊ø✠✂➈ç✐è✥ê î➓å✪✘✢ä✥ÿ✙û➀ï➞ì➈ÿ✙è✞❶ï✖ä✤è❿å➄ø➀é✫ç☎å➄ä❿ù✂ó✛å➄ä✥ï➤ø➑î➻æ✙û➀ï➊î➻ï➊é✐è❿å✠è✧ø➀ì➈é✎ê➊ð✬✚➵ÿ✂è★➌◆è✧ç✙ï î➓å➄ø➀é❤ù✂ï✄➞✥❶ø➀ï➊é✗✄✘❺ì➈ë➔ÿ✙é☎ê✤è✧ä✥ÿ✗↔è✥ÿ✙ä✥ï✖ù➷é✙ï➊è✥ê✒ëíì➈ä✒ø➀î➓å❄✂❛ï➽ì➈ä✒ê✧æ◆ï➊ï✒❿ç å➄æ☎æ✙û➑ø✁➊å➄è✧ø➀ì➈é☎ê➤ø✓ê✌è✥ç☎å✠è✌è✥ç✙ï✄✘ ç☎årú❛ï❭é☎ì✆✡✙ÿ✙ø➀û➩è❺✝♠ø➀é ø➑é➇ú✠å➄ä✥ø✓å➄é✗➊ï❭ó❨ø➑è✧ç ä✥ï✖ê✧æ✎ï★↔è➞è✥ì✫è✧ä❿å➄é☎ê✧û✓å✠è✧ø➀ì➈é✎ê✄➌❯ì➈ä➞û➀ì✦➊å➈û❫ù✂ø✓ê④è✥ì➈ä✧è✧ø➀ì➈é☎ê✒ì➄ë✛è✧ç✙ï✶ø➑é✙æ☎ÿ✂è✥ê✖ð ✚➵ï❶ëíì❛ä✧ï➚✡✎ï✖ø➑é✗✂✺ê✧ï➊é✐è➳è✧ì✿è✧ç☎ï➑➞✗↔➇ï❞ù☛✝➠ê✤ø✠➽➊ï❭ø➑é☎æ✙ÿ✂è➞û✓å✪✘➈ï➊ä➳ì➄ë➵å✢é☎ï➊ÿ✙ä❿å➄û é✙ï➊è✒➌✇❿ç☎å➄ä❿å✑❶è✧ï➊ä➉ø➑î➓å❄✂❛ï✖ê✒➌☎ì❛ä➉ì➄è✥ç✙ï➊ä❊✑❄✧➶ì➈ä➉➾★✧✹ê✧ø✙✂❛é☎å➄û✓ê✒➌☎î✒ÿ✎ê④è✞✡✎ï å➄æ☎æ✙ä✧ì✪↔✂ø➀î➓å✠è✧ï✖û✙✘ ê✧ø✠➽➊ï❯✝⑥é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù❤å➄é☎ù ❶ï✖é✐è✧ï➊ä✥ï✖ù❑ø➀é❤è✥ç✙ï✫ø➑é☎æ✙ÿ✂è ➞☎ï✖û➀ù☛ð❹→➉é✂ëíì➈ä✧è✧ÿ☎é☎å✠è✥ï➊û✠✘✑➌➇é✙ì➻ê✤ÿ✗❿ç✢æ✙ä✥ï➊æ✙ä✥ì✦❶ï✖ê✥ê✧ø➑é❼✂➑➊å➄é➙✡✎ï➤æ◆ï➊ä✧ëíï✒❶è✽✰ ç☎å➈é☎ù✂ó❨ä✥ø➩è✥ø➑é❼✂❖ø➀ê✒ì➄ë➺è✧ï✖é é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✖ù å➄è➞è✧ç☎ï✶ó➵ì➈ä❿ù❺û➀ï➊ú❛ï➊û✶➌❦ó❨ç✙ø✠❿ç ➊å➈é ➊å➈ÿ☎ê✧ï♣ê✧ø✙➽✖ï✑➌✂ê✤û✓å➄é✐è★➌➇å➈é☎ù➽æ◆ì❛ê✧ø➑è✧ø➀ì➈é➽ú✠å➄ä✥ø✓å✠è✧ø➀ì➈é✎ê➏ëíì❛ä✇ø➀é☎ù✂ø➀ú➇ø➀ù✂ÿ✎å➄û ❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê✖ð✝ã✛ç✙ø✓ê✄➌✪❶ì❛î➉✡✙ø➀é✙ï❞ù♣ó❨ø➑è✧ç✒úrå➈ä✧ø✓å❄✡☎ø➑û➀ø➩è❅✘➉ø➑é✌ó❨ä✥ø➩è✥ø➑é✗✂➔ê④è❅✘➇û➀ï✑➌ ó❨ø➀û➑û❨➊å➈ÿ☎ê✤ï✒úrå➈ä✧ø✓å✠è✥ø➑ì❛é☎ê❨ø➀é✫è✧ç☎ï✒æ◆ì❛ê✧ø➩è✥ø➑ì❛é✫ì➄ë❫ù✂ø✓ê④è✥ø➑é✥↔è✧ø➀ú➈ï✒ëíï✖å➄è✧ÿ✙ä✥ï✖ê ø➀é➽ø➑é☎æ✙ÿ✂è✛ì✑✡✔✓④ï★↔è✥ê✖ð❜➏➠é✶æ✙ä✥ø➑é✥❶ø➀æ✙û➑ï➎➌✐å➞ëíÿ✙û➀û✙✘❩✝r❶ì➈é☎é✙ï✒❶è✧ï✖ù➽é✙ï❶è④ó➵ì➈ä✥ô✒ì➈ë ê✧ÿ✯➣❶ø➀ï➊é✐è➞ê✤ø✠➽➊ï➵❶ì❛ÿ✙û✓ù❖û➀ï✖å➈ä✧é❖è✧ì✺æ✙ä✥ì➇ù✙ÿ✗❶ï❭ì➈ÿ✂è✥æ✙ÿ✂è❿ê➤è✧ç☎å➄è➞å➄ä✥ï❙ø➀é✦✝ ú✠å➄ä✥ø➀å➈é✐è✛ó❨ø➑è✧ç✫ä✥ï✖ê✧æ◆ï✒↔è❨è✥ì✶ê✧ÿ✗❿ç✺úrå➈ä✧ø✓å✠è✥ø➑ì❛é☎ê✖ð❫õ➔ì✠ó➵ï➊ú❛ï➊ä★➌✐û➀ï✖å➈ä✧é☎ø➑é❼✂ ê✧ÿ✗❿ç å✫è❿å➈ê✧ô❺ó➵ì➈ÿ✙û✓ù æ✙ä✥ì✑✡✎å❄✡✙û✠✘❖ä✥ï✖ê✧ÿ✙û➑è❙ø➀é î✒ÿ✙û➑è✧ø➀æ✙û➀ï✶ÿ✙é☎ø➩è❿ê❙ó❨ø➑è✧ç ê✧ø➑î➻ø➀û➀å➈ä➞ó✇ï✖ø✙✂❛ç✐è✒æ☎å✠è✧è✧ï✖ä✧é☎ê✒æ✎ì✐ê✤ø➑è✧ø➀ì➈é☎ï✖ù å✠è❙úrå➈ä✧ø➀ì➈ÿ✎ê✌û➑ì✦✖å✠è✧ø➀ì➈é✎ê✌ø➑é è✧ç☎ï➉ø➀é✙æ✙ÿ✂è✛ê✤ì➞å❛ê❣è✧ì❙ù✂ï❶è✥ï✒↔è✛ù✂ø✓ê④è✥ø➑é✗❶è✧ø➀ú➈ï❨ëíï❞å✠è✥ÿ✙ä✧ï❞ê➏ó❨ç✙ï✖ä✧ï✖ú➈ï✖ä❣è✧ç✙ï✒✘ å➄æ☎æ✎ï❞å➄ä✛ì➈é✢è✧ç✙ï➤ø➀é✙æ✙ÿ✂è❞ð ✗✝ï❞å➄ä✥é✙ø➑é✗✂✒è✥ç✙ï✖ê✧ï➤ó✇ï✖ø✙✂❛ç❛è✩❶ì❛é✦➞✗✂❛ÿ✙ä✥å➄è✧ø➀ì➈é☎ê