CXC.Ob CRE IEEE,AOVEy BE XFV B quite different.Characters must generally be segmented 1 2 out of their context prior to recognition.Segmentation algorithms are rarely perfect and often leave extraneous marks in character images(noise,underlines,neighboring F100,X1,W1) 长3 characters),or sometimes cut characters too much and pro- F3X3,X4) duce incomplete characters.Those images cannot be re- liably size-normalized and centered.Normalizing incom- plete characters can be very dangerous.For example,an F202,W2) enlarged stray mark can look like a genuine 1.Therefore many systems have resorted to normalizing the images at the level of fields or words.In our case,the upper and lower Des profiles of entire fields (amounts in a check)are detected Fig.1'.A trainable system composed of heterogeneous modules. and used to normalize the image to a fixed height.While this guarantees that stray marks will not be blown up into character-looking images,this also creates wide variations words,that can be trained to simultaneously segment and of the size and vertical position of characters after segmen-recognize words,without ever being given the correct seg- tation.Therefore it is preferable to use a recognizer that is mentation. robust to such variations.Figure 13 shows several exam- Figure 14 shows an example of a trainable multi-modular ples of distorted characters that are correctly recognized by system.A multi-module system is defined by the function LeNet-5.It is estimated that accurate recognition occurs implemented by each of the modules,and by the graph of for scale variations up to about a factor of 2,vertical shift interconnection of the modules to each other.The graph variations of plus or minus about half the height of the implicitly defines a partial order according to which the character,and rotations up to plus or minus 30 degrees. modules must be updated in the forward pass.For exam- While fully invariant recognition of complex shapes is still ple in Figure 14,module 0 is first updated,then modules 1 an elusive goal,it seems that Convolutional Networks offer and 2 are updated (possibly in parallel),and finally mod- a partial answer to the problem of invariance or robustness ule 3.Modules may or may not have trainable parameters. with respect to geometrical distortions. Loss functions,which measure the performance of the sys- Figure 13 includes examples of the robustness of LeNet-tem,are implemented as module 4.In the simplest case, 5 under extremely noisy conditions.Processing those the loss function module receives an external input that images would pose unsurmountable problems of segmen-carries the desired output.In this framework,there is no tation and feature extraction to many methods,but qualitative difference between trainable parameters (Wv,W LeNet-5 seems able to robustly extract salient features in the figure),external inputs and outputs (j,>),and from these cluttered images.The training set used for intermediate state variables(Xv,X,XXMXI). the network shown here was the MNIST training set with salt and pepper noise added.Each pixel was ran- A.An Object-Oriented Approach domly inverted with probability 0.1.More examples Object-Oriented programming offers a particularly con- of LeNet-5 in action are available on the Internet at venient way of implementing multi-module systems.Each CHHV ]UUUUU .I D7DEI 5C.EHH.5FMU PESSUF5I. module is an instance of a class.Module classes have a "for- IR.MULTUMODULE SYSTEMS AND e RAPH ward propagation"method (or member function)called fyr Fy whose arguments are the inputs and outputs of the RANSFORMER NETWORKS module.For example,computing the output of module 3 The classical back-propagation algorithm,as described in Figure 14 can be done by calling the method fyr ry on and used in the previous sections,is a simple form of module 3 with the arguments XXMXI.Complex mod- Gradient-Based Learning.However,it is clear that the ules can be constructed from simpler modules by simply gradient back-propagation algorithm given by Equation 4 defining a new class whose slots will contain the member describes a more general situation than simple multi-layer modules and the intermediate state variables between those feed-forward networks composed of alternated linear trans- modules.The fyrry method for the class simply calls the formations and sigmoidal functions.In principle,deriva- fyr ry methods of the member modules,with the appro- tives can be back-propagated through any arrangement of priate intermediate state variables or external input and functional modules,as long as we can compute the prod- outputs as arguments.Although the algorithms are eas- uct of the Jacobians of those modules by any vector.Why ily generalizable to any network of such modules,including would we want to train systems composed of multiple het- those whose in Zuence graph has cycles,we willlimit the dis- erogeneous modules?The answer is that large and complex cussion to the case of directed acyclic graphs(feed-forward trainable systems need to be built out of simple,specialized networks). modules.The simplest example is LeNet-5,which mixes Computing derivatives in a multi-module system is just convolutional layers,sub-sampling layers,fully-connected as simple.A "backward propagation"method,called layers,and RBF layers.Another less trivial example,de- byI Fy,for each module class can be defined for that pur- scribed in the next two sections,is a system for recognizing pose.The byr Fy method of a module takes the same ar-
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✜ ➍✐ÿ✙ø➑è✧ï✫ù✂ø✙➘◆ï✖ä✧ï✖é❛è❞ð❭☞✇ç✎å➄ä❿å✑↔è✥ï➊ä❿ê✒î❙ÿ☎ê✤è➝✂❛ï➊é✙ï✖ä✥å➈û➑û✠✘ ✡✎ï➲ê✧ï✄✂❛î➻ï➊é✐è✧ï❞ù ì➈ÿ✙è✿ì➈ë➞è✧ç✙ï✖ø➑ä✆➊ì➈é✐è✧ï✄↔✐è✿æ✙ä✥ø➑ì❛ä✶è✥ì❑ä✥ï✒➊ì✑✂➈é☎ø➩è✥ø➑ì❛é✝ð▲þ✂ï✄✂➈î➻ï✖é❛è❿å✠è✥ø➑ì❛é å➄û✠✂➈ì❛ä✧ø➑è✧ç☎î➻ê✶å➄ä✥ï✫ä✥å➈ä✧ï✖û✙✘❑æ✎ï✖ä✤ëíï★↔è✢å➈é☎ù➘ì➄ë➺è✥ï➊é✻û➀ï✖årú❛ï✫ï❯↔➇è✧ä❿å➄é✙ï✖ì➈ÿ☎ê î➓å➄ä✥ô✂ê➉ø➀é➒❿ç✎å➄ä❿å✑↔è✥ï➊ä➳ø➑î➓å✑✂➈ï✖ê➚➪íé☎ì➈ø✓ê✤ï➎➌☛ÿ✙é☎ù✂ï✖ä✧û➀ø➑é☎ï✖ê✒➌✝é✙ï➊ø✠✂➈ç☛✡◆ì➈ä✥ø➑é✗✂ ❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê✴➶✹➌❞ì➈ä❦ê✧ì➈î➻ï❶è✥ø➑î➻ï✖ê❜❶ÿ✙è❨❿ç☎å➈ä✥å➎↔è✥ï➊ä❿ê✟è✧ì➇ì➉î❙ÿ✗❿ç✒å➈é☎ù✌æ✙ä✥ì❄✝ ù✂ÿ✗➊ï✢ø➀é✗➊ì➈î➻æ✙û➀ï❶è✧ï✫❿ç☎å➈ä✥å➎↔è✧ï✖ä✥ê✖ð❖ã✛ç✙ì✐ê✤ï✢ø➑î➓å✑✂➈ï✖ê➚➊å➈é✙é✙ì➈è➑✡◆ï✢ä✥ï❯✝ û➀ø➀å✑✡✙û✙✘❑ê✤ø✠➽➊ï✄✝♠é✙ì❛ä✧î➓å➈û➑ø✠➽➊ï❞ù å➄é☎ù⑨❶ï✖é✐è✧ï➊ä✥ï✖ù✝ð ñ➉ì➈ä✥î➻å➈û➑ø✠➽➊ø➀é❼✂ ø➑é✥❶ì➈î➚✝ æ✙û➀ï❶è✥ï ❿ç☎å➄ä❿å✑❶è✧ï✖ä✥êt➊å➈é➛✡◆ï➽ú❛ï➊ä❘✘❖ù✙å➄é✗✂➈ï➊ä✥ì➈ÿ✎ê➊ð➣✜☎ì➈ä✌ï❯↔✙å➈î❭æ☎û➑ï➎➌❯å➈é ï➊é☎û➀å➈ä❺✂❛ï✖ù❖ê✤è✧ä❿å✪✘✫î➓å➄ä✥ô➔➊å➈é❺û➀ì➇ì➈ô✫û➀ø➑ô❛ï➓å➙✂❛ï➊é➇ÿ✙ø➀é✙ï✫➾➈ð➻ã✛ç✙ï➊ä✥ï❶ëíì❛ä✧ï î➓å➄é☛✘✫ê❇✘✂ê✤è✧ï✖î➻ê➳ç☎årú❛ï➞ä✥ï✖ê✧ì➈ä✧è✧ï✖ù✿è✥ì✢é☎ì➈ä✥î➻å➈û➑ø✠➽➊ø➀é❼✂➓è✥ç✙ï➻ø➑î➓å✑✂➈ï✖ê♣å✠è è✧ç☎ï✇û➀ï➊ú❛ï➊û✐ì➄ë✦➞☎ï✖û➀ù✙ê❯ì❛ä❀ó➵ì➈ä❿ù✙ê➊ð❀➏➠é✒ì➈ÿ☎ä❳➊å❛ê✤ï➎➌❞è✧ç✙ï➵ÿ✙æ✙æ◆ï➊ä❣å➄é✎ù✌û➑ì✠ó➵ï➊ä æ✙ä✥ì❄➞☎û➀ï✖ê➞ì➄ë✛ï➊é✐è✧ø➀ä✧ï➝➞✎ï➊û✓ù✙ê ➪üå➄î➻ì➈ÿ✙é✐è❿ê✌ø➑é å✆❿ç☎ï✒❿ô❼➶➤å➄ä✥ï➽ù✂ï❶è✥ï✒❶è✧ï✖ù å➄é✎ù✫ÿ☎ê✧ï✖ù✺è✥ì✶é✙ì❛ä✧î➓å➄û➀ø✠➽➊ï✌è✥ç✙ï✒ø➀î➓å❄✂➈ï✒è✧ì✿å➝➞❼↔✂ï✖ù❖ç✙ï➊ø✠✂➈ç✐è✖ð ✎ç✙ø➀û➑ï è✧ç☎ø➀ê✎✂➈ÿ✎å➄ä❿å➄é✐è✧ï✖ï✖ê❣è✥ç☎å✠è❨ê✤è✧ä❿å✪✘❙î➓å➄ä✥ô✂ê❫ó❨ø➀û➀û◆é☎ì➄è✔✡✎ï✞✡✙û➀ì✠ó❨é➽ÿ✙æ✶ø➀é✐è✧ì ❿ç☎å➈ä✥å➎↔è✧ï✖ä❇✝⑥û➀ì✐ì❛ô➇ø➑é❼✂❭ø➀î➻å✑✂➈ï❞ê✄➌☎è✧ç✙ø✓ê♣å➈û➀ê✧ì➣❶ä✥ï✖å➄è✧ï❞ê❨ó❨ø➀ù✙ï✒ú✠å➄ä✥ø➀å➄è✧ø➀ì➈é☎ê ì➄ë✟è✥ç✙ï♣ê✧ø✙➽✖ï➔å➄é☎ù➻ú❛ï➊ä✧è✧ø✁➊å➈û✙æ✎ì✐ê✤ø➑è✧ø➀ì➈é➓ì➈ë✏❿ç☎å➄ä❿å✑❶è✧ï➊ä❿ê❦å✠ë➺è✥ï➊ä✇ê✧ï✄✂❛î➻ï➊é✦✝ è✥å➄è✧ø➀ì➈é✝ð❣ã✛ç✙ï✖ä✧ï➊ëíì➈ä✥ï✛ø➩è➵ø➀ê❣æ☎ä✧ï➊ëíï➊ä❿å❄✡✙û➀ï✛è✧ì➞ÿ✎ê✤ï➉å➤ä✥ï✒➊ì✑✂❛é✙ø✙➽✖ï➊ä❦è✥ç☎å✠è❫ø✓ê ä✥ì✑✡✙ÿ☎ê✤è➤è✧ì✫ê✧ÿ✗❿ç❺ú✠å➈ä✧ø✓å✠è✥ø➑ì❛é☎ê➊ð➵✜❯ø✠✂➈ÿ✙ä✥ï➙➾ ❜✺ê✤ç✙ì✠ó➔ê➤ê✧ï➊ú❛ï➊ä❿å➄û❦ï❯↔✙å➄î➚✝ æ✙û➀ï✖ê❯ì➄ë◆ù✂ø➀ê✤è✧ì❛ä✤è✥ï✖ù➑❿ç☎å➄ä❿å✑❶è✧ï➊ä❿ê☛è✥ç☎å✠è➏å➈ä✧ï✎❶ì➈ä✥ä✥ï✒↔è✥û✙✘➤ä✥ï✒➊ì✑✂➈é☎ø✙➽✖ï✖ù✌✡☛✘ ✗✝ï❞ñ➔ï➊è❇✝ ✙✂ð➚➏⑥è➞ø➀ê➤ï❞ê④è✥ø➑î➓å✠è✥ï✖ù❖è✧ç☎å➄è✒å✑✒❶ÿ✙ä❿å✠è✥ï✒ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é❺ì✦✄❶ÿ☎ä✥ê ëíì➈ä➔ê❘➊å➈û➑ï➤ú✠å➄ä✥ø✓å✠è✧ø➀ì➈é✎ê✇ÿ☎æ✶è✥ì➽å❄✡◆ì➈ÿ✂è➔å❙ë⑨å✑❶è✧ì➈ä❨ì➈ë ✑❼➌✂ú➈ï➊ä✧è✧ø✁➊å➈û✝ê✤ç☎ø➩ë➺è ú✠å➄ä✥ø➀å➄è✧ø➀ì➈é☎ê➽ì➄ë➤æ✙û➑ÿ✎ê➽ì➈ä✶î❭ø➀é➇ÿ☎ê✶å✑✡✎ì❛ÿ✂è✶ç☎å➈û➩ë➳è✧ç☎ï❖ç✙ï✖ø✙✂❛ç❛è➽ì➈ë➤è✧ç✙ï ❿ç☎å➈ä✥å➎↔è✧ï✖ä✒➌❣å➈é☎ù ä✥ì➄è❿å✠è✧ø➀ì➈é✎ê➞ÿ✙æ❤è✧ì æ✙û➑ÿ✎ê❙ì❛ä❙î➻ø➀é➇ÿ☎ê❑❜✻✘➲ù✂ï✒✂➈ä✥ï➊ï❞ê➊ð ✎ç✙ø➀û➑ï➤ëíÿ☎û➑û✠✘✢ø➀é➇ú✠å➄ä✥ø➀å➈é❛è✛ä✥ï✒❶ì➎✂➈é✙ø➑è✧ø➀ì➈é✢ì➄ë❷❶ì❛î❭æ☎û➑ï✄↔✢ê✤ç✎å➄æ◆ï✖ê✛ø✓ê➔ê④è✥ø➑û➀û å➄é➓ï✖û➑ÿ✎ê✤ø➀ú➈ï✛✂❛ì❛å➈û✻➌❛ø➩è✛ê✤ï✖ï➊î➓ê❣è✥ç☎å✠è✔☞✇ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û☎ñ➉ï❶è④ó➵ì➈ä✥ô✂ê❣ì❄➘✟ï➊ä å➤æ☎å➈ä✤è✥ø➀å➈û☎å➄é☎ê✧ó➵ï➊ä❣è✥ì➤è✧ç✙ï➔æ☎ä✧ì➎✡✙û➑ï✖î✹ì➈ë✟ø➀é➇ú✠å➄ä✥ø➀å➈é✗❶ï❨ì❛ä❣ä✧ì➎✡✙ÿ☎ê✤è✧é✙ï❞ê✧ê ó❨ø➑è✧ç✺ä✧ï❞ê✤æ◆ï✒❶è❨è✧ì➝✂➈ï✖ì➈î➻ï❶è✥ä✧ø✁➊å➈û☛ù✂ø➀ê✤è✧ì❛ä✤è✥ø➑ì❛é☎ê✖ð ✜❇ø✠✂➈ÿ☎ä✧ï➚➾✖❜❙ø➑é✥❶û➀ÿ☎ù✂ï✖ê✛ï✄↔✙å➄î➻æ✙û➀ï✖ê➵ì➄ë❇è✧ç✙ï➤ä✥ì✑✡✙ÿ☎ê✤è✧é☎ï✖ê✥ê✇ì➈ë ✗✝ï❞ñ➔ï➊è❇✝ ✙ ÿ☎é☎ù✂ï➊ä ï❯↔➇è✥ä✧ï✖î❭ï✖û✙✘▲é✙ì❛ø➀ê❺✘ ❶ì➈é✎ù✂ø➩è✥ø➑ì❛é☎ê✖ð❏✓➏ä✥ì✦❶ï❞ê✧ê✧ø➑é✗✂ è✧ç✙ì✐ê✤ï ø➀î➻å✑✂➈ï❞ê❙ó➵ì➈ÿ☎û➀ù❑æ✎ì✐ê✤ï✺ÿ✙é✎ê✤ÿ✙ä✥î➻ì➈ÿ✙é✐è❿å❄✡✙û➀ï✢æ☎ä✧ì➎✡✙û➑ï✖î➓ê➻ì➄ë➳ê✧ï✄✂❛î➻ï➊é✦✝ è✥å➄è✧ø➀ì➈é å➄é✎ù ëíï✖å✠è✥ÿ✙ä✥ï ï❯↔➇è✧ä❿å✑❶è✧ø➀ì➈é❂è✧ì î➓å➄é☛✘❋î➻ï➊è✧ç✙ì✂ù✙ê✒➌❭✡✙ÿ✂è ✗✝ï❞ñ➔ï➊è❇✝ ✙✲ê✧ï➊ï✖î➻ê å❄✡☎û➑ï❤è✧ì❲ä✧ì➎✡✙ÿ☎ê✤è✧û✠✘❲ï✄↔➇è✧ä❿å✑↔è➷ê✧å➈û➑ø➀ï➊é✐è ëíï✖å➄è✧ÿ✙ä✥ï✖ê ëíä✥ì➈î è✥ç✙ï✖ê✧ï①❶û➀ÿ✂è✤è✥ï➊ä✥ï✖ù✾ø➑î➓å❄✂❛ï✖ê✖ð▼ã✛ç☎ï❺è✥ä✥å➈ø➑é☎ø➑é❼✂❍ê✧ï❶è✫ÿ☎ê✧ï✖ù✲ëíì➈ä è✧ç☎ï é✙ï❶è④ó➵ì➈ä✥ô✴ê✧ç✙ì✠ó❨éPç☎ï➊ä✥ï❲ó✛å➈ê è✥ç✙ïòö➲ñ✬➏✤þ✂ã è✥ä✥å➈ø➑é☎ø➑é❼✂ ê✧ï❶è ó❨ø➑è✧ç✲ê✥å➄û➑è✢å➄é✎ù➘æ✎ï✖æ✙æ✎ï✖ä✢é☎ì➈ø✓ê✤ï❖å❛ù✙ù✂ï❞ù☛ð ✭❫å➎❿ç æ☎ø➟↔✂ï➊û♣ó➵å❛ê➓ä❿å➄é✦✝ ù✂ì❛î❭û✠✘ ø➑é➇ú❛ï➊ä✧è✧ï✖ù▼ó❨ø➑è✧ç æ☎ä✧ì➎✡☎å❄✡✙ø➀û➀ø➩è❅✘ ✘✙ð✠➾➈ð ö✫ì❛ä✧ï✹ï❯↔✙å➄î➻æ✙û➀ï✖ê ì➄ë✏✗✝ï❞ñ➔ï➊è❇✝ ✙✹ø➀é å✑❶è✧ø➀ì➈é å➄ä✥ï❲årú✠å➄ø➀û✓å❄✡✙û➀ï ì➈éPè✧ç✙ï ➏➠é✐è✧ï✖ä✧é☎ï❶è✻å✠è ✄✝✆✌✆✘✟✡✠☞☛✌☛✣✍✞✍✌✍✡✏✒✑✫✓✫✔✖✓✌✗✘✑✕✙✚✄✛✏ ✗✣✆✌✆✛✏☞✙✖✢✱✤☎☛ ★✫✗✎✩✌✩☛✘✢✫✙✚✑✟ð ➇✂✁✫➈ ✥❤ß✝Ü❶Þ❢➋☎✄✥❑×❀➊❇ß✝Ü✐Ù✝✆✟✞✤✣➊Þ☛Ù✙Ø✕✣❭Ý❇Ú✏➊✡✠➻à✟Ý✜ ★ ☛✌à✟Ý❀Ú✬✣ ✥✐×❀à☛Ø♣Ù✙à❑Ö❭Ù✂Þ✜✛➻×❇à✚✢✣ ã✛ç✙ï➐➊û➀å❛ê✧ê✧ø✁➊å➄û➃✡☎å✑❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é å➈û✙✂❛ì➈ä✥ø➩è✥ç✙î✫➌➏å➈ê❙ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù å➄é✎ùòÿ☎ê✧ï✖ù ø➀éòè✥ç✙ï æ✙ä✧ï✖ú➇ø➑ì❛ÿ☎ê❖ê✧ï✒❶è✧ø➀ì➈é☎ê✒➌❙ø✓ê å✲ê✧ø➑î➻æ✙û➀ï❤ëíì❛ä✧î ì➈ë â➳ä❿å➈ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù☞✗❀ï✖å➄ä✥é✙ø➀é❼✂☎ð õ➉ì✠ó✇ï✖ú➈ï➊ä★➌❨ø➑è✿ø✓ê✫❶û➀ï✖å➄ä✢è✧ç✎å✠è✿è✧ç✙ï ✂➈ä❿å➈ù✙ø➑ï✖é❛è✞✡☎å✑❿ô❩✝⑥æ✙ä✥ì➈æ☎å✑✂❛å✠è✥ø➑ì❛é✫å➄û✠✂➈ì➈ä✥ø➑è✧ç✙î ✂➈ø➀ú➈ï✖é↕✡☛✘ ✭➜➍✐ÿ☎å➄è✧ø➀ì➈é ❝ ù✂ï❞ê❺➊ä✧ø✠✡✎ï❞ê➔å➻î➻ì➈ä✥ït✂➈ï➊é☎ï➊ä❿å➄û✝ê✧ø➩è✥ÿ☎å✠è✥ø➑ì❛é✢è✥ç☎å➄é✫ê✧ø➑î➻æ✙û➀ï➞î❙ÿ✙û➩è✥ø➟✝⑥û✓å✪✘➈ï➊ä ëíï➊ï❞ù☛✝♠ëíì➈ä✥ó➵å➈ä✥ù✒é✙ï❶è④ó➵ì➈ä✥ô✂ê❜➊ì➈î➻æ◆ì❛ê✧ï✖ù❙ì➄ë☛å➄û➑è✧ï➊ä✥é☎å➄è✧ï✖ù✒û➑ø➀é✙ï❞å➄ä❦è✧ä❿å➄é✎ê❅✝ ëíì➈ä✥î➓å✠è✥ø➑ì❛é☎ê✒å➈é☎ù ê✧ø✙✂❛î❭ì❛ø➀ù☎å➄û➏ëíÿ☎é✗↔è✥ø➑ì❛é☎ê➊ð ➏➠é æ☎ä✧ø➀é✗❶ø➀æ✙û➀ï✑➌➏ù✂ï✖ä✧ø➀ú✠å♦✝ è✧ø➀ú➈ï❞ê✛➊å➈é✫✡✎ï➓✡✎å✑❿ô❩✝♠æ☎ä✧ì❛æ☎å❄✂✐å✠è✧ï❞ù➓è✥ç✙ä✧ì❛ÿ❼✂➈ç✺å➄é☛✘✿å➄ä✥ä✥å➈é❼✂➈ï✖î❭ï✖é✐è➵ì➈ë ëíÿ✙é✗❶è✧ø➀ì➈é☎å➈û❣î➻ì➇ù✙ÿ✙û➑ï❞ê✄➌❇å➈ê➤û➀ì➈é❼✂✺å➈ê➳ó➵ï➝✖å➄é➒❶ì➈î➻æ✙ÿ✙è✧ï➻è✧ç☎ï❭æ☎ä✧ì✂ù☛✝ ÿ✗❶è✛ì➈ë✝è✧ç☎ï ✎✐å✑➊ì✑✡✙ø✓å➄é✎ê❫ì➈ë✝è✧ç✙ì✐ê✤ï➳î➻ì✂ù✂ÿ✙û➀ï✖ê✎✡☛✘➽å➄é☛✘➓ú➈ï★↔è✧ì❛ä✖ð ✎ç☛✘ ó➵ì➈ÿ✙û✓ù➽ó➵ï♣ó✛å➄é✐è➵è✧ì❙è✧ä❿å➄ø➀é✢ê❺✘✂ê④è✥ï➊î➓ê✔❶ì➈î➻æ◆ì❛ê✧ï✖ù✶ì➄ë❀î❙ÿ✙û➩è✥ø➑æ☎û➑ï➤ç✙ï➊è❇✝ ï➊ä✥ì✑✂❛ï➊é✙ï✖ì➈ÿ☎ê✝î❭ì✂ù✂ÿ☎û➑ï❞ê ✑✿ã✛ç✙ï✛å➄é☎ê✧ó➵ï➊ä✝ø✓ê❇è✧ç☎å➄è❇û✓å➄ä❘✂➈ï❫å➈é☎ù➉➊ì➈î➻æ✙û➀ï❯↔ è✧ä❿å➄ø➀é☎å✑✡✙û➑ï➵ê❇✘✂ê✤è✧ï✖î➻ê❇é✙ï✖ï✖ù✌è✥ì✬✡◆ï➃✡☎ÿ✙ø➑û➑è❦ì➈ÿ✂è❦ì➄ë☎ê✧ø➑î➻æ✙û➀ï✑➌❛ê✤æ◆ï✒➊ø➀å➈û➑ø✠➽➊ï❞ù î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð❺ã✛ç✙ï✿ê✤ø➀î➻æ✙û➑ï❞ê④è➻ï❯↔✙å➈î❭æ☎û➑ï➽ø✓ê❄✗✝ï✖ñ➉ï❶è❺✝✝✙❼➌❯ó❨ç☎ø✠❿ç î➻ø✙↔✂ï✖ê ❶ì❛é➇ú➈ì➈û➀ÿ✂è✥ø➑ì❛é☎å➄û✇û➀å✪✘❛ï➊ä❿ê✄➌❣ê✧ÿ❼✡✦✝➠ê✧å➈î❭æ☎û➑ø➀é❼✂❖û✓å✪✘➈ï✖ä✥ê✒➌❯ëíÿ✙û➀û✙✘❩✝r❶ì➈é☎é✙ï✒❶è✧ï✖ù û✓å✪✘➈ï➊ä❿ê✒➌✟å➈é☎ù➔✍✬✚✎✜✻û➀å✪✘❛ï➊ä❿ê➊ð➓✕➉é✙ì➄è✥ç✙ï➊ä✌û➀ï✖ê✥ê♣è✧ä✥ø➑ú➇ø✓å➄û❦ï❯↔✙å➄î➻æ✙û➀ï✑➌✝ù✙ï❯✝ ê❘❶ä✥ø✙✡◆ï✖ù❭ø➀é✒è✥ç✙ï❨é✙ï❯↔➇è❣è④ó➵ì✌ê✤ï★↔è✧ø➀ì➈é✎ê✄➌➄ø✓ê❣å✌ê❺✘➇ê✤è✧ï✖î ëíì➈ä❣ä✥ï✒➊ì✑✂❛é✙ø✙➽✖ø➑é✗✂ F0(X0) E W1 D X1 F1(X0,X1,W1) F2(X2,W2) X2 X3 X4 X5 F3(X3,X4) Function Z t Input Desired Output Loss W2 ✁✗✿▲❍✪❦➎❬ ✰✪❦➣❱✆✵✶✸✶✱✴✿▲❈♦✱✴◗✪❴▲✰❷✺✻❙✒✺✻✵✻✰❅❋⑨❆❅✷✹❋✩❃❄✷✹✺✶✰❅❉✞✷✴⑥❼✯★✰❅✵✻✰❅✸✶✷✹❍✹✰❅❈✪✰❅✷✹✳✪✺❷❋✩✷✄❉✪✳✪❴▲✰❅✺❺❦ ó➵ì➈ä❿ù✙ê✄➌✐è✥ç☎å✠è✬✖å➄é✫✡✎ï➤è✥ä✥å➈ø➑é☎ï✖ù✶è✧ì➓ê✤ø➀î✒ÿ☎û➩è❿å➄é✙ï✖ì➈ÿ☎ê✧û✙✘✢ê✤ï✒✂➈î➻ï➊é✐è➔å➄é☎ù ä✥ï✒❶ì➎✂➈é✙ø✠➽➊ï➤ó➵ì➈ä❿ù✙ê✒➌✂ó❨ø➩è✥ç✙ì➈ÿ✂è♣ï➊ú➈ï✖ä✛✡◆ï➊ø➀é❼✂➵✂➈ø➀ú➈ï✖é✢è✥ç✙ï➉➊ì➈ä✥ä✧ï★↔è➔ê✤ï✒✂❄✝ î➻ï➊é✐è✥å➄è✧ø➀ì➈é✝ð ✜❇ø✠✂➈ÿ☎ä✧ï④➾❪❝➉ê✧ç✙ì✠ó➔ê❯å➄é➞ï✄↔✂å➈î➻æ✙û➑ï✇ì➄ë☎å➔è✧ä❿å➄ø➀é☎å❄✡✙û➀ï❫î❙ÿ✙û➑è✧ø✙✝♠î➻ì✂ù✂ÿ✙û✓å➄ä ê❺✘➇ê✤è✧ï✖î✺ð❹✕ î✒ÿ✙û➑è✧ø✙✝⑥î❭ì✂ù✂ÿ☎û➑ï➞ê❺✘✂ê④è✥ï➊î✴ø✓ê❨ù✂ï✄➞☎é✙ï✖ù✫✡☛✘➓è✥ç✙ï➤ëíÿ✙é✗❶è✧ø➀ì➈é ø➀î❭æ☎û➑ï✖î❭ï✖é✐è✧ï✖ù✫✡☛✘✿ï✖å➎❿ç✿ì➈ë❇è✥ç✙ï➞î➻ì✂ù✂ÿ✙û➀ï✖ê✒➌✎å➈é☎ù✫✡☛✘➽è✧ç☎ï➉✂➈ä❿å➄æ☎ç✢ì➈ë ø➀é❛è✥ï➊ä✴❶ì❛é✙é✙ï✒❶è✧ø➀ì➈é ì➄ë✇è✧ç✙ï➓î➻ì✂ù✂ÿ✙û➀ï✖ê➤è✥ì✺ï✖å✑❿ç ì➄è✥ç✙ï➊ä❞ð➽ã✛ç✙ï➝✂➈ä❿å➄æ☎ç ø➀î❭æ☎û➑ø✁❶ø➑è✧û✠✘ ù✙ï❯➞☎é✙ï❞ê✶å➷æ☎å➈ä✤è✥ø➀å➈û❨ì➈ä❿ù✂ï➊ä➽å➎✄➊ì➈ä❿ù✂ø➑é✗✂❺è✥ì ó❨ç✙ø✁❿ç➘è✧ç✙ï î➻ì✂ù✂ÿ✙û➀ï✖ê➔î❙ÿ☎ê✤è④✡◆ï➞ÿ✙æ✟ù✙å✠è✥ï✖ù✫ø➀é✺è✧ç✙ï✒ëíì➈ä✥ó➵å➈ä✥ù✢æ☎å❛ê✧ê✖ð✩✜✙ì➈ä♣ï❯↔✙å➄î➚✝ æ✙û➀ï➔ø➀é ✜❯ø✙✂❛ÿ✙ä✧ï➉➾❪❝✗➌❛î➻ì➇ù✙ÿ✙û➑ï ✘➤ø➀ê➜➞☎ä❿ê④è✇ÿ✙æ◆ù☎å✠è✧ï❞ù❢➌❛è✧ç✙ï✖é➽î➻ì➇ù✙ÿ✙û➑ï❞ê✛➾ å➄é✎ù ✑➓å➄ä✥ï➞ÿ✙æ✟ù✙å➄è✧ï✖ù➹➪íæ◆ì❛ê✥ê✤ø✠✡✙û✠✘✿ø➀é➲æ☎å➈ä✥å➈û➑û➀ï➊û●➶✹➌◆å➄é☎ù✆➞☎é☎å➄û➀û✠✘✢î➻ì✂ù☛✝ ÿ✙û➀ï❇❜✙ð❦ö✫ì✂ù✂ÿ✙û➀ï✖ê❫î➓å✪✘✒ì➈ä❫î➻å✪✘❙é✙ì➄è✇ç☎årú➈ï✛è✥ä✥å➈ø➑é☎å✑✡✙û➀ï❨æ☎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê➊ð ✗✝ì✐ê✧ê❫ëíÿ✙é✗❶è✧ø➀ì➈é☎ê✒➌✙ó❨ç✙ø✁❿ç✶î➻ï❞å➈ê✧ÿ✙ä✧ï➉è✧ç✙ï➤æ◆ï➊ä✧ëíì➈ä✥î➓å➄é✗➊ï♣ì➈ë✝è✧ç✙ï✌ê❺✘✂ê❅✝ è✧ï✖î✫➌❇å➄ä✥ï➻ø➑î➻æ✙û➀ï➊î➻ï✖é❛è✥ï✖ù➷å➈ê➤î➻ì✂ù✂ÿ✙û➀ï ❝☎ð➝➏➠é è✧ç☎ï➽ê✤ø➀î➻æ✙û➑ï❞ê④è➉✖å➈ê✧ï✑➌ è✧ç☎ï✫û➑ì✐ê✧ê❭ëíÿ✙é✥↔è✧ø➀ì➈é➘î➻ì✂ù✂ÿ✙û➀ï✺ä✧ï★❶ï➊ø➀ú➈ï❞ê➓å➄é ï❯↔➇è✧ï✖ä✧é✎å➄û✛ø➑é☎æ✙ÿ✂è➽è✥ç☎å✠è ➊å➈ä✧ä✥ø➀ï✖ê❨è✥ç✙ï➓ù✂ï✖ê✧ø➀ä✧ï❞ù✫ì➈ÿ✂è✥æ✙ÿ✂è❞ð➓➏➠é➲è✥ç✙ø➀ê➳ëíä❿å➄î➻ï✖ó✇ì❛ä✧ô✇➌✎è✧ç✙ï✖ä✧ï❙ø➀ê➳é✙ì ➍✐ÿ☎å➄û➀ø➑è✥å✠è✥ø➑ú❛ï❨ù✂ø➟➘✟ï➊ä✥ï➊é✥❶ï✩✡◆ï❶è④ó➵ï➊ï➊é❭è✥ä✥å➈ø➑é☎å✑✡✙û➀ï➵æ✎å➄ä❿å➄î➻ï❶è✥ï➊ä❿ê✩➪✌☞✎✍✑✏✒☞✟✓ ø➀é❤è✥ç✙ï✫➞✗✂➈ÿ☎ä✧ï✪➶✹➌❫ï✄↔➇è✧ï➊ä✥é☎å➈û➵ø➀é✙æ✙ÿ✙è✥ê➽å➈é☎ù❤ì❛ÿ✂è✧æ☎ÿ✂è✥ê✆➪✕✔✖✏✘✗✙✏✘✚❼➶✹➌❣å➄é☎ù ø➀é❛è✥ï➊ä✥î➻ï✖ù✂ø✓å✠è✥ï➤ê✤è✥å➄è✧ï✌ú✠å➄ä✥ø✓å❄✡✙û➀ï✖ê★➪✜✛✢✍✑✏✘✛✣✓✤✏✘✛✦✥✤✏✧✛✩★✙✏✘✛✩✪☛➶❶ð ✚✜✛✒✚ ➨ ☞✯✬✫✄➳❄✄➺❩✭✒☞✎➡▼✣ ➳✄➨✗➺r➳✱✪ ✚ ➤➎➤✈➡❺➯★➢✻❄✴➥ ý④✡✔✓④ï✒❶è❇✝✤ý♣ä✧ø➀ï➊é✐è✧ï❞ù✿æ✙ä✧ì➎✂➈ä❿å➄î➻î➻ø➑é✗✂❭ì✑➘◆ï✖ä✥ê➉å➓æ☎å➄ä✧è✧ø✁❶ÿ☎û➀å➈ä✧û✠✘➙➊ì➈é✦✝ ú➈ï✖é✙ø➀ï➊é✐è➉ó✛å✪✘✶ì➄ë❣ø➀î➻æ✙û➑ï✖î➻ï➊é✐è✧ø➀é❼✂➽î❙ÿ✙û➩è✥ø➟✝⑥î➻ì✂ù✂ÿ✙û➀ï✒ê❇✘✂ê✤è✧ï✖î➻ê✖ð ✭❫å➎❿ç î➻ì✂ù✂ÿ✙û➀ï❫ø✓ê❀å➄é✌ø➀é☎ê✤è✥å➈é✗❶ï➏ì➈ë✙å✛❶û✓å➈ê✥ê➊ð❇ö✫ì✂ù✂ÿ✙û➀ï➜❶û✓å➈ê✥ê✤ï❞ê☛ç☎årú❛ï❫å❑☛④ëíì❛ä❇✝ ó✛å➄ä❿ù✻æ✙ä✥ì➈æ☎å✑✂❛å➄è✧ø➀ì➈é✤✌ î➻ï➊è✧ç✙ì✂ù ➪íì➈ä✺î➻ï➊î➑✡✎ï✖ä✺ëíÿ✙é✗❶è✧ø➀ì➈é✥➶➐➊å➈û➑û➀ï✖ù ✭ ✟✝✑✝✢✖✟➽ó❨ç✙ì✐ê✤ï➞å➈ä❺✂❛ÿ✙î➻ï➊é✐è✥ê❨å➈ä✧ï➳è✥ç✙ï➞ø➀é✙æ✙ÿ✂è❿ê➉å➄é✎ù✢ì❛ÿ✂è✧æ☎ÿ✂è✥ê❨ì➈ë❦è✧ç✙ï î➻ì✂ù✂ÿ✙û➀ï➈ð➉✜☎ì➈ä➳ï❯↔✙å➄î➻æ✙û➀ï✑➌✏➊ì➈î➻æ✙ÿ✂è✥ø➑é❼✂✢è✧ç☎ï❙ì❛ÿ✂è✧æ✙ÿ✙è➤ì➈ë❫î➻ì✂ù✂ÿ✙û➀ï❄❜ ø➀é➔✜❯ø✙✂❛ÿ✙ä✧ï ➾❪❝ ✖å➄é✆✡◆ï❙ù✂ì❛é✙ï➉✡☛✘✫➊å➄û➀û➀ø➑é❼✂➽è✧ç✙ï❙î❭ï➊è✧ç✙ì✂ù ✭ ✟✝✑✫✢✖✟✶ì➈é î➻ì✂ù✂ÿ✙û➀ï✳❜❖ó❨ø➩è✥ç è✧ç☎ï✿å➈ä❺✂❛ÿ✙î➻ï➊é✐è✥ê✮✛✣✥✤✏✘✛✩★✙✏✧✛✦✪✎ð➛☞✇ì➈î➻æ✙û➀ï❯↔ î➻ì✂ù☛✝ ÿ✙û➀ï✖ê➝➊å➈é✢✡◆ï✫➊ì➈é☎ê✤è✧ä✥ÿ✗❶è✧ï✖ù ëíä✧ì❛î❅ê✤ø➀î➻æ✙û➑ï✖ä❙î➻ì✂ù✂ÿ✙û➀ï✖ê➚✡☛✘ ê✧ø➑î➻æ✙û✠✘ ù✂ï✄➞☎é✙ø➀é❼✂➲å✿é✙ï➊ó ❶û✓å➈ê✥ê➤ó❨ç✙ì❛ê✧ï➓ê✤û➀ì➄è❿ê➤ó❨ø➑û➀û➜➊ì➈é✐è✥å➈ø➑é❺è✧ç✙ï➓î➻ï✖î➉✡◆ï➊ä î➻ì✂ù✂ÿ✙û➀ï✖ê❀å➈é☎ù➤è✥ç✙ï❫ø➀é✐è✧ï➊ä✥î➻ï✖ù✂ø✓å✠è✥ï❫ê✤è✥å✠è✥ï➏ú✠å➄ä✥ø➀å✑✡✙û➀ï✖ê❢✡◆ï❶è④ó➵ï➊ï✖é➤è✧ç✙ì✐ê✤ï î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð✇ã✛ç✙ï ✭ ✟✝✑✫✢✣✟➽î➻ï➊è✧ç✙ì✂ù✿ëíì➈ä✛è✥ç✙ï➉➊û➀å❛ê✧ê❨ê✧ø➑î➻æ✙û✠✘➙✖å➄û➀û➀ê❨è✧ç✙ï ✭ ✟✝✑✝✢✖✟ î❭ï➊è✧ç✙ì✂ù✙ê❙ì➄ë❨è✥ç✙ï✶î➻ï✖î➉✡◆ï➊ä❭î❭ì✂ù✂ÿ☎û➑ï❞ê✄➌❣ó❨ø➑è✧ç è✥ç✙ï✺å➄æ✙æ✙ä✥ì❄✝ æ✙ä✥ø➀å➄è✧ï✿ø➑é✐è✥ï➊ä✥î❭ï❞ù✂ø✓å✠è✧ï✫ê✤è✥å➄è✧ï✺ú✠å➄ä✥ø➀å✑✡✙û➑ï❞ê✒ì❛ä➻ï❯↔➇è✧ï✖ä✧é☎å➈û❨ø➑é☎æ✙ÿ✂è➽å➄é☎ù ì➈ÿ✙è✧æ✙ÿ✂è❿ê➻å➈ê❭å➄ä❘✂➈ÿ✙î➻ï➊é✐è❿ê➊ð➒✕➔û➑è✧ç✙ì❛ÿ❼✂➈ç❤è✧ç✙ï✺å➄û✠✂➈ì❛ä✧ø➑è✧ç✙î➓ê❙å➄ä✥ï➽ï✖å❛ê❅✝ ø➀û✙✘➑✂➈ï➊é☎ï➊ä❿å➄û➀ø✙➽❞å❄✡✙û➀ï❫è✥ì➞å➄é☛✘➞é✙ï➊è④ó✇ì❛ä✧ô✒ì➄ë☛ê✤ÿ✗❿ç➻î➻ì✂ù✂ÿ✙û➀ï✖ê✒➌➈ø➀é✗❶û➀ÿ☎ù✂ø➀é❼✂ è✧ç☎ì❛ê✧ï❣ó❨ç✙ì❛ê✧ï❣ø➑é❂✾☎ÿ✙ï➊é✥❶ï❹✂❛ä✥å➈æ✙ç♣ç✎å➈ê✏✄✘☛➊û➑ï❞ê✄➌❞ó➵ï❣ó❨ø➑û➀û➄û➀ø➑î➻ø➑è❀è✧ç☎ï➏ù✂ø➀ê❇✝ ❶ÿ✎ê✧ê✧ø➑ì❛é❭è✧ì✌è✥ç✙ï④✖å➈ê✧ï❨ì➄ë✝ù✂ø➀ä✧ï★↔è✧ï❞ù➓å✑✄✘☛➊û➑ø✁✩✂❛ä✥å➈æ✙ç☎ê✛➪íëíï➊ï✖ù✦✝üëíì❛ä✧ó✛å➄ä❿ù é✙ï➊è④ó✇ì❛ä✧ô✂ê✴➶↔ð ☞✇ì➈î➻æ✙ÿ✂è✥ø➑é❼✂✢ù✂ï➊ä✥ø➑ú✠å➄è✧ø➀ú➈ï✖ê➔ø➑é➲å➓î❙ÿ✙û➑è✧ø✙✝♠î➻ì✂ù✂ÿ✙û➀ï✒ê❺✘➇ê✤è✧ï✖îPø✓ê ✓④ÿ☎ê✤è å➈ê❑ê✤ø➀î➻æ✙û➀ï➈ð ✕ ☛❇✡☎å➎❿ô✐ó✛å➄ä❿ù➶æ✙ä✧ì❛æ☎å❄✂✐å✠è✥ø➑ì❛é✤✌❲î❭ï➊è✧ç✙ì✂ù❢➌✫➊å➈û➑û➀ï✖ù ✯ ✟✝✑✝✢✖✟❢➌✎ëíì➈ä➳ï✖å✑❿ç❖î❭ì✂ù✂ÿ☎û➑ï➝❶û✓å➈ê✥ê④✖å➄é↕✡◆ï➻ù✂ï❯➞☎é☎ï✖ù✫ëíì❛ä➳è✧ç☎å➄è➳æ✙ÿ☎ä❇✝ æ◆ì❛ê✧ï➈ð➻ã✛ç✙ï ✯ ✟✝✑✫✢✖✟✫î❭ï➊è✧ç✙ì✂ù❺ì➄ë✛å✢î➻ì✂ù✂ÿ✙û➀ï❭è✥å➈ô➈ï✖ê♣è✥ç✙ï➽ê✧å➈î➻ï❭å➈ä❇✝
CXoC.ob CRE IEEE,AOVEy BEXFV B S2 C3 S4 C5 E F6 4 3 Fig.13.Examples of unusual,distorted,and noisy characters correctly recognized by LeNet-R The grey-level of the output label represents the penalty qighter for higher penalties& guments as the Epzop method.All the derivatives in the used to extend the procedures to networks with recurrent system can be computed by calling the fozop method on all connections. the modules in reverse order compared to the forward prop- agation phase.The state variables are assumed to contain B.Special Modules slots for storing the gradients computed during the back- Neural networks and many other standard pattern recog- ward pass,in addition to storage for the states computed in nition techniques can be formulated in terms of multi- the forward pass.The backward pass effectively computes modular systems trained with Gradient-Based Learning. the partial derivatives of the loss E with respect to all the Commonly used modules include matrix multiplications state variables and all the parameters in the system.There and sigmoidal modules,the combination of which can be is an interesting duality property between the for ward and used to build conventional neural networks.Other mod- backward functions of certain modules.For example,a ules include convolutional layers,sub-sampling layers,RBF sum of several variables in the forward direction is trans- layers,and "softmax"layers [65].Loss functions are also formed into a simple fan-out (replication)in the backward represented as modules whose single output produces the direction.Conversely,a fan-out in the forward direction value of the loss.Commonly used modules have simple is transformed into a sum in the backward direction.The fpzop methods.In general,the fpzop method of a func- software environment used to obtain the results described tion F is a multiplication by the Jacobian of F.Here are in this paper,called SN3.1,uses the above concepts.It is a few commonly used examples.The fpzop method of a based on a home-grown object-oriented dialect of Lisp with fanout (a "Y"connection)is a sum,and vice versa.The a compiler to C. fzop method of a multiplication by a coefficient is a mul- The fact that derivatives can be computed by propaga- tiplication by the same coefficient.The pzop method of a tion in the reverse graph is easy to understand intuitively. multiplication by a matrix is a multiplication by the trans- The best way to justify it theoretically is through the use of pose of that matrix.The zop method of an addition with Lagrange functions [21],[22].The same formalism can be a constant is the identity
✂✁☎✄✝✆✟✞✠✄☛✡✌☞✎✍✟✏✒✑✓✏✂✏✂✏✎✔✖✕☛✄☎✗☛✏✙✘✛✚✙✏✂✁✢✜✤✣✥✣✧✦ ✜✝ 3 4 4 4 4 4 3 8 3 C1 S2 C3 S4 C5 F6 Output ✁❼✿▲❍✪❦➎❬❇➆★❦➚❧✴☎★✱✴❋✩❃✪❴▲✰❅✺✇✷✴⑥✦✳✪❈✄✳✪✺✻✳♦✱✴❴✁❚✪❉✪✿▲✺✵✶✷✹✸✵✶✰❅❉❩❚★✱✴❈✪❉✬❈✪✷✹✿▲✺✻❙✔❆❖✯♦✱✴✸❖✱✴❆r✵✻✰❅✸✶✺❀❆❇✷✹✸✻✸✶✰❅❆r✵✶❴❙✔✸✶✰❅❆❅✷✹❍✹❈✪✿▲❵❅✰❇❉④◗✄❙✔P☛✰❇❤❳✰r✵✻❑ ✁★❦✈✮✏✯✪✰❳❍✹✸✶✰r❙✒❑✁❴▲✰❅▼❯✰❅❴➎✷✴⑥❩✵✻✯✪✰❨✷✹✳★✵✻❃✪✳★✵❀❴➂✱✴◗❄✰❅❴➎✸✻✰❇❃★✸✶✰❅✺✶✰❅❈❯✵✶✺ ✵✻✯✪✰❷❃❄✰❇❈✪✱✴❴✵●❙ ✙❴▲✿▲❍✹✯✄✵✻✰❅✸❀⑥✙✷✹✸✏✯✪✿▲❍✹✯✪✰❅✸❀❃✑✰❅❈♦✱✴❴✵✻✿▲✰❅✺✄✂❖❦ ✂➈ÿ☎î❭ï✖é✐è✥ê✌å➈ê➳è✥ç✙ï ✭ ✟✝✑✝✢✖✟✫î➻ï❶è✥ç✙ì✂ù☛ð➣✕➔û➀û❣è✧ç☎ï➽ù✂ï➊ä✥ø➑ú✠å➄è✧ø➀ú➈ï✖ê➳ø➀é è✧ç✙ï ê❺✘➇ê✤è✧ï✖î✖✖å➄é✌✡◆ï➃➊ì➈î➻æ✙ÿ✂è✥ï✖ù✌✡☛✘t➊å➄û➀û➀ø➑é❼✂✛è✥ç✙ï ✯ ✟✝✑✫✢✖✟♣î➻ï❶è✥ç✙ì✂ù➤ì➈é✒å➈û➑û è✧ç☎ï❫î➻ì✂ù✂ÿ✙û➀ï✖ê❇ø➑é❙ä✧ï✖ú➈ï➊ä❿ê✧ï❣ì➈ä❿ù✂ï➊ä❀➊ì➈î➻æ☎å➈ä✧ï❞ù➳è✧ì➔è✥ç✙ï➏ëíì❛ä✧ó✛å➄ä❿ù➳æ✙ä✥ì➈æ✦✝ å❄✂✐å✠è✥ø➑ì❛é✶æ✙ç☎å❛ê✤ï❛ð❣ã✛ç✙ï➞ê④è❿å✠è✥ï➳ú✠å➈ä✧ø✓å❄✡✙û➀ï✖ê✛å➈ä✧ï➤å➈ê✥ê✤ÿ☎î❭ï❞ù➽è✥ì➵❶ì➈é✐è❿å➄ø➀é ê✧û➑ì➈è✥ê➳ëíì➈ä✌ê✤è✧ì❛ä✧ø➀é❼✂✢è✧ç✙ï➝✂➈ä❿å➈ù✙ø➑ï✖é❛è❿ê④➊ì➈î➻æ✙ÿ✂è✥ï✖ù➷ù✂ÿ✙ä✥ø➀é❼✂✶è✥ç✙ï➝✡☎å✑❿ô❩✝ ó✛å➄ä❿ù✌æ☎å➈ê✥ê✒➌❞ø➀é❙å❛ù✙ù✂ø➑è✧ø➀ì➈é✒è✧ì♣ê✤è✧ì❛ä✥å✑✂➈ï❣ëíì❛ä❯è✧ç✙ï✛ê④è❿å✠è✥ï✖ê❜❶ì➈î➻æ✙ÿ✙è✧ï✖ù✒ø➑é è✧ç☎ï➤ëíì➈ä✥ó➵å➈ä✥ù✶æ☎å➈ê✥ê➊ð❣ã✛ç✙ï✌✡✎å✑❿ô➇ó➵å➈ä✥ù➽æ☎å➈ê✥ê✛ï❯➘✟ï✒❶è✧ø➀ú➈ï➊û✠✘ ➊ì➈î➻æ✙ÿ✂è✥ï✖ê è✧ç☎ï✌æ☎å➄ä✧è✧ø✓å➄û☛ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï❞ê➵ì➄ë❀è✥ç✙ï➤û➑ì✐ê✧ê❅❉✭ó❨ø➑è✧ç✿ä✧ï❞ê✤æ◆ï✒❶è➵è✥ì➓å➄û➀û✟è✧ç✙ï ê✤è✥å✠è✥ï✛úrå➈ä✧ø✓å❄✡☎û➑ï❞ê❯å➈é☎ù❭å➄û➀û➇è✧ç✙ï❨æ☎å➈ä✥å➈î➻ï❶è✧ï✖ä✥ê❯ø➀é✒è✥ç✙ï➔ê❺✘➇ê✤è✧ï✖î✺ð❯ã✛ç✙ï✖ä✧ï ø✓ê✇å➈é✶ø➀é❛è✥ï➊ä✥ï✖ê✤è✧ø➀é❼✂✒ù✂ÿ✎å➄û➀ø➩è❅✘➻æ✙ä✥ì➈æ◆ï➊ä✧è❅✘➝✡✎ï➊è④ó✇ï✖ï➊é➓è✥ç✙ï➉ëíì❛ä✧ó✛å➄ä❿ù➻å➄é☎ù ✡☎å➎❿ô✐ó✛å➄ä❿ù❑ëíÿ✙é✗❶è✧ø➀ì➈é☎ê✿ì➄ë➉➊ï➊ä✧è✥å➈ø➑é❍î➻ì✂ù✂ÿ✙û➀ï✖ê✖ð❫✜☎ì➈ä✿ï❯↔✙å➄î➻æ✙û➀ï✑➌➉å ê✧ÿ✙î ì➄ë✛ê✤ï✖ú➈ï✖ä✥å➈û❇ú✠å➄ä✥ø✓å❄✡✙û➀ï✖ê➳ø➀é❖è✥ç✙ï❭ëíì➈ä✥ó➵å➈ä✥ù➲ù✙ø➑ä✥ï✒❶è✧ø➀ì➈é ø➀ê♣è✧ä❿å➄é✎ê❅✝ ëíì➈ä✥î➻ï✖ù✶ø➑é✐è✧ì➻å❭ê✤ø➀î➻æ✙û➑ï➳ë⑨å➈é✦✝♠ì❛ÿ✂è✌➪íä✥ï➊æ✙û➀ø✠✖å✠è✥ø➑ì❛é✥➶➏ø➀é✢è✧ç✙ï✞✡☎å✑❿ô➇ó✛å➄ä❿ù ù✂ø➀ä✧ï★↔è✥ø➑ì❛é✝ð⑧☞✇ì➈é➇ú➈ï✖ä✥ê✧ï➊û✠✘✑➌❣å➲ë⑨å➄é✦✝⑥ì➈ÿ✂è➻ø➀é è✧ç☎ï✶ëíì❛ä✧ó✛å➄ä❿ù ù✂ø➀ä✧ï★↔è✥ø➑ì❛é ø✓ê❨è✧ä❿å➄é✎ê④ëíì❛ä✧î➻ï✖ù✺ø➀é✐è✧ì✶å✶ê✤ÿ✙î▼ø➀é✫è✧ç✙ï➉✡☎å✑❿ô➇ó✛å➄ä❿ù✿ù✙ø➑ä✥ï✒❶è✧ø➀ì➈é✝ð❨ã✛ç✙ï ê✧ì➄ë➺è④ó✛å➄ä✥ï✌ï➊é➇ú➇ø➑ä✥ì➈é✙î➻ï✖é❛è➔ÿ✎ê✤ï❞ù✺è✧ì➽ì➎✡✂è✥å➈ø➑é➲è✧ç✙ï➞ä✥ï✖ê✧ÿ✙û➑è✥ê♣ù✂ï✖ê❘❶ä✥ø✙✡◆ï✖ù ø➀é✫è✧ç✙ø✓ê♣æ☎å➄æ◆ï➊ä★➌✈✖å➄û➀û➑ï❞ù❖þ✂ñ✟❜✙ð✠➾✑➌◆ÿ☎ê✤ï❞ê❨è✥ç✙ï❭å✑✡✎ì✠ú❛ï➓❶ì❛é✗❶ï✖æ✂è✥ê✖ð✛➏⑥è♣ø✓ê ✡☎å❛ê✤ï❞ù➞ì➈é➻å♣ç✙ì➈î➻ï❯✝❖✂➈ä✥ì✠ó❨é✌ì✑✡✓④ï✒↔è❺✝♠ì❛ä✧ø➀ï➊é✐è✥ï✖ù➞ù✂ø✓å➄û➀ï✒❶è❣ì➄ë✴✗❀ø➀ê✧æ❙ó❨ø➑è✧ç å➝❶ì❛î❭æ☎ø➑û➀ï➊ä✛è✥ì➙☞♣ð ã✛ç✙ï➞ë⑨å➎↔è➳è✧ç☎å➄è➳ù✂ï✖ä✧ø➀ú✠å✠è✧ø➀ú➈ï❞ê✬✖å➄é↕✡◆ï➑➊ì➈î➻æ✙ÿ✂è✥ï✖ù↕✡☛✘✺æ✙ä✥ì➈æ☎å✑✂❛å❄✝ è✧ø➀ì➈é✺ø➀é✿è✧ç✙ï✌ä✥ï➊ú❛ï➊ä❿ê✤ï✞✂➈ä❿å➄æ✙ç✿ø✓ê❨ï✖å❛ê❇✘➓è✥ì➓ÿ✙é☎ù✂ï✖ä✥ê✤è✥å➈é☎ù✢ø➑é✐è✧ÿ☎ø➩è✥ø➑ú❛ï➊û✠✘➈ð ã✛ç✙ï✎✡✎ï❞ê④è❣ó✛å✪✘➉è✥ì ✓④ÿ✎ê④è✥ø➩ë➭✘➞ø➑è❯è✧ç✙ï✖ì➈ä✥ï❶è✧ø✁➊å➈û➑û✠✘➤ø➀ê❇è✥ç✙ä✧ì❛ÿ❼✂➈ç➞è✧ç✙ï➵ÿ☎ê✧ï✇ì➈ë ✗❀å✑✂➈ä❿å➄é❼✂❛ï➤ëíÿ✙é✗❶è✧ø➀ì➈é☎ê❄✞✑❼➾✡✠❖➌ ✞✑✫✑ ✠♠ð➉ã✛ç☎ï❙ê✥å➄î➻ï✌ëíì➈ä✥î➓å➄û➀ø➀ê✧î ➊å➄é➔✡✎ï ÿ☎ê✧ï✖ù✺è✥ì✶ï✄↔✐è✥ï➊é☎ù➲è✧ç✙ï❙æ✙ä✥ì☛➊ï✖ù✂ÿ☎ä✧ï❞ê❨è✧ì✢é✙ï❶è④ó➵ì➈ä✥ô✂ê➔ó❨ø➑è✧ç❺ä✧ï★❶ÿ✙ä✥ä✧ï✖é✐è ❶ì❛é✙é✙ï★↔è✧ø➀ì➈é✎ê➊ð ✬✛ ✝✑➤❼➳❄❪✣ ➢✲ ★➔➯✪✡✔✲✙➳❯➩ ñ➔ï➊ÿ☎ä✥å➈û❛é✙ï➊è④ó✇ì❛ä✧ô✂ê✝å➈é☎ù✌î➓å➄é☛✘➳ì➄è✥ç✙ï➊ä❦ê④è❿å➄é☎ù☎å➄ä❿ù➤æ☎å✠è✧è✧ï➊ä✥é➞ä✥ï✒➊ì✑✂❄✝ é✙ø➑è✧ø➀ì➈é è✥ï✒❿ç✙é☎ø✠➍✐ÿ✙ï❞ê➛✖å➄é②✡◆ï ëíì❛ä✧î❙ÿ✙û➀å➄è✧ï❞ù ø➑éòè✧ï➊ä✥î➓ê❖ì➈ë➽î✒ÿ✙û➑è✧ø✙✝ î➻ì✂ù✂ÿ✙û✓å➄ä✺ê❇✘✂ê✤è✧ï➊î➓ê✶è✧ä❿å➄ø➀é✙ï✖ù✻ó❨ø➩è✥ç❲â➳ä❿å➈ù✂ø➀ï➊é✐è❇✝r✚✛å➈ê✧ï✖ù✩✗✝ï❞å➄ä✥é✙ø➀é❼✂☎ð ☞✇ì➈î➻î➻ì➈é☎û✙✘✾ÿ☎ê✤ï❞ù î➻ì➇ù✙ÿ✙û➑ï❞ê➲ø➀é✗❶û➀ÿ☎ù✂ï❑î➓å✠è✧ä✥ø✙↔✾î✒ÿ☎û➩è✥ø➑æ✙û➀ø✁➊å✠è✥ø➑ì❛é☎ê å➄é✎ù ê✧ø✠✂➈î➻ì➈ø✓ù✙å➄û❫î❭ì✂ù✂ÿ☎û➑ï❞ê✄➌❇è✧ç✙ï➣➊ì➈î➑✡✙ø➑é✎å✠è✧ø➀ì➈é➷ì➈ë➵ó❨ç✙ø✁❿ç✢➊å➈é ✡✎ï ÿ☎ê✧ï✖ù➷è✥ì➔✡✙ÿ☎ø➑û✓ù➹➊ì➈é➇ú➈ï✖é❛è✥ø➑ì❛é☎å➄û❫é✙ï➊ÿ✙ä❿å➄û➵é✙ï➊è④ó✇ì❛ä✧ô✂ê✖ð ý➉è✧ç✙ï✖ä✒î➻ì✂ù☛✝ ÿ✙û➀ï✖ê❇ø➑é✥❶û➀ÿ☎ù✂ï✔❶ì➈é➇ú❛ì➈û➀ÿ✂è✧ø➀ì➈é☎å➈û➈û✓å✪✘➈ï✖ä✥ê✒➌✖ê✧ÿ❼✡✦✝➠ê✧å➈î➻æ✙û➑ø➀é❼✂♣û➀å✪✘❛ï➊ä❿ê✄➌✪✍✛✚✔✜ û✓å✪✘➈ï➊ä❿ê✒➌✝å➄é☎ù ☛✧ê✧ì➄ë➺è✥î➓å♦↔✌✶û✓å✪✘➈ï✖ä✥ê❑✞✓ ✙ ✠♠ð❵✗❀ì❛ê✥ê➉ëíÿ☎é✗↔è✥ø➑ì❛é☎ê➞å➄ä✥ï➓å➄û✓ê✤ì ä✥ï➊æ✙ä✥ï✖ê✧ï➊é✐è✧ï❞ù❖å❛ê♣î➻ì✂ù✂ÿ✙û➀ï✖ê➞ó❨ç✙ì❛ê✧ï❭ê✧ø➀é❼✂➈û➀ï➻ì➈ÿ✂è✥æ✙ÿ✂è✌æ✙ä✥ì➇ù✙ÿ✗❶ï❞ê➳è✧ç✙ï ú✠å➄û➀ÿ✙ï➲ì➈ë♣è✥ç✙ï➲û➀ì❛ê✥ê➊ð②☞✇ì➈î➻î➻ì➈é✙û✠✘❤ÿ✎ê✤ï❞ù➘î❭ì✂ù✂ÿ☎û➑ï❞ê➓ç☎årú➈ï➲ê✧ø➀î❭æ☎û➑ï ✯ ✟✝✑✝✢✖✟✫î➻ï❶è✥ç✙ì✂ù✙ê➊ð➝➏➠é➒✂➈ï✖é✙ï➊ä❿å➄û✶➌✟è✧ç✙ï ✯ ✟✝✑✫✢✣✟✫î➻ï❶è✥ç✙ì✂ù ì➄ë➵å✢ëíÿ✙é✥✹✝ è✧ø➀ì➈é ✼▲ø➀ê➳å➽î❙ÿ✙û➩è✥ø➑æ☎û➑ø✁➊å➄è✧ø➀ì➈é↕✡☛✘✿è✧ç✙ï✗✎❛å➎❶ì➎✡✙ø➀å➈é✫ì➄ë ✼➻ð➳õ➔ï➊ä✥ï❙å➈ä✧ï å✿ëíï➊ó ❶ì❛î➻î❭ì❛é✙û✠✘❖ÿ☎ê✧ï✖ù ï❯↔✙å➈î❭æ☎û➑ï❞ê➊ð➽ã✛ç✙ï ✯ ✟✝✑✫✢✖✟➲î➻ï❶è✧ç☎ì➇ù ì➈ë✛å ë⑨å➄é☎ì➈ÿ✂è ➪üå◗☛■❊✌✫❶ì❛é✙é✙ï★↔è✧ø➀ì➈é✈➶➉ø✓ê➞å✺ê✧ÿ✙î✫➌❇å➄é✎ù❺ú➇ø✁❶ï➻ú➈ï✖ä✥ê✥å✙ð➻ã✛ç✙ï ✯ ✟✝✑✝✢✖✟➻î➻ï❶è✧ç☎ì➇ù✶ì➄ë❀å❙î✒ÿ✙û➑è✧ø➀æ✙û➀ø✠✖å✠è✥ø➑ì❛é➣✡☛✘➽å➉❶ì➇ï✱✯➵➊ø➑ï✖é✐è✇ø✓ê➵å✒î✒ÿ✙û✙✝ è✧ø➀æ✙û➀ø✠✖å✠è✥ø➑ì❛é➣✡☛✘✒è✥ç✙ï♣ê✥å➄î➻ï✬➊ì✐ï✱✯➣❶ø➀ï➊é✐è✖ð❣ã✛ç✙ï ✯ ✟✝✑✫✢✖✟❙î➻ï❶è✥ç✙ì✂ù➓ì➄ë✝å î✒ÿ☎û➩è✥ø➑æ✙û➀ø✁➊å✠è✥ø➑ì❛é➵✡☛✘❭å✌î➓å➄è✧ä✥ø➟↔➻ø➀ê✇å➤î✒ÿ✙û➑è✧ø➀æ✙û➀ø✠✖å✠è✥ø➑ì❛é➣✡☛✘✒è✥ç✙ï❨è✧ä❿å➄é✎ê❅✝ æ◆ì❛ê✧ï❫ì➄ë✂è✥ç☎å✠è❦î➓å✠è✥ä✧ø✙↔☛ð❇ã✛ç✙ï ✯ ✟✝✑✫✢✖✟♣î➻ï❶è✥ç✙ì✂ù➞ì➈ë☎å➄é✒å❛ù✙ù✂ø➑è✧ø➀ì➈é✒ó❨ø➑è✧ç å➝❶ì❛é☎ê④è❿å➄é✐è❨ø➀ê❨è✧ç✙ï✌ø✓ù✂ï➊é✐è✥ø➩è❅✘❛ð