Vanilla rnns RNNs have Memory The computational graph of an rnn can be folded in time Black squares denotes memory access C (Ct-1) C (cr+) a(0 a(4+1) Unfold H (2) (2) awi anl, mAy a(a、UFa) x x(-) x(0 x(+1)
Vanilla RNNs ▪ RNNs have Memory The computational graph of an RNN can be folded in time Black squares denotes memory access
Vanilla rnns EXample Application Slot-Filling Spoken Language Understanding) I would like to arrive Shenyang on November 2nd ticket booking system Destination Shenyang Slot time of arrival: November 2nd Sourceofslidehttp://speech.ee.ntu.edu.tw_/-tilkagk/coursesMl16.html
Vanilla RNNs ▪ Example Application ▪ Slot-Filling (Spoken Language Understanding) I would like to arrive Shenyang on November 2nd . ticket booking system Destination: time of arrival: Shenyang Slot November 2nd Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns EXample Application Slot-Filling Spoken Language Understanding) Solving slot filling by Feedforward network? Input: a word (Each word is represented as a vector) Shenyang→ x Sourceofslidehttp://speech.ee.ntu.edu.tw/-tlkagk/coursesMl16.html
Vanilla RNNs ▪ Example Application ▪ Slot-Filling (Spoken Language Understanding) Solving slot filling by Feedforward network? Input: a word (Each word is represented as a vector) 1 x 2 x 2 y 1 y Shenyang Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns EXample Application 1-of-N encoding How to represent each word as a vector? 1-of-N Encoding lexicon =(apple, bag, cat, dog, elephant The vector is lexicon size apple =[1000 0 Each dimension corresponds to a bag=[01000] word in the lexicon cat=[00100] The dimension for the word is 1, and others are o dog=[00010] elephant =[0000 1 Sourceofslidehttp://speech.ee.ntu.edu.tw/-tlkagk/coursesMl16.hTml
Vanilla RNNs ▪ Example Application ▪ 1-of-N encoding How to represent each word as a vector? Each dimension corresponds to a word in the lexicon The dimension for the word is 1, and others are 0 lexicon = {apple, bag, cat, dog, elephant} apple = [ 1 0 0 0 0] bag = [ 0 1 0 0 0] cat = [ 0 0 1 0 0] dog = [ 0 0 0 1 0] elephant = [ 0 0 0 0 1] The vector is lexicon size. 1-of-N Encoding Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns EXample Application Beyond 1-of-N encoding Dimension for“ Other Word hashing apple a-a-a bag a-a-b 0 cat 0000 a-p-p dog 26X26X26 o-|-e eternal p-p other w=apple W=“ Gandalf w=“ Sauron Sourceofslidehttp://speech.ee.ntu.edu.tw_/-tikagk/coursesMl16.html
Vanilla RNNs ▪ Example Application ▪ Beyond 1-of-N encoding w = “apple” a-a-a a-a-b p-p-l 26 X 26 X 26 … … a-p-p … p-l-e … … … … … 1 1 1 0 0 Dimension for “Other” Word hashing w = “Sauron” … apple bag cat dog elephant “other” 0 0 0 0 0 1 w = “Gandalf” Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html