Vanilla rnns EXample Application dest time of departure Slot-Filling Solving slot filling by Feedforward network? Input: a word (Each word is represented as a vector) Shenyang Sourceofslidehttp://speech.ee.ntu.edu.tw_/-tilkagk/coursesMl16.html
Vanilla RNNs ▪ Example Application ▪ Slot-Filling Solving slot filling by Feedforward network? Input: a word (Each word is represented as a vector) 1 x 2 x 2 y 1 y Shenyang 1 x 2 x 2 y 1 y dest time of departure Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns EXample Application dest time of departure Slot-Filling arrive Shenyang on November 2nd other dest other time time Problem? i leave shenyang on November 2nd place of departure Neural network needs memory Shenyang Sourceofslidehttp://speech.ee.ntu.edu.tw_/-tilkagk/coursesMl16.html
Vanilla RNNs ▪ Example Application ▪ Slot-Filling 1 x 2 Shenyang x arrive Shenyang on November 2nd other dest other time time leave Shenyang on November 2nd place of departure Neural network needs memory! Problem? 2 y 1 y dest time of departure Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns Recurrent Neural Network The output of hidden layer are stored in the memory store Memory can be considered as another input Sourceofslidehttp://speech.ee.ntu.edu.tw/-tlkagk/coursesMl16.hTml
Vanilla RNNs ▪ Recurrent Neural Network 1 x 2 x 2 y 1 y 1 a a2 Memory can be considered as another input. The output of hidden layer are stored in the memory. store Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns Recurrent neural network Input sequence 出{[ 4 4 output sequence:「4 store given Initial 2 2 values 0 0 All the weights are 1", no bias All activation functions are linear Sourceofslidehttp://speech.ee.ntu.edu.tw_/-tilkagk/coursesMl16.html
Vanilla RNNs ▪ Recurrent Neural Network 1 x 2 x 2 y 1 y 1 a 2 a store All the weights are “1”, no bias All activation functions are linear Input sequence: 1 1 1 1 2 2 … … 1 1 0 0 given Initial values 2 2 4 4 output sequence: 4 4 Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html
Vanilla rnns Recurrent Neural Network Input sequence 12 12 output sequence: [4][12 4]L12 store 6 6 All the weights are 1, no bias All activation functions are linear Sourceofslidehttp://speech.ee.ntu.edu.tw/-tlkagk/coursesMl16.hTml
Vanilla RNNs ▪ Recurrent Neural Network 1 x 2 x 2 y 1 y a1 2 a store All the weights are “1”, no bias All activation functions are linear Input sequence: 1 1 1 1 2 2 … … 1 1 2 2 6 6 12 12 output sequence: 4 4 12 12 Source of slide: http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML16.html