西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (3)交通大学ChenLicli@xjtu.edu.cn2023
Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (3) Natural language processing with deep learning
Outlines1. NNLM2. CBOW3. Skip-gram4. Hierarchical softmax& Negative sampling5. Glove
Outlines 1. NNLM 2. CBOW 3. Skip-gram 4. Hierarchical softmax & Negative sampling 5. Glove
Outlines1.NNLM2. CBOW3. Skip-gram4. Hierarchical softmax& Negative sampling5. Glove
Outlines 1. NNLM 2. CBOW 3. Skip-gram 4. Hierarchical softmax & Negative sampling 5. Glove
Neural NetworkLanguage ModelsReviewthetaskof language modelsx(t)Input: word sequence x(1), x(2)...Output: the probability distribution of the next word P(x(t+1) |x(t),..r(1)NNLM road map (1):HLBL(Mnih,2009)NNLMGloveWord2vec(Turian,(Huang)(Bengio,(Pennington,2010)2012)(Mikolov,2013)C&W2003)2014)(Collobert,2008)1)Task-specificembedding交通大2)X-word2vec3)Understandingandinterpretation
Neural Network Language Models Review the task of language models • Input: word sequence � (1) , � (2) ,., � (�) • Output: the probability distribution of the next word �(� (�+1)|� (�) ,., � (1)) l NNLM road map (1)： NNLM (Bengio, 2003) HLBL (Mnih, 2009) C&W (Collobert, 2008) (Turian, 2010) (Huang, 2012) Word2vec (Mikolov, 2013) Glove (Pennington, 2014) 1) Task-specific embedding 2) X-word2vec 3) Understanding and interpretation
Neural Network Language ModelsReviewthetaskof language modelsInput: word sequence x(1), x(2)... x(t)Output: the probability distribution of the next word P(x(t+1)|x(t),...x(1Neural network model based on window?girlheropened
Neural Network Language Models Review the task of language models • Input: word sequence � (1) , � (2) ,., � (�) • Output: the probability distribution of the next word �(� (�+1)|� (�) ,., � (1)) Neural network model based on window？ girl opened her