Language Models Formal Language(model) Traditional generative model: generates strings Finite state machines or regular grammars, etc Example: I wish I wish i wish I wish i wish i wish wish I wish i wish i wish i wish *wish i wish
Language Models 6 Formal Language (Model) ▪ Traditional generative model: generates strings ▪ Finite state machines or regular grammars, etc. ▪ Example: I wish I wish I wish I wish I wish I wish I wish I wish I wish I wish I wish … *wish I wish
Language Models Stochastic Language Models Models probability of generating strings in the language(commonly all strings over alphabet 2) Model m 0.2 the the man likes the woman 0.1 0.01 man 0.20.010.020.20.01 0.01 woman 0.03 said multiply 0.02 likes P(S|M)=0.0000008
Language Models 7 Stochastic Language Models ▪ Models probability of generating strings in the language (commonly all strings over alphabet ∑) 0.2 the 0.1 a 0.01 man 0.01 woman 0.03 said 0.02 likes … the man likes the woman 0.2 0.01 0.02 0.2 0.01 multiply Model M P(s | M) = 0.00000008
Language Models Stochastic Language Models Model probability of generating any string Model mi Model m2 0.2 the 0.2th 0.0001cass the class pleaseth yon maiden 0.01 class 0.0001 sayst 0.03 sayst 0.2 0.010.00010.00010.0005 0.001 oleasethll0.02 pleaseth)0200100201001 0.0001yon yon 0.0005 m aiden 0.01 m aiden (SM2)> P(SM1) 0.01 wom an 0.0001 woman 8
Language Models 8 Stochastic Language Models ▪ Model probability of generating any string 0.2 the 0.01 class 0.0001 sayst 0.0001 pleaseth 0.0001 yon 0.0005 maiden 0.01 woman Model M1 Model M2 the class pleaseth yon maiden 0.2 0.01 0.0001 0.0001 0.0005 0.2 0.0001 0.02 0.1 0.01 P(s|M2) > P(s|M1) 0.2 the 0.0001 class 0.03 sayst 0.02 pleaseth 0.1 yon 0.01 maiden 0.0001 woman
Language Models Stochastic Language Models A statistical model for generating text Probability distribution over strings in a given language → P(o●oM)=P(。|M (o|M,● P(。|M,●d P(。|M,●o
Language Models 9 Stochastic Language Models ▪ A statistical model for generating text ▪ Probability distribution over strings in a given language M P ( | M ) = P ( | M ) P ( | M, ) P ( | M, ) P ( | M, )
Language Models Unigram and higher-order models ○0●● P()P(o|P(●oP(。l●o刂 Unigram Language models Easy P()P(o)P(●)P( Effective Bigram( generally n-gram) Language models P(°)P(ol●)P(olo)P(●●) Other Language models Grammar-based models(PCFGs),etc Probably not the first thing to try in ir
Language Models 10 Unigram and higher-order models ▪ Unigram Language Models ▪ Bigram (generally, n-gram) Language Models ▪ Other Language Models ▪ Grammar-based models (PCFGs), etc. ▪ Probably not the first thing to try in IR = P ( ) P ( | ) P ( | ) P ( | ) P ( ) P ( ) P ( ) P ( ) P ( ) P ( ) P ( | ) P ( | ) P ( | ) Easy. Effective!