2.1.Example 1:Normal Distribution:unknown mean u and variance o2 Example 1:Assuming the ID Gaussian probability density function with unknown mean u and variance o2,which generated N sample x1,x2,...,xN, calculate the maximum likelihood estimates of mean and variance. logarithmic likelihood function: 9=hpe的-nap- k=1 or u,o=-n2)-2a- 10/88
2.1. Example 1: Normal Distribution: unknown mean µ and variance σ 2 Example 1: Assuming the 1D Gaussian probability density function with unknown mean µ and variance σ 2 , which generated N sample x1, x2, ..., xN, calculate the maximum likelihood estimates of mean and variance. ▶ logarithmic likelihood function: L(µ, σ2 ) = lnY N k=1 p(xk ; µ, σ2 ) = lnY N k=1 1 √ 2π √ σ 2 exp(− (xk − µ) 2 2σ 2 ) or L(µ, σ2 ) = − N 2 ln(2πσ2 ) − 1 2σ 2 X N k=1 (xk − µ) 2 10 / 88
(1)Differentiate w.r.t.u,and let be 0: =2 (2)Differentiate w.r.t.o2,and let be 0: +-P-0 k=1 get: 品u=∑-mP 11/88
▶ (1) Differentiate w.r.t. µ, and let be 0: µˆ = 1 N X N k=1 xk ▶ (2) Differentiate w.r.t. σ 2 , and let be 0: − N 2σ 2 + 1 2σ 4 X N k=1 (xk − µ) 2 = 0 get: σˆ 2 ML = 1 N X N k=1 (xk − µˆ) 2 11 / 88
Outline (Level 2) 2 Maximum Likelihood Estimation Example 1:Normal Distribution:unknown mean u and variance o2 Unbiased biased estimates ofu&o2 What is the global optimum of Gauss distribution ML? Exponential famil小y o Example 2:Normal Distribution:known covariance matrix and unknown mean u 12/88
Outline (Level 2) 2 Maximum Likelihood Estimation Example 1: Normal Distribution: unknown mean µ and variance σ 2 Unbiased & biased estimates of µ & σ 2 What is the global optimum of Gauss distribution ML? Exponential family Example 2: Normal Distribution: known covariance matrix Σ and unknown mean µ 12 / 88
2.2.Unbiased biased estimates of u&o2 When N is limited: (1)The above mean estimate is the unbiased estimation of mean. 钢-空2-p -1 (2)The above variance estimate is the biased estimation of variance. 6-哈立e-明-2-明=Vo=（-)p When N is very large: 1-32so2 13/88
2.2. Unbiased & biased estimates of µ & σ 2 ▶ When N is limited: (1) The above mean estimate is the unbiased estimation of mean. E[ˆµ] = E " 1 N X N k=1 xk # = 1 N X N k=1 E [xk ] = µ (2) The above variance estimate is the biased estimation of variance. E[ˆσ 2 ML] = E[ 1 N X N k=1 (xk − µˆ) 2 ] = 1 N X N k=1 E[(xk − µˆ) 2 ] = N − 1 N σ 2 = 1 − 1 N σ 2 ▶ When N is very large: (1 − 1 N )σ 2 ≈ σ 2 13 / 88
We replace the sample mean and variance with other symbols: X=在=∑名，s”=成=员∑出- The expectation of sample variance is: 耶]=∑x-的=时∑(x-川--四1 =∑w-2-2-四∑x-川+-1 =1 ✉1 =∑x-42-20--四+-四9 =∑--R-四9 =∑K-9-G-49=02--门< 14/88
▶ We replace the sample mean and variance with other symbols: X = ˆµ = 1 N X N i=1 Xi , S 2 = ˆσ 2 ML = 1 N X N i=1 (Xi − X) 2 ▶ The expectation of sample variance is: E[S 2 ] = E[ 1 N X N i=1 (Xi − X) 2 ] = E[ 1 N X N i=1 ((Xi − µ) − (X − µ))2 ] = E[ 1 N X N i=1 (Xi − µ) 2 − 2(X − µ) 1 N X N i=1 (Xi − µ) + (X − µ) 2 ] = E[ 1 N X N i=1 (Xi − µ) 2 − 2(X − µ)(X − µ) + (X − µ) 2 ] = E[ 1 N X N i=1 (Xi − µ) 2 − (X − µ) 2 ] = 1 N X N i=1 E[(Xi − µ) 2 ] − E[(X − µ) 2 ] = σ 2 − E[(X − µ) 2 ] < σ2 14 / 88