3-11 Ordinary least squares The most common method used to fit a line to the data is known as Ols (ordinary least squares) What we actually do is take each distance and square it (i. e. take the area of each of the squares in the diagram) and minimise the total sum of the squares(hence least squares) Tightening up the notation, let v, denote the actual data point t D, denote the fitted value from the regression line u, denote the residual, yt-yt
3-11 Ordinary Least Squares • The most common method used to fit a line to the data is known as OLS (ordinary least squares). • What we actually do is take each distance and square it (i.e. take the area of each of the squares in the diagram) and minimise the total sum of the squares (hence least squares). • Tightening up the notation, let yt denote the actual data point t denote the fitted value from the regression line denote the residual, t t y − y ˆ t y ˆ t u ˆ
3-12 Actual and fitted value
3-12 Actual and Fitted Value y i x x yi i yˆ u i ˆ
3-13 How OLs Works So min. uA+u2+3+u2+i, or minimise 2u. This is known as the residual sum of squares(rss) But what was i It was the difference between the actual point and the line, y-yt o minimising >o-D) is equivalent to minimising ∑ u, with respect to a and B
3-13 How OLS Works • So min. , or minimise . This is known as the residual sum of squares(RSS). • But what was ? It was the difference between the actual point and the line, . • So minimising is equivalent to minimising with respect to $ and $ . 2 5 2 4 2 3 2 2 2 1 u ˆ + u ˆ + u ˆ + u ˆ + u ˆ t u ˆ = 5 1 2 ˆ t t u ( ) 2 − ˆ t t y y 2 ˆ t u t t y − y ˆ
3-14 Deriving the ols estimator Butj,=a+,, so let L=∑(y,-)2=∑(y-a-B,)2 Want to minimise L with respect to(wrt. a and B,so differentiate wrt. d and b a 2∑(y,-a-Bx,)=0 ∑x1(y1--kx1)=0 From(1),∑(y1-a-x,)=0∑y1-a-Bx1=0 But∑ Ty and ∑
3-14 Deriving the OLS Estimator • But , so let • Want to minimise L with respect to (w.r.t.) and , so differentiate L w.r.t. and (1) (2) • From (1), • But and . $ $ $ $ yt xt ˆ ˆ = ˆ + = − − − = t yt xt L 2 ( ˆ ˆ ) 0 ˆ = − − − = t xt yt xt L 2 ( ˆ ˆ ) 0 ˆ 0 ˆ ) 0 ˆ ˆ ( − ˆ − = t − − t = t t t y x y T x yt = Ty xt = Tx = − = − − t t L yt yt yt xt 2 2 ) ˆ ( ˆ ) ( ˆ
3-15 Deriving the Ols estimator So we can write Ty-Ta-tBx=o or y-d-Bx=0(3) From(2),∑x(y-a-x)=0 From(3),a=y-所x (5) Substitute into (4)for a from(5) ∑x;(y2-+所-Bx,)=0 ∑x,y1-υ∑x1+压∑x-B∑x1=0 ∑xy-1x+B1x2-B∑x12=0
3-15 Deriving the OLS Estimator • So we can write or (3) • From (2), (4) • From (3), (5) • Substitute into (4) for $ from (5), y − ˆ − ˆ x = 0 − − = t xt (yt ˆ ˆ xt ) 0 y x ˆ ˆ = − − + − = − + − = − + − = t t t t t t t t t t t t t t x y Tyx Tx x x y y x x x x x y y x x 0 ˆ ˆ 0 ˆ ˆ ) 0 ˆ ˆ ( 2 2 2 0 ˆ Ty −T ˆ −Tx =