Multivariate linear regression In a general case, given a dataset d with d> 1, we try to learn a model: f(ri=wxi+b, such that f(xiyi We can also use the least square method to estimate w and b Firstly, denote w=(w, b) and x11x12…x1a1 x21x22…x2a1 y m1xm2∵xmd1 m XW=(xiw+bx,wtb,xmw+ b) We want to minimize Xw-yll2 2/3/2021 PATTERN RECOGNITION
Multivariate linear regression In a general case, given a dataset D with 𝑑 ≥ 1, we try to learn a model: 𝑓 𝒙𝑖 = 𝒘𝑇𝒙𝑖 + 𝑏, 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡 𝑓(𝒙𝑖 ) ≅ 𝑦𝑖 We can also use the least square method to estimate w and b Firstly, denote 𝒘 ෝ = 𝒘, 𝑏 𝑇 and 𝐗 = 𝑥11 𝑥21 ⋮ 𝑥𝑚1 𝑥12 𝑥22 ⋮ 𝑥𝑚2 ⋯ ⋯ ⋱ ⋯ 𝑥1𝑑 𝑥2𝑑 ⋮ 𝑥𝑚𝑑 1 1 ⋮ 1 = 𝒙1 𝑇 1 𝒙2 𝑇 1 ⋮ 𝒙𝑚 𝑇 ⋮ 1 ; 𝒚 = 𝑦1 𝑦2 ⋮ 𝑦𝑚 𝐗𝒘 ෝ = 𝒙1 𝑇𝒘 + 𝑏,𝒙2 𝑇𝒘 + 𝑏,… , 𝒙𝑚 𝑇 𝒘 + 𝑏 𝑇 2/3/2021 PATTERN RECOGNITION 11 We want to minimize 𝐗𝒘 ෝ − 𝒚 2
Pre-requisite Matrix differentiation 1. Function is a vector and the variable is a scalar T f(t)=(1(t),2(t)…fn() Definition df/dfi(t)df2(t) dfn(t) dt dt at 2/3/2021 PATTERN RECOGNITION
Pre-requisite Matrix differentiation 1. Function is a vector and the variable is a scalar 𝑓 𝑡 = 𝑓1 𝑡 , 𝑓2 𝑡 , … , 𝑓𝑛 𝑡 𝑇 Definition 𝑑𝑓 𝑑𝑡 = 𝑑𝑓1 (𝑡) 𝑑𝑡 , 𝑑𝑓2 (𝑡) 𝑑𝑡 , … , 𝑑𝑓𝑛(𝑡) 𝑑𝑡 𝑇 2/3/2021 PATTERN RECOGNITION 12
Pre-requisite Matrix differentiation 2. Function is a matrix and the variable is a scalar f1(t)f2()…,fn(t) f1(t)f2(),f2m(t f(t) nI 2 Definition dr d2 d2(0 dfm (t) dJ厂(t) dt n 2/3/2021 PATTERN RECOGNITION
Pre-requisite Matrix differentiation 2. Function is a matrix and the variable is a scalar Definition 2/3/2021 PATTERN RECOGNITION 13 11 12 1 21 22 2 1 2 ( ) ( ),..., ( ) ( ) ( ),..., ( ) ( ) ( ) ( ) ( ),..., ( ) m m ij n m n n nm f t f t f t f t f t f t f t f t f t f t f t = = 11 12 1 21 22 2 1 2 ( ) ( ) ( ) ,..., ( ) ( ) ( ) ,..., ( ) ( ) ( ) ( ) ,..., m m ij n m n n nm df t df t df t dt dt dt df t df t df t df df t dt dt dt dt dt df t df t df t dt dt dt = =
Pre-requisite Matrix differentiation 3. Function is a scalar and the variable is a vector f(x),X=( 5·· Definition T df of of of ax. ax In a similar way f(x)X=(x12x2…,xn) dx ax,a 2/3/2021 PATTERN RECOGNITION
Pre-requisite Matrix differentiation 3. Function is a scalar and the variable is a vector Definition In a similar way 2/3/2021 PATTERN RECOGNITION 14 1 2 ( ), ( , ,..., )T n f x x x x x = 1 2 , ,..., T n df f f f d x x x = x 1 2 ( ), ( , ,..., ) n f x x x x x = 1 2 , ,..., n df f f f d x x x = x
Pre-requisite Matrix differentiation 4. Function is a vector and the variable is a vector 1M2 ,y=[(x)y2(x)…n(x) Definition X X Oy,(x)ay,(x) ay,(x) 2 (x)Oy (x) Oy(x) OX 2/3/2021 PATTERN RECOGNITION
Pre-requisite Matrix differentiation 4. Function is a vector and the variable is a vector Definition 2/3/2021 PATTERN RECOGNITION 15 1 2 1 2 , ,..., , ( ), ( ),..., ( ) T T n m x y x x x = = x x x y y y 1 1 1 1 2 2 2 2 1 2 1 2 ( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,..., n T n m m m n m n y y y x x x y y y d x x x d y y y x x x = x x x x x x y x x x x