2.2.Types of SVM SVM models can be classified into 3 types: 1Linear SVM in linearly separable case:When training data is linearly separable,hard margin maximization is used to learn linear classifier,i.e. linear SVM in linearly separable case (hard margin SVM). 2 Linear SVM:When training data is approximately linearly separable,soft margin maximization is used to learn linear classifier,i.e.linear SVM(soft margin SVM). 3 Non-linear SVM:When training data is linearly inseparable,kernel trick and soft margin maximization are used to learn non-linear SVM. A simple model is the basis of a complex model and a special case of a complex model. 15/263
2.2. Types of SVM SVM models can be classified into 3 types: 1 Linear SVM in linearly separable case: When training data is linearly separable, hard margin maximization is used to learn linear classifier, i.e. linear SVM in linearly separable case (hard margin SVM). 2 Linear SVM: When training data is approximately linearly separable, soft margin maximization is used to learn linear classifier, i.e. linear SVM (soft margin SVM). 3 Non-linear SVM: When training data is linearly inseparable, kernel trick and soft margin maximization are used to learn non-linear SVM. ▶ A simple model is the basis of a complex model and a special case of a complex model. 15 / 263
Outline (Level 1-2) SVM Linear SVM in LSC o Convex optimization Linear separability and classifier o Linear SVM in LSC Maximum Margin Types of SVM SV and margin boundary Three Types of Margin Lagrange Duality Modeling of SVM Optimization o Dual method of linear SVM in o Primal optimization problem for LSC 16/263
Outline (Level 1-2) 2 Linear SVM in LSC Linear separability and classifier Types of SVM Three Types of Margin Modeling of SVM Optimization Primal optimization problem for SVM Convex optimization Linear SVM in LSC - Maximum Margin SV and margin boundary Lagrange Duality Dual method of linear SVM in LSC 16 / 263
2.3.Three Types of Margin B 00 0 Samples 4,B,C on the positive side of the separation hyperplane S. A is far away from S,and it is more confident that 4 is positive. C is closer to S,and it is less confident that C is positive. o B is between A and C,and the confidence that B is positive is also between 4 and C. 17/263
2.3. Three Types of Margin Samples A, B, C on the positive side of the separation hyperplane S. A is far away from S, and it is more confident that A is positive. C is closer to S, and it is less confident that C is positive. B is between A and C, and the confidence that B is positive is also between A and C. 17 / 263
Outline (Level 2-3) o Three Types of Margin o Distance of point to hyperplane o Functional margin o Geometric margin 18/263
Outline (Level 2-3) Three Types of Margin Distance of point to hyperplane Functional margin Geometric margin 18 / 263
2.3.1.Distance of point to hyperplane Consider some point x.Let d be the vector from hyperplane H to x of minimum length.Letx be the projection of x onto H.It follows then that: xP=x-d d is parallel to w,so d=ow for some a E R. xEH which implies wxP+b=0,therefore wxP+b=wT(x-d)+b=w(x-aw)+b=0 which implies a=(wx+b)/(ww) 19/263
2.3.1. Distance of point to hyperplane ▶ Consider some point x. Let d be the vector from hyperplane H to x of minimum length. Let x P be the projection of x onto H. It follows then that: x P = x − d ▶ d is parallel to w, so d = αw for some α ∈ R. ▶ x P ∈ H which implies w T x P + b = 0, therefore w T x P + b = w T (x − d) + b = w T (x − αw) + b = 0 which implies α = (w T x + b)/(w Tw) 19 / 263