Bootstrap Bootstrap Works well with small data sets Samples the given training tuples uniformly with replacement n i.e., each time a tuple is selected, it is equally likely to be selected again and re-added to the training set a Several bootstrap methods and a common one is. 632 bootstrap o Adata set with d tuples is sampled d times, with replacement resulting in a training set of d samples. The data tuples that did not make it into the training set end up forming the test set. about 63.2%of the original data end up in the bootstrap and the remaining 36.8%form the test set(since(1-1/d)d=e1=0.368) o Repeat the sampling procedure k times, overall accuracy of the model: ACC( M) Xi=10.632*Acc(Mi)testset +0.368* Acc Trainset
Bootstrap ◼ Bootstrap ◆ Works well with small data sets ◆ Samples the given training tuples uniformly with replacement i.e., each time a tuple is selected, it is equally likely to be selected again and re-added to the training set ◼ Several bootstrap methods and a common one is .632 bootstrap ◆ A data set with d tuples is sampled d times, with replacement, resulting in a training set of d samples. The data tuples that did not make it into the training set end up forming the test set. About 63.2% of the original data end up in the bootstrap, and the remaining 36.8% form the test set (since (1-1/d) d =e-1=0.368) ◆ Repeat the sampling procedure k times, overall accuracy of the model: 𝐴𝑐𝑐(𝑀) = 1 𝑘 σ𝑖=1 𝑘 (0.632 ∗ 𝐴𝑐𝑐(𝑀𝑖 )𝑡𝑒𝑠𝑡𝑠𝑒𝑡 + 0.368 ∗ 𝐴𝑐𝑐 𝑀𝑖 𝑡𝑟𝑎𝑖𝑛𝑠𝑒𝑡 )
Confidence Intervals a Suppose we have 2 classifiers, M, and M2 which one is better? a Use 10-fold cross-validation to obtain error(M,) and error(M2) a These mean error rates are just estimates of error on the true population of future data cases a What if the difference between the 2 error rates is just attributed to chance? o Use a test of statistical significance e Obtain confidence limits for our error estimates
Confidence Intervals ◼ Suppose we have 2 classifiers, M1 and M2 , which one is better? ◼ Use 10-fold cross-validation to obtain error(M1 ) and error(M2 ) ◼ These mean error rates are just estimates of error on the true population of future data cases ◼ What if the difference between the 2 error rates is just attributed to chance? ◆ Use a test of statistical significance ◆ Obtain confidence limits for our error estimates
Null Hypothesis Perform 10-fold cross-validation a Assume samples follow a t distribution with k-1 degrees of freedom(k=10) a Use t-test (or Students t-test) a Null hypothesis: M, and M2 are the same a If we can reject null hypothesis, then e We conclude that the difference between m, and ma is statistically significant Choose model with lower error rate
Null Hypothesis ◼ Perform 10-fold cross-validation ◼ Assume samples follow a t distribution with k-1 degrees of freedom (k=10) ◼ Use t-test (or Student’s t-test) ◼ Null hypothesis: M1 and M2 are the same ◼ If we can reject null hypothesis, then ◆ We conclude that the difference between M1 and M2 is statistically significant ◆ Choose model with lower error rate
Estimating confidence intervals: t-test a If only 1 test set available: pairwise comparison For ith round of 1o-fold cross-validation the same cross partitioning is used to obtain error(M,) and error(M2) o Average over 10 rounds to get error(M,) and error(M2 T-test computes t-statistic with k-1 degrees of freedom error(M,)-error(M2) varM,-M2/k var(M,-M2)=∑=1ror(M)2-eror(M2)-erorM,-eror(M2)2 a If two test sets available: use non- paired t-test r (M, var(M2) var (1-M2 where k,& k, are of cross-validation samples used for M, and M2 resp
Estimating confidence intervals: t-test ◼ If only 1 test set available: pairwise comparison ◆ For ith round of 10-fold cross-validation, the same cross partitioning is used to obtain error(M1 ) and error(M2 ) ◆ Average over 10 rounds to get error(M1 ) and error(M2 ) ◆ T-test computes t-statistic with k-1 degrees of freedom ◼ If two test sets available: use non-paired t-test where k1 & k1 are # of cross-validation samples used for M1 and M2 resp. 𝑡 = error(M1 ) − error(M2 ) 𝑣𝑎𝑟(M1−M2 )/𝑘 𝑣𝑎𝑟(M1−M2 ) = 1 𝑘 σ𝑖=1 𝑘 [𝑒𝑟𝑟𝑜𝑟 M1 𝑖 − 𝑒𝑟𝑟𝑜𝑟 M2 𝑖 − (error(M1 ) − error(M2 ))] 2 𝑣𝑎𝑟(M1−M2 ) = 𝑣𝑎𝑟(M1 ) k1 + 𝑣𝑎𝑟(M2 ) k2