The space of choices is large0.0,0.01,0,000,1,0,00,0,1,00,0,0,11,0,1,00,1,1,01,0,0,10,1,0.11,1,0,00,0,1.11,0,1,11,1,1,00,1,1,11,1,0,11,1,1,1Kohavi-John,1997n features, 2n possible feature subsets!DataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques The space of choices is large n features, 2n possible feature subsets! Kohavi-John, 1997
Comparsion of filter and wrapper methodsforfeature selection: Wrapper method (+: optimized for learningalgorithm) tied to a classification algorithm very time consumingFiltering method (+: fast) Tied to a statistical method not directly related to learning objectiveDataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques Comparsion of filter and wrapper methods for feature selection: ◼ Wrapper method (+: optimized for learning algorithm) ◼ tied to a classification algorithm ◼ very time consuming ◼ Filtering method (+: fast) ◼ Tied to a statistical method ◼ not directly related to learning objective
Feature Selection using Chi-SquareQuestion: Are attributesOutlook TempreatureHumidityWindy ClassA1 and A2 independent?hotNhighfalsesunnyIftheyareveryNhothightruesunnyPhighfalseovercasthotdependent, we canPmildhighfalserainPremove eithercoolfalserainnormalNcoolrainnormaltrueA1 or A2PovercastcooltruenormalNmildfalsehighsunnyIf A1 is independentPcoolnormalfalsesunnyPon a class attribute A2rainmildfalsenormalPmildtruesunnynormalwe canPhighovercastmildtruePovercasthotnormalfalseremove A1 from ourNrainmildhightruetraining dataData Mining:ConceptsandTechniques
Data Mining: Concepts and Techniques Feature Selection using Chi-Square ◼ Question: Are attributes A1 and A2 independent? ◼ If they are very dependent, we can remove either A1 or A2 ◼ If A1 is independent on a class attribute A2, we can remove A1 from our training data Outlook Tempreature Humidity Windy Class sunny hot high false N sunny hot high true N overcast hot high false P rain mild high false P rain cool normal false P rain cool normal true N overcast cool normal true P sunny mild high false N sunny cool normal false P rain mild normal false P sunny mild normal true P overcast mild high true P overcast hot normal false P rain mild high true N
Chi-Squared Test (cont.)Question: Are attributes A1 and A2 independent?These features are nominal valued (discrete)-Null Hypothesis: we expect independenceOutlookTemperatureHighSunnyCloudyLowSunnyHighDataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques Chi-Squared Test (cont.) ◼Question: Are attributes A1 and A2 independent? ◼These features are nominal valued (discrete) ◼Null Hypothesis: we expect independence Outlook Temperature Sunny High Cloudy Low Sunny High