FeatureSelection MethodsQiang YangMSC IT 5210DataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques Feature Selection Methods Qiang Yang MSC IT 5210
FeatureSelection Also known as dimensionality reduction subspace learning Two types: subset vs. new featuresi.. ... f. eletonff.. i,. f..FFKF'f.. .. .ratitongf..f...f.. f...fi.. .)TeCu
Data Mining: Concepts and Techniques Feature Selection ◼ Also known as ◼ dimensionality reduction ◼ subspace learning ◼ Two types: subset vs. new features F F‘ F F‘ 1 1 . { ,., ,., } { ,., ,., } j m i n i i i f selection f f f f f f ⎯⎯⎯⎯→ 1 1 1 1 1 . { ,., ,., } { ( ,., ),., ( ,., ),., ( ,., )} i n n j n m n f extraction f f f g f f g f f g f f ⎯⎯⎯⎯→
MotivationThe objective of feature reduction is three-fold:- Improving the accuracy of classification Providing a faster and more cost-effectivepredictors (CPU time)Providing a better understanding of theunderlying process that generated the dataDataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques Motivation The objective of feature reduction is three-fold: ◼ Improving the accuracy of classification ◼ Providing a faster and more cost-effective predictors (CPU time) ◼ Providing a better understanding of the underlying process that generated the data
Filtering methods Assume that you have both the feature Xi and theclass attribute Y Associate a weight Wi with Xi Choose the features with largest weightsInformation Gain (Xi, Y) Mutual Information (Xi, Y) Chi-Square value of (Xi, Y)DataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques Filtering methods ◼ Assume that you have both the feature Xi and the class attribute Y ◼ Associate a weight Wi with Xi ◼ Choose the features with largest weights ◼ Information Gain (Xi, Y) ◼ Mutual Information (Xi, Y) ◼ Chi-Square value of (Xi, Y)
Wrapper MethodsClassifier is considered a black-box: Say KNNLoopChoose a subset of featuresClassify test data using classifierObtain error ratesUntil error rate is low enough (< threshold)One needs to define:how to search the space of all possible variablesubsets ?how to assess the prediction performance of alearner ?5/54DataMining:ConceptsandTechniques
Data Mining: Concepts and Techniques 5/54 ◼ Classifier is considered a black-box: Say KNN ◼ Loop ◼ Choose a subset of features ◼ Classify test data using classifier ◼ Obtain error rates ◼ Until error rate is low enough (< threshold) ◼ One needs to define: ◼ how to search the space of all possible variable subsets ? ◼ how to assess the prediction performance of a learner ? Wrapper Methods