SVM Eliminate useless feature（s） ClopiNet支持向量机训练消除无用的功能（S） 课件.ppt

Lecture 3:Introduction toFeature Selection Isabelle Guyon isabelle@ Notations and Examples Feature Selection Thousands to millions of low level features: select the most relevant one to build better, faster, and easier to understand learning machines. Leukemia Diagnosis RFE SVM for cancer diagnosis QSAR: Drug Screening Text Filtering Top 3 words of some categories: Alt.atheism: atheism, atheists, morality Comp.graphics: image, jpeg, graphics Sci.space: space, nasa, orbit Soc.religion.christian: god, church, sin Talk.politics.mideast: israel, armenian, turkish Talk.religion.misc: jesus, god, jehovah Face Recognition Feature extraction Feature construction: PCA, ICA, MDS… Sums or products of features Normalizations Denoising, filtering Random features Ad-hoc features Feature selection Nomenclature Univariate method: considers one variable (feature) at a time. Multivariate method: considers subsets of variables (features) together. Filter method: ranks features or feature subsets independently of the predictor (classifier). Wrapper method: uses a classifier to assess features or feature subsets. Univariate Filter Methods Individual Feature Irrelevance P(Xi, Y) = P(Xi) P(Y) P(Xi| Y) = P(Xi) P(Xi| Y=1) = P(Xi| Y=-1) Individual Feature Relevance S2N Univariate Dependence Independence: P(X, Y) = P(X) P(Y) Measure of dependence: MI(X, Y) = ? P(X,Y) log dX dY = KL( P(X,Y) || P(X)P(Y) ) Correlation and MI Gaussian Distribution Other criteria ( chap. 3) T-test Statistical tests ( chap. 2) Multivariate Methods Univariate selection may fail Filters vs. Wrappers Main goal: rank subsets of useful features. Search Strategies ( chap. 4) Sequential Forward Selection (SFS). Sequential Backward Elimination (SBS). Beam search: keep k best path at each step. Floating search (SFFS and SBFS): Alternate betweem SFS and SBS as long as we find better subsets than those of the same size obtained so far. Extensive search (simulated annealing, g