中文分类竞赛准备.pptVIP

An Efficient Centroid Based Chinese Web Page Classifier LIU Hui EE Dept of Tsinghua Univ. China Aug 28, 2003 Outline Background Basic Technique Classifier Design Implementation Idea Architecture Feature Experiment Summary Background of Web Page Classification Explosive information need organization Digital Library Search Engine Special (Categorized) Sites Research hot points Data Mining Information Retrieval Pattern Recognition Text Automatic Categorization Background of Our Classifier Net-compass Search Engine An emerging large and distributed search engine Embedded in its new version Chinese web page categorization competition Held on March 14th –15th, 2003 Ranked first Workgroup EE Dept of Tsinghua Univ., 3 master students 1 undergraduate student Basic Text Categorization System Feature Selection Term Frequency (TF) Term Frequency Inverse Document Frequency (TF.IDF) Mutual Information (MI) Statistics Training - Statistical Machine Learning Vector Distance Centroid Based Method k-Nearest Neighbor: lazy learning Support Vector Machine: Structural Risk Minimization Feedback Combining Classifiers Neuron Network Boosting method Probability Na?ve Bayes: Pr (Term/Class) - Pr(Text/Class) Idea Large Database Net-compass Search Engine Fast Speed Tolerable Precision Web Resource Fast changing Easy building Classifier Fast Training Supporting multi-language Word segmentation Easy Training Set Building Updating Architecture Features Preprocessing Chinese Word Segmentation Dictionary built on search engine log Adaptability, Manageability, Accuracy Maximum Matching Segmenting Method Fast, tolerable accuracy Noise Filtering Stop word: common word, abandon word Advertising links: length content Features Combined Feature Selection Statistics: tend to choose high-freq words Mutual Information: tend to low-freq words Subspace Features Adaptive Factors Adjust model, compensate for deficiency of training set Class Weight VIP word factor Implementa