一种快速高效的文本分类方法-智能科学与人工智能.PDF

一种快速高效的文本分类方法 石志伟1,2 2 2 ， 刘涛 ， 吴功宜 （1.天津师范大学计算机与信息工程学院，天津 300074 ；2.南开大学信息技术科学学院，天津300071 ） shi.zw@ 摘要：近年来，垃圾邮件成为困扰人们的一个严重问题，解决此问题的方案之一是：利用文本分类方法进 行基于内容的邮件过滤。传统的文本分类研究通常侧重于分类效果而非分类分类效率，因而许多分类方法 并不适合实时性要求较高的邮件过滤。本文讨论了两个常用的文本分类算法：向量空间法和 k 近邻方法。 前者速度快，但是分类精度通常不能令人满意。后者则相反，它在分类时要花费更多的时间，但分类效果 要好很多。我们通过将两者的优点有效地结合提出了一个新的文本分类算法：向量空间法和 k 近邻的组合 方法。实验表明，新算法能在较少的时间复杂度上达到甚至超过k 近邻的分类效果。 关键字：文本分类；向量空间法；k 近邻 中图法分类号：TP181 文献标识码：A An Effective and Efficient Algorithm for Text Categorization Shi Zhi-wei1,2 2 2 , Liu Tao , Wu Gong-yi (1.College of Computer and Information Engineering, Tianjin Normal University, Tianjin 300074; 2.Department of Information Science, Nankai University, Tianjin 300071) Abstract ：In recent years, spam has become one of the severe problems that disturb us. One effective solution to this problem is content-based email filter using text categorization (TC) method. Classical TC methods often stress effectiveness rather than efficiency, so they can not perfectly serve the email filtering application which requires both effectiveness and efficiency. In this paper we discuss two popular algorithms for text categorization: Vector Space Model (VSM) and k Nearest Neighbor (kNN). The former is a simple and fast algorithm, but its precision is often not satisfying. On the contrary, the latter spends much time determining the class label of a query document, but often gains better categorization performance. We propose a new algorithm, hybrid of VSM and kNN, by combining the strength of these two algorithms effectively. We also perform an experimental evalua