一种构建信息检索系统评测集方法.doc

本科生学位论文 题目：一种构建信息检索系统评测集的方法 姓 名： 学 号： 院 系：信息科学技术学院 专 业：计算机科学 导 师： 教授 二零零九年五月 摘要 随着互联网的飞速发展，信息检索技术在实际生活中的作用越来越重要，在学术界也引起了研究学者的重视。参考国外测试集的构建经验，天网实验室构建了大规模中文网页信息检索测试集CWT，并组织了SEWM中文网页检索评测，希望在国内外各个研究小组的共同参与下建立并完善CWT，一起推动中文检索技术的发展。’s life. Meanwhile, it has become a crucial research topic across different research entities. Following the existing test collection framework established by foreign research conference (especially TREC Test REtrieval Conference), TianWang research team has constructed a large-scale Chinese Web Test collection (CWT), and is organizing SEWM Chinese Web search evaluation on a yearly basis. Test collections is of great importance in the study of Information Retrieval, it encompass corpora of documents, sets of topics and relevance judgment indicating which documents are relevant to which topics. Topics could be retrieved from current web user log, or established by annotators experienced in certain field of study. And accurate estimation of information retrieval evaluation metrics such as Average Precision require large sets of relevance judgment. Building sets large enough for evaluation of real world implementation is at best inefficient, at worst infeasible. In the work, we tried to come up with an algorithm that requires minimal human effort in gaining an appropriate topic set as well as relevance judgment set. We firstly conducted a close study of web search engine user log, especially the distribution on clicks, frequency and length of searching items. Later, with a smaller set of sampling is tested on different ranking algorithm, which could be viewed as different information retrieval system. Using what has been discovered about the AP (Average Precision), MAP (Mean Average Precision) metrics variation, we came to find a method that is better at distinguishing good IR systems from worse ones in a high confidence of evaluation outcome and within an competitively short time period. Keywords: Information Retrieval, Evaluatio