测试集.ppt

测 试 集（Test Collection） 闫宏飞，北京大学信息科学技术学院 2004年12月2日 Outline The TREC (Text REtrieval Conference) Collection CWT100g (Chinese Web Test collection with 100 gigabyte web pages) What is TREC? A workshop series that provides the infrastructure for large-scale testing of (text) retrieval technology realistic test collections uniform, appropriate scoring procedures a forum for the exchange of research ideas and for the discussion of research methodology TREC Conference Established in 1992 to evaluate large-scale IR Retrieving documents from a gigabyte collection Has run continuously since then TREC 2004(13th) meeting is in November Run by NIST’s Information Access Division Probably most well known IR evaluation setting Started with 25 participating organizations in 1992 evaluation In 2003, there were 93 groups from 22 different countries Proceedings available on-line ( ) Overview of TREC 2003 at /pubs/trec12/papers/OVERVIEW.12.pdf TREC General Format TREC consists of IR research tracks Ad hoc, routing, confusion ( scanned documents, speech recognition ), video, filtering, multilingual ( cross-language, Spanish, Chinese ), question answering, novelty, high precision, interactive, Web, database merging, NLP, … Each track works on roughly the same model November: track approved by TREC community Winter: track’s members finalize format for track Spring: researchers train system based on specification Summer: researchers carry out format evaluation Usually a “blind” evaluation: research do not know answer Fall: NIST carries out evaluation November: Group meeting (TREC) to find out: How well your site did How others tackled the program Many tracks are run by volunteers outside of NIST (e.g. Web) “Coopetition” model of evaluation Successful approaches generally adopted in next cycle TREC主要测试项目运作图 Yearly Conference Cycle Creating a test collection for an ad hoc task Creating Relevance Judgments Documents Must be representative of real task of interest Genre Diversity (subjects