逃鸵肿右州.PDF

Tow ards a \Universal Dictionary for Multi-Language Information Retrieval Applications J. Michael Schultz, Mark Y. Lib erman May 22, 2000 1 Intro duction Recent formal evaluations suggest that simple techniques can accomplish high- quality automatic retrieval from multilingual do cument sets, at least for some tasks. These techniques are based on p er-do cument word counts | what in the monolingual case are familiarly called \bag-of-words mo dels | generalized across languages using simple word-by-word translation. However, an imp or- tant practical imp ediment to coverage of a large numb er of languages is the need for translation dictionaries. These are time-consuming and exp ensive to create by hand, and few language pairs have large enough parallel or comparable text corp ora for statistical induction metho ds to b e feasible. In this pap er, we show that an appropriate metric for term selection in a monolingual English corpus allows us to dene a fairly small list, containing ab out ten thousand in ected forms or ab out 7500 lemmas, which works essentially as well (for a particular monolingual do cument classication evaluation) as an unlimited vo cabulary of more than 300,000 word forms do es. We suggest that such a list can b e taken to form the English axis of a sort of \universal dictionary for do cument clas- sication tasks, providing a much more eÆcient path to the addition of new languages. If prop er names can b e treated separately, then the \universal dictionary b ecomes even smaller | ab out 5 thousand terms. Given a new language for which no prior resources of any kind exist, an initial mapp