针对大规模语料语言模型研究新进展.pdf

技术支持:风淋室/ 净化台/modpic-13/ 计算机研究与发展 ISSN1000—1239／CN 11—1777／TP JournalofComputerResearchandDevelopment 46(10)：1704-1712，2009 面向大规模语料 的语言模型研 究新进展 骆卫华 刘 群。 白 硕。 (中国科学院研究生院 北京 100049) 。(中国科学院计算技术研究所智能信息处理重点实验室 北京 100190) 。(上海证券交易所 上海 200120) (1uoweihua@ ict．ac．cn) A Review oftheState-of-the-ArtofResearch on Large-ScaleCorpora Oriented LanguageM odeling LuoWeihua ，LiuQun ，andBaiShuo。 (GraduateUniversityofChineseAcademyofSciences，Beijing100049) (KeyLaboratoryofIntelligentInformationProcessing，InstituteofComputingTechnology，ChineseAcademyofSciences， Beijing 100190) 。(ShanghaiSecuritiesExchange，Shanghai200120) Abstract N—gram languagemodel(IM ) isa key componentin many research areasofnatural language processing， such as statistical machine translation， information retrieval， speech recognition，etc． Using higher—ordermodelsandmore training data can significantly improve the performanceofapplications．However，forlim itedresourcesofthesystems (e．g．，memory，usageof CPU ，etc)，thecostoftrainingandaccessinglarge—scaleIM becomesprohibitivewithmoreandmore monolingualcorporaavailable．Therefore，theresearchon large—scalelanguagemodelingdrawsmore attention．Theauthorsintroduce the state—of-the—artoftheideasand progressofthe issue，which focusesonsomerepresentativeapproaches，includinganadhocmethod，arandomizedrepresentation modelandadistributedparallelframework．Theadhocmethod isaunifiedoneintegrating division andconqueringofdata，compactdatastructrue，datacompressionbasedon quantizationandmemory mapping．TherandomizedrepresentationofLM isalossycompressionmodelbasedonBloom filter． ThedistributedparallelframeworkcarriesoutthetrainingofLM basedonM apReduceandperforms therequestsofN—gramsinabatch modeofremotecal1．Theperformanceofsystemsofstatistical machinetranslation utilizing theapproachesisdescribed respectivelyw