基于完全二阶隐马尔可夫模型的汉语词性标注.pdfVIP

第31 卷 第 10 期 计 算 机 工 程 2005 年 5 月 Vol.31 № 10 Computer Engineering May 2005 ·人工智能及识别技术· 文章编号：1000—3428(2005)10—0177—03 文献标识码：A 中图分类号：TP18 基于完全二阶隐马尔可夫模型的汉语词性标注 梁以敏，黄德根 （大连理工大学计算机科学与工程系，大连 116023 ） 摘 要：该文基于隐马尔可夫理论，提出了一种三元词汇概率和词性概率相结合的汉语词性标注模型，并对传统的 Viterbi 算法进行了扩展。 对统计模型中出现的数据稀疏问题，给出了基于线性插值法的平滑算法。实验表明，完全二阶隐马尔可夫模型比标准的二元、三元模型有 更高的词性标注正确率和消歧率。 关键词：完全二阶隐马尔可夫模型；汉语词性标注；平滑算法；Viterbi 算法 Chinese Part-of-speech Tagging Based on Full Second-order Hidden Markov Model LIANG Yimin, HUANG Degen (Department of Computer Science and Engineering , Dalian University of Science and Technology, Dalian 116023 ) 【Abstract 】This paper describes an extension to the hidden Markov model for Chinese part-of-speech tagging using second-order approximations for both contextual and lexical probabilities, as well as the traditional Viterbi algorithm is extended. The model makes use of more contextual information than standard statistical models. A smoothing algorithm based on the linear interpolation algorithm is introduced to solve the sparse data problem. The new full second-order HMM is proved to improve Chinese part-of-speech tagging accuracies and disambiguation accuracies over current models. 【Key words 】Full second-order hidden Markov model; Chinese part-of-speech tagging; Smoothing algorithm; Viterbi algorithm 词性标注是自然语言处理的难点之一，在许多应用领 M