UnsupervisedOverlappingFeatureSelectionfor.docxVIP

  Unsupervised Overlapping Feature Selection for Conditional Random Fields Learning in Chinese Word Segmentation for Rocling 2011 Ting-hao Yang, Tian-jian Jiang, Chan-hung Kuo , Richard Tzong-han Tsai, Wen-lian Hsu Institute of Information Science, Academia Sinica Department of Computer Science Engineering, Yuan Ze University Term Contributed Boundary Feature using Conditional Random Fields in 2010 A unified view of several unsupervised feature selection based on frequent strings 2 Introduction 3 Flow chart 4 Flow chart SRILM YASA 5 Toolkit C++ libraries The toolkit supports N-gram statistics for language model 6 SRILM Automatically extract frequent strings from unlabeled corpus 7 YASA Pattern: 自然科學 Frequency Net Frequency (自然科學 ,4) (自然科,4) (自然,10) (自然科學 ,4) (自然科,0) (自然,6) 8 Flow chart Character Label 反 B1 而 E 會 S 欲 B1 速 B2 則 B3 不 M 達 E 9 6-Tag [0 -9 ] + [B1|B2|B3|M|E|S] 10 Extended Label N-Gram score Frequent string score Accessor variety score 11 Score Convert from term frequency and N-Gram frequency Logarithm ranking mechanism 12 Score Pattern Frequency Logarithm ranking mechanism Score 塑膠原料的 10 log2(10) =3 塑膠原料 5 log2(5) = 2 原料的 3 log2(3)=1 的生產 2 Log2(2)=1 塑膠 4 log2(4)=2 13 Score Consider the score of outer pattern Equation of AV 14 Score 15 Score AV(開發與法制) AV(的開發與法制),AV(是開發與法制),AV(有開發與法制),AV(開發與法制的),AV(開發與法制是),AV(開發與法制為) Pattern Logarithm ranking mechanism Score 6-Tag Label Label with score 塑膠原料的 log2(10) =3 塑 B1 膠 B2 原 B3 料 M 的 E 塑 3B1 膠 3B2 原 3B3 料 3M 的 3E 16 Score Scores are also used for filtering overlapping pattern 17 Overlapping and Non-overlapping Character TCB Feature 塑 B1 膠 B2 原 B3 料 M 的 E 生 -1 產 -1 18 Non-overlapping “塑膠原料的” score 3 conflicts with ”的生產” score 1 ”的生產” is labeled as unseen Term Label 反 5S3B14B1 而 6S3E4B2 會 6S4E 欲 4S 速 4S 則 6S3B1 不 7S3E 達 5S3E 19 Overlapping information? 20 Overlapping String Input Unsupervised Feature Selection 1 char 2 char 3 char 4 char 5 char 反 5S 3B1 4B1 0B1 0B1 而 6S 3E 4B2 0B2 0B2 會 6S 0E