|
|
- 2015
融合词频特性及邻接变化数的微博新词识别
|
Abstract:
摘要： 大量的新词伴随着微博的快速发展而产生,这些新词具有传播速度快及与其他词组合方式灵活的特点,而且在进行分词处理时容易被切分为不同的字符串。提出了一种融合词频特性及邻接变化数的微博新词识别方法。该方法首先对大规模的微博语料进行分词,然后将在两停用词间的相邻字串两两组合,根据组合后的字串频率统计取得新词候选串,再通过组合成词规则进行筛选获得候选新词,最后通过词的邻接域变化特性去除垃圾串获得新词。利用该方法在COAE 2014评测任务上进行了新词的发现实验,准确率达到36.5%,取得了较好的成绩。
Abstract: Along with the rapid development of Weibo, a lot of new words have appeared. These words have characteristic that spread fast and flexible combination with other words. They are easy to be cut apart into different string in segmentation processing. Therefore a new word recognition method that combines word frequency characteristics and accessor variety was proposed. The first step was to segment the large scale Weibo sentences into words, and then combine the two adjacent strings between stop words. The new word candidate strings could be obtained according to the string frequency of the combination. After the filtration through the word formation rules, the candidate new words would be found. Finally, through the characteristics of the word accessor variety, the garbage string was removed to get the new words. Experiments of new word recognition on COAE 2014 task 3 show that the accuracy can reach 36.5% and this method has a good performance
| [1] | WANG Aobo, KAN Min-Yen.Mining informal language from Chinese microtext: joint word recognition and segmentation[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2013:731-741. |
| [2] | 郑家恒, 李文花. 基于构词法的网络新词自动识别初探[J]. 山西大学学报:自然科学版, 2002, 25(2):115-119. ZHENG Jiaheng, LI Wenhua. A study on automatic identification for internet new words according to word-building rule[J].Journal of Shanxi University: Natural Science Edition, 2002, 25(2):115-119. |
| [3] | 刘建舟, 何婷婷,骆昌日.基于语料库和网络的新词自动识别[J].计算机应用,2004, 24(7):132-134. LIU Jianzhou, HE Tingting, LUO Changri. Automatic new words detection based on corpus and web[J]. Journal of Computer Applications, 2004, 24(7):132-134. |
| [4] | 何赛克,王小捷,董远,等.归一化的邻接变化数方法在中文分词中的应用[J].中文信息学报,2010,24(1):15-19. HE Saike, WANG Xiaojie, DONG Yuan, et al.Apply normalized accessor variety in Chinese word segmentation[J]. Journal of Chinese Information Processing, 2010, 24(1):15-19. |
| [5] | LING G C, ASAHARA M, MATSUMOTO Y. Chinese unknown word identification using character-based tagging and chunking[C]//Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2003:197-200. |
| [6] | 周正宇, 李宗葛.一种新的基于统计的词典扩展方法[J].中文信息学报, 2001, 15(5):46-51. ZHOU Zhengyu, LI Zongge. A new statistical method of automatic lexicon augmentation[J]. Journal of Chinese Information Processing, 2001, 15(5):46-51. |
| [7] | 邹纲,刘洋,刘群,等.面向Internet的中文新词语检测[J]. 中文信息学报,2004,18(6):1-9. ZOU Gang, LIU Yang, LIU Qun, et al. Internet-oriented Chinese new words detection[J]. Journal of Chinese Information Processing, 2004, 18(6):1-9. |
| [8] | FENG Haodi, CHEN Kang, KIT Chunyu, et al. Unsupervised segmentation of Chinese corpus using accessor variety[C]//Proceeding of the 1st International Joint Conference on Natural Language Processing-IJCNLP 2004. Berlin: Springer, 2005:694-703. |
| [9] | 崔世起,刘群,孟遥,等.基于大规模语料库的新词检测[J].计算机研究与发展,2006, 43(5):927-932. CUI Shiqi, LIU Qun, MENG Yao, et al. New word detection based on large-scale corpus[J]. Journal of Computer Research and Development, 2006, 43(5):927-932. |
