【作者】 沙芸； 周俊武； 张国英；

【Author】 Sha Yun Zhou Jun-wu Zhang Guo-ying~+ Department of computer science,in Beijing Institute of Petrochemical Technology,Beijing:102617

【机构】 北京石油化工学院计算机系；

【摘要】 去除内容相同或相近的新闻是提高搜索引擎关键技术之一。提出了一种基于主题关键词的新闻去重算法。该方法可基于小规模语料库识别新词,通过以标题为种子点构建词汇链的方法,能够找到对主题贡献大的非高频词,从而抽取出能够表述文档主题的关键词集合。为了在相似新闻中挑选质量高的作为去重结果,需要进行聚类,本文采用动态生成相似文档类别标签的方法进行文档聚类。通过对该类文档总关键词的分析不断调整该类别的文档标签内容,从而适应变化的文档集合。最后,从相似文档类中选出一个或多个作为文档去重结果。本文所采用的聚类方法的好处是无需叠代计算,对文档只扫描一次。实验结果表明与传统方法相比排斥错误率降低了5%左右,运算速度提高了15-20%。更多还原

【Abstract】 Weeding out duplicated news is an important technique of search engine.Based on keywords,the paper proposes a new algorithm to weed duplicated news.The algorithm uses small scale corpus to recognize unknown words,and build lexical chain by title as seeds,obtain integrated keywords set by screening out important but low occurrence words.In order to improve the speed and quality of weeding,the news is clustered based on keyword by dynamic cluster label.To gain the cluster label exactly,the label is adjusted according to the key words of this cluster.After that,one or few news are selected as weeding result according to their quality.The advantage of the cluster algorithm lies in scanning the news only once rather than iteratively calculated.The experimental result shows that exclusive error rate of this algorithm is lower 5% than that of classical algorithms, and the time of weeding duplicated news drops 15-20%.更多还原