【作者】 樊勇； 郑家恒； 张虎；

【Author】 Fan Yong Zheng Jia-heng Zhang Hu 1.Department of Computer and Information Technology,Shanxi University,Taiyuan 030006; 2.Key Laboratory of Education for Computation Intelligence and Chinese Information Processing,Taiyuan 030006

【机构】 山西大学计算机与信息技术学院； 计算智能与中文信息处理省教育部重点实验室；

【摘要】 针对互连网上近似镜像网页严重降低搜索引擎效率的现象。本文提出了一种去重的方法。该方法通过句子在文本中的位置和组块的重要度,提取出网页正文的主题句向量,然后对主题句向量进行语义相似度计算,把重复的网页去除。实验证明,该方法在提高运算效率的同时,保证了较高的准确率和召回率。更多还原

【Abstract】 Similar web pages that search engine returns not only waste storage resources but also increase the burden on web users.In this paper,a method to detect similar web pages is proposed.This method picks up topic sentence vector of web pages through location of the sentence in the text and importance of chunking.Then it detects the similar web pages by semantic calculate similar degree of topic sentence vector.The experiment results show that not only completely similar web pages are detected accurately but also partly similar web pages are detected exactly.更多还原