【作者】 王旭； 杜军平；

【Author】 Wang Xu Du Junping (Beijing Key Lab of Intelligent Telecommunication Software and Multimedia,School of Computer Science, Beijing University of Posts and Telecommunications,Beijing 100876)

【机构】 北京邮电大学计算机学院智能通信软件与多媒体北京市重点实验室；

【摘要】 本文对质检总局互联网舆情监控系统中聚焦爬虫进行了研究,给出了一些爬虫算法和关键技术,研究了聚焦爬虫的工作流程、组成、架构。聚焦爬虫由页面相关度评价模块、搜索链接评价模块和页面提取模块组成。本文给出了以上模块的组成及相互关系。针对聚焦爬虫中存在的问题,给出了具体的解决方案。更多还原

【Abstract】 A research about the behavior characteristics of focused crawler and its evaluation algorithms is done in this paper, base on the public opinion supervisory system.And some algorithms and key technique about crawler is given.There are also some work on process,component and architecture of crawler.Focused crawler is made up of page relevance evaluation module,search link evaluation module and page extraction module.The composition and relationship of these modules is represented.At the end of the paper is a specific solution for problems of the focused crawler.更多还原