【作者】 李婷；

【导师】 周涛；

【作者基本信息】 电子科技大学 ， 计算机软件与理论， 2015， 硕士

【摘要】 随着互联网的迅速发展,数据呈现爆炸性增长,同时人们对数据的采集需求也与日俱增。网络爬虫作为数据获取的有效手段被广泛应用于各类系统中,诸如搜索引擎、舆情监控系统等。然而,面向中小规模系统的网络爬虫面临着两大难题:其一,单机爬虫程序抓取速度慢,已有的开源分布式爬虫框架实现复杂,灵活性不强;其二,虽然Ajax技术以异步加载的方式与服务器交换必要的数据能够提高用户界面响应速度,带来良好的用户体验,但是传统的网页抓取方式无法完整的获取应用了这项技术的页面所包含的信息。这些缺失的信息通常具有重要的研究价值。在面向中小规模系统的分布式爬虫中,任务调度算法的优劣将直接影响系统的抓取效率。因此,本文重点对分布式爬虫的任务调度策略和Ajax页面抓取算法进行了研究。在分布式爬虫的任务调度方面,本文重点研究了主从式架构下的任务调度算法。为了保证各个爬行节点的负载均衡以及系统的可扩展性,本文提出了一种基于一致性哈希的均分负载空间算法。该算法采用了不同于机器节点复制的虚拟节点添加方法,以解决在机器数量较少的情况下系统负载不均衡问题。中心节点在把握系统整体运行状态的基础上采用该算法进行任务调度,并在爬行节点发生变化后进行相应的任务调整。通过实验比较,验证了该算法在负载均衡效率方面获得了提高。在Ajax页面抓取方面,由于单个Ajax页面中包含了许多状态,本文首先采用了经典的状态流图对Ajax页面进行建模,然后提出了基于网页主体内容变化的重复状态检测算法,并利用该算法进行有效元素的XPath特征训练,最后将训练后的结果应用到Ajax页面抓取中。通过实验比较,验证了该方法在保证获取所有数据的情况下能够进一步地减少事件触发总数,缩短页面抓取所消耗的时间,提高了Ajax页面抓取效率。最后,本文提出了支持Ajax页面抓取的分布式爬虫系统总体设计方案,并详细介绍了中心节点和爬行节点各模块的实现细节。通过将分布式爬虫系统成功的应用到网络舆情监控项目中,验证了本文所提出技术的有效性。更多还原

【Abstract】 With the rapid development of Internet, various data show explosive growth. At the same time, the demand for data collection is growing. As the effective way of data acquisition, web crawler has been widely applied to a large variety of systems, such as search engine, public opinion monitoring system and so on. But web crawler for small to medium-size system faces two difficult problems: for one thing, deploying crawler system on single machine slows down the speed of data acquisition and the existing open source distributed web crawler frame is complex and lack of flexibility; For another, although Ajax technology which exchanges the necessary data with server in an asynchronous way can improve response speed of user interface and bring good user experience, the traditional web crawling fails to get complete information of the web pages which apply Ajax technology. These missing data usually has a great value for research. In distributed web crawler for small to medium-size system, task scheduling algorithm directly influences the fetching efficiency of system. So this thesis focuses on the study of task scheduling strategy of distributed crawler and Ajax page fetching algorithm.In task scheduling strategy of distributed crawler, this thesis mainly studies the task scheduling algorithm under the master-slave architecture. In order to ensure the load balance and scalability, we propose an averaging load space algorithm based on consistent hashing. The algorithm adopts the different way to add virtual nodes from copying each machine node, in order to solve the load imbalance problem in the case of small number of machines. The center node uses the algorithm to schedule tasks on the basis of knowing the running state of the whole system, and adjusts task allocation of each machine after the number of machines changes. By experimental comparison, the algorithm is verified to increase the efficiency of load balance.As for Ajax page fetching, since one Ajax page contains many states, we firstly refer to a classic state flow graph to model Ajax pages. Then, we raise a new method of repeat states detection based on the change of the main content of page, and use the proposed method to train XPath features of valid elements. Finally, we apply the training results to fetch Ajax pages. By experimental comparison, this method is verified to have the ability to further reduce the total number of triggered events, shorten the consumed time in the case of obtaining all data, improving the efficiency of Ajax page fetching.Finally, we present the overall design of a distributed crawler system which support Ajax page fetching, and introduce modules of the center node and the crawling node in detail. By successfully applying the distributed crawler system to a network public opinion monitoring project, we verify the effectiveness of the proposed technologies.更多还原