【摘要】 针对当前分布式爬虫任务调度策略存在的节点任务分配不均、发生异常后引起的负载均衡问题,提出了一种基于SP-cycle算法的节点地址空间分配策略。通过该策略构造环形序列取代传统散列函数的节点地址空间映射方法,可使所有抓取服务器分配的地址空间达到动态均等,从而实现可调度节点的负载均衡。实验结果表明,相对于传统一致性哈希方法,改进的任务调度策略具有更好的负载均衡效果,并且发生异常时具有较好的健壮性,有效地提升了分布式爬虫整体性能。更多还原

【Abstract】 The task scheduling strategy is the problem to the distributed crawler performance,and the node load balancing is becoming the focus in task scheduling strategy research. An improvement is made on the basis of the task scheduling strategy of consistent hashing. A new node address allocation strategy is proposed based on SP-cycle algorithm rather than the original hash functions to make all servers allocate address space equally so as to realize the load balance. The experimental results show that the improved task scheduling strategy will perform better load balancing,and will appear better robustness in the breakdown conditions. In this way,the performance of the distributed crawler would be improved effectively.更多还原