【摘要】 介绍了一种大规模、高性能、分布式的Web信息搜集器的设计及其Java实现.提出了Crawler设计中数据结构、系统功能模块和相关算法新的设计思想;对设计与实现过程中需要解决的关键问题分布式协调机制、基于内存的URL存储管理等进行了讨论,并提供了现阶段的设计、实现方法和分布式无损链接分析算法.更多还原

【Abstract】 Web crawler is the core component of WWW search engine and information retrieval systems. This paper discussed the architecture of a distributed Web crawler and the design ideas about the Web crawler data structure, system modules and related algorithms. The key problems encountered in the design and implementations were also commented, and the solutions to those problems were presented.更多还原