主题（topical）crawler及其应用——主题搜索引擎

【作者】 吕韩飞；

【导师】 王申康；

【作者基本信息】 浙江大学 ， 计算机软件与理论， 2005， 硕士

【摘要】 www信息量的飞速增长对通用网络机器人(crawler，spider)和搜索引擎呈现了空前的挑战。越来越多的人希望能够快速且有效地找到自己所需要的信息。文章在jeffheaton的bot包(又称spider)基础上，提出了一个新的文本资源发现系统称为主题crawler(topical crawler)。主题crawler的目标是有选择地查找与预定主题相关的网页，而不像spider一样，收集和索引所有可访问的web页，这样可以避免访问不相关的网页，同时也有助于节省硬件和网络资源。主题crawler对jeffheaton的spider主要在网页的解析，爬行算法等方面进行了改进；并为它添加了中文分词，通用的输入输出接口等功能；同时，在已有的网页更新策略的基础上，提出了一种新网页更新算法，从理论上证明了该算法的有效性。 文章使用网络机器人的一个应用——搜索引擎来检验jeffheaton的spider及主题crawler的性能。主题搜索引擎主要由搜索网页的主题crawler，为所下载网页建立索引库的索引部分和查找部分组成。其中后两部分的功能我们使用开源软件lucene来实现。更多还原

【Abstract】 The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. More and more people hope to achieve the information what they needed quickly and effectively. In this paper we propose a new hypertext resource discovery system called a Topical Crawler based on jeffheaton’ s bot package(spider). Rather than collecting and indexing all accessible Web documents, the goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. It will avoid irrelevant regions of the Web, at the same time, it will lead to significant savings in hardware and network resources. We improve jeffheaton’ s spider such as parsing web pages, crawling algorithm and so on, add two new functions: Chinese analyzer and a general I/O interface, and put forward a new page refreshing strategy based on the exited, we have proved it’ s effectiveness theoretically.We use crawler’ s application-search engine to evaluate the performance of jeffheaton’ s spider and the topical crawler. The topical search engine is composed of three parts: crawler for searching for web pages, indexer for indexing the downloaded pages and the search part for searching the relevant pages with the user query. The later two parts are implemented by lucene (the open source software).更多还原