【作者】 张航；

【导师】 严春；

【作者基本信息】 武汉理工大学 ， 计算机应用技术， 2010， 硕士

【摘要】 随着互联网的快速发展,信息的发布与传播变得越来越快捷,网络上的信息量也积累到了非常庞大的规模,这使得检索信息变得日益困难,所幸搜索引擎能够为用户提供快速的信息检索,它被人越来越多的使用并成为日常生活中的一种依赖。网络爬虫作为搜索引擎的抓取模块,负责搜集互联网上的网页,爬虫系统的爬行性能和搜集到的网页质量很大程度上影响了搜索引擎的服务质量,所以爬虫是搜索引擎的重要组成部分,值得研究和改进。近年来,受制于网络的规模,通用爬虫的负担越来越繁重,而专注于特定领域的主题爬虫搜集的信息更有针对性,并且运行效率较高,于是得到了广泛的重视和深入的研究。主题爬虫为爬虫领域开辟了一条新的路径,具有很高的研究价值和实用价值,本文围绕设计和实现一个主题爬虫所涉及的各项技术展开讨论,所做的主要工作和研究成果包括：(1)提出和实现了一种改进的搜索策略,主要是通过动态调整阈值,使爬虫的搜索过程更有灵活性,可以有效解决主题爬虫领域的隧道现象,平衡主题爬虫的搜集覆盖率和准确率,使两者都达到较高的水平；(2)提出和实现了一种改进的相关度计算模型,主要是通过合理压缩空间向量的维度,在不丢失准确度的情况下,减少了计算的时间复杂度,可以有效提高计算速度；(3)研究了当数据规模很大时,爬虫系统如何实现URL的消重和网页的存储,前者主要是通过将索引建立成B-树的结构,从而有效的压缩了维护URL队列所需的空间,后者主要是放弃数据库的存储方式,将网页按照一定的存储格式,以文件方式予以保存,有利于数据的安全和维护；(4)研究了涉及主题爬虫领域的各项技术问题,主要有网页的获取,网页内容的分析,中文分词和搜索策略等,综合上述各方面的理论,设计和实现了个主题爬虫系统,利用该系统对搜索策略和相关度计算的改进方法进行了实验测试,数据显示上述改进方法效果良好。更多还原

【Abstract】 With the rapid development of internet, information publishing and dissemination have become more and more quickly. The amount of information on the network has accumulated to a very large scale, making it increasingly difficult to retrieve information. Fortunately search engines can provide users with rapid information retrieve, which is more and more used in people’s daily life. Web crawler is crawl module of a search engine, whose main task is collecting internet webs. The performance of crawler system and the quality of collected webs seriously affect the service quality of search engines, so crawler system is a very important part of a search engine and it is worthy to be researched and improved. In recent years, subject to the size of the network, the common crawler undertakes more and more onerous burden. However, focused crawler which pays more attention on special topic areas has higher operating efficiency. It has been widely concerned and deeply researched in recent years. Focused crawler opens up a new path for crawler field, with high research value and practical value. This paper mainly discusses various technologies involved in designing and implementing a focused crawler system. The main research and work as follows.(1)Proposed and implemented an improved search strategy, mainly through the dynamic adjustment of threshold, so that the search process would be more flexible. This method can effectively resolve the tunnel phenomenon that commonly exists in focused crawler field. And also it can balance coverage and accuracy of focused crawler system, so that both of them will reach a higher level.(2) Proposed and implemented an improved model of the correlation calculation, mainly through compressing the vector space dimension, which can reduce the computation time complexity and improve the calculation speed without losing much accuracy.(3) Studied when the data size is too large, how the crawler re-works the news URL and how to story webs. The former is mainly through the establishment into a B-tree index structure, which effectively reduced the space required for maintaining URL queues. The latter is mainly to give up the webs stored in the database, in stead of, webs would be stored in documents by the certain format. This is conducive to the maintenance and security of data.(4) Studied various technical issues in focused crawler area, including web access, web content analysis, Chinese word segmentation and search strategies. Summing up all aspects of the above theories, designed and implemented a focused crawler system. Using this system, the improved search strategy and correlation calculation works well verified by relevant experiments data.更多还原