【作者】 赵勇；

【导师】 卢正鼎；

【作者基本信息】 华中科技大学 ， 计算机应用技术， 2009， 硕士

【摘要】 随着互联网中信息的爆炸式增长,搜索引擎在人们生活中的作用越来越重要,人们对于搜索结果精度要求的提高,使得面向特定领域的垂直搜索引擎的诞生了。对于服务于垂直搜索引擎的主题爬虫,其主要任务就是在一定的时间内尽可能多的采集相关度高的网页信息,从而可以提高搜索引擎的效率,缓解急剧增长的网络资源与有限的存储资源的矛盾。主题爬虫的核心是主题相关度计算的方法和爬行的链接选取策略,现有的主题爬虫系统的相关度计算算法分成基于网页内容、基于网页链接结构和基于分类器预测三种,都不能够很好的适应社会化网络中发展。社会标注作为社会化网络发展的产物,是对网络资源的一种标示,是从网页浏览者角度赋予网页的一种特性。在分析并得出了社会标注的语义性和社会性的基础上,提出了一种基于标注的相关度计算算法,给出了算法的详细计算公式和步骤,由此来计算页面的主题相关度,最终将计算结果用于指导爬虫的爬行。以提出的社会标注的相关度计算算法为核心,实现了一个主题爬虫系统,采用异步IO以及线程池处理技术加速了爬虫的爬行速度,并通过多级缓存,DNS预取以及采用具有高速访问以及高并发访问特性的Berkeley DB等多种方法提高了爬虫爬行的效率。将社会标注的相关度值作为爬行过程中相关性判定的标准,提高相关网页判别的准确性,在相同时间和空间内,增加获取相关页面的数量,并通过与其他算法的实验数据的对比,验证了算法在相关度判断上的效率,提高了主题爬虫的相关网页率,最后给出了需要解决的问题和进一步研究的方向。更多还原

【Abstract】 As the information in Internet keeping growing incredibly, search engine play more and more important roles in our daily life .In this process the precision of the result set of certain search become a very important factor for people to judge and choose to use certain search engine service .For this request, specific search engine get more popular and topic crawler which extent the general crawler replace the latter. Topic crawler need to compute the relevance between the topic and the page crawled. The algorithm of computing the relevance and the policy which determine the order of the URL to be crawled is the two core aspects to be researched about the topic crawler.The relevance computing algorithms in traditional topic crawler have been analyzied in this thesis. Three tradtional topic crawlers which are web-content based topic crawler, web-link based topic crawler and the classified based topic crawler have been studied and discussed. Social annotations as a given property to webpage in social web have the semantic and the sociality which could be used in the relevance computation of the topic and webpage. An algorithm of relevance computing based on the annotation relevance is proposed.Relevance computing based on the social annotations is applied in a topic crawler system. In the implementation the asynchronous IO and thread pool are used to accelerate the crawl speed of the topic crawler, and multi-level cache, DNS prefetech and Berkeley DB which high speed data access and thread-safe in nature etc. are been applied in the system to enhance the efficiency of the crawler to crawl the webpages.especially the algorithm to compute the relevance between the topic and webpage is used to judge the webpage is relevancy or not, it improve the precious of the judgment and under limited time and spaces it could crawl more relevancy webpage.By the comparing of the experimental data, the algorithm proposed have make a impressive improvement and at last also give the problem to be fixed and further search could be made.更多还原