【作者】 龚诚；

【导师】 秦兵；

【作者基本信息】 哈尔滨工业大学 ， 计算机科学与技术， 2007， 硕士

【摘要】 网页增量采集系统是信息检索领域中一个重要的研究方向。增量采集的目的是采集变化的网页、新出现的网页和消失的网页,其中最重要的就是新出现网页的增量采集。这种采集方式能有效地缩短采集周期,实时更新网页数据,所以在大规模搜索引擎和垂直搜索中有着广泛的应用。本文针对新网页增量采集过程中的三大任务,即增量采集网页树、主题团网页采集和网页树剪枝进行了深入的研究,并实现了一个增量采集新网页的系统。由于以网站首页为根的网页树中各层次的网页类型不同,因此无法直接按照层次进行采集。在增量采集网页树中,本文提出了基于网页树的增量采集,首先通过目录型网页识别技术识别出网站中的目录型网页,得到以目录型网页为根的网页树,然后应用目录型网页更新时间预测技术计算出采集网页树的时间。通过这种处理,将不同更新周期的网页分成不同的网页树,而且网页树中每层的网页类型相似,便于按层采集。在主题团网页采集中,由于从目录型网页向下采集一层的方法只能采集到主题团中的一个网页,主题团中的其他网页都被漏掉了,所以造成了对新网页的采集召回率不高。本文介绍了主题团的定义、主题团的链接特征以及主题团中网页的采集方法。在网页树剪枝中,由于并不是网页树的每个分支都是新网页,为了避免采集旧的网页分支,通过已访问URL缓存和URL中日期识别技术对网页树进行剪枝,由此提高了对新网页采集的准确率。应用了上述技术的增量网页采集系统对网站中新出现的网页的采集准确率和召回率都得到了很大的提高,本系统对新网页采集的平均准确率是92.12%,平均召回率是93.38%。更多还原

【Abstract】 Incremental crawler is an important research issue in the field of information retrieval. The aim of incremental crawler is to gather changed pages, new pages and died pages, in which the most important part is the new pages. This method can reduce the gathering period, and update the pages up to the minute, so it is widely used in large-scale search engines and vertical search engines. This thesis makes an intensive study of the three stages of incremental crawler for new pages: incremental crawling web pages’trees, the content pages’groups and the pruning for web pages’trees.Because the types of pages in different depth are different, the crawler can not gather pages according to depth in website. In the part of incremental crawling web pages’trees, this thesis proposes a gathering method based on web pages’trees. First, the crawler recognizes the index pages in websites, and gets the web pages’trees whose roots are index pages. By this way, the crawler divide pages into several sets of pages according to different updating period, and there is only one type of pages in the same depth, they can be gathered easily.In the part of gathering the content pages’groups, the crawler can only get one content page in content pages’group from the index page, the other pages in the same content pages’group are lost. So the gathering recall is not high. This paper introduces the definition of content pages’groups, the link relation of content pages’groups and the gathering method for content pages’groups.In the part of pruning for web pages’trees, because not every branch of the web pages’trees is new page, the crawler can get a high gathering precision by cutting the old branch by visited URL storage technology and date recognition technology in URL.By applying the technology mentioned above, the gathering precision and recall for new pages increase largely, the average precision achieves 92.12% and the average recall achieves 93.38%.更多还原