基于领域关键词自动扩展的变量主题Hopfield聚焦爬虫的研究

【作者】 彭浩；

【导师】 王雅琳；

【作者基本信息】 中南大学 ， 计算机应用技术， 2008， 硕士

【摘要】 近些年随着互联网信息量的快速增长,如何准确、快速、高效地从互联网上获取信息的问题变得越来越突出,面向主题的聚焦爬虫技术应运而生。现有的面向主题的聚焦爬虫系统均不同程度地存在领域或主题依赖,即主题受限的不足。为了节约资源、提高建立的快速性、增加适应的广泛性,研究一种面向多领域的、具有主题可移植性的聚焦爬虫系统是互联网应用快速发展的需要。本文在分析总结了现有主题聚焦爬虫技术的基础上,提出了常量与变量主题聚焦爬虫的概念。针对变量主题聚焦爬虫中主题特征的动态生成以及网页抓取准确性下降这两个核心问题,提出了一个基于领域关键词自动扩展的变量主题Hopfield聚焦爬虫参考模型,并就其中的关键技术进行了研究。首先,提出了一个改进的网页结构化表示模型DocView-(?),在量化表示中,它考虑了节点影响因子对节点权值的重要影响,以及节点类型对节点权值的不同影响,使得网页的量化表示更加准确,基于此,提出网页关键词自动提取改进算法HTTE-a成功地增强了算法抗主题噪声的能力,在性能上优于现有算法HTTE-b,为实现领域关键词自动扩展,提供关键的技术基础;接着,提出了一个面向Hopfield Web理论模型的综合结构预测相关性分析模型,成功克服了现有模型的计算不稳定问题;最后,提出了一个基于Hopfield人工神经网络的URL搜索算法HNcrawl,它在搜索URL的同时,通过神经元的相关性传播、增强与弱化,结合神经元筛选与剪技策略,不断清晰Web社区的主题边界。本文还给出了变量主题聚焦爬虫系统的一个性能评测标准,并实现了变量主题聚焦爬虫参考模型。实验证明,系统的准确率、效率等主要指标均优于经典的聚焦爬虫,达到了应用的基本要求,本文提出的参考模型是可行的、关键技术是有效的。更多还原

【Abstract】 With the rapid increase of web information, topic-specific focus crawlers have been proposed because of the problem " how to get information correctly, quickly and efficiently ", but now some of this kind of technology could not satisfy the application demand , so in order to conserve resources, to increase the establishment rapidly , and to adapt comprehensively , it’s necessary to research the topic portable focused crawlers, which can be adopted in various fields.This paper produced two definitions such as constant topic-specific crawler and variable topic-specific crawler based on current topic focused crawlers technology, and also produced and researched variable topic focused hopfield crawler reference model based on keywords’ automatically expand, which focused on two key problems: topic feature’s dynamic generation and the decline of web crawling’s accuracy.First, an improved web page structure model named DocView-a is proposed in this paper, in which the node factor’s and the node type’s different influence on node weights is discussed, so that the quantify of web can be more exact. Besides, HTTE-a which is presented here will be better at anti-topic noise, and its quality will be better than HTTE-b, so it’s possible to realize automatically expand of keywords.Secondly, we proposed a new integrated structure prediction correlation analyze model which overcome the fault of instability of calculation.At last, we proposed a new URL search algorithm HNcrawl based on hopfield artificial neural network.While searching URL, it can distinguish the different topic border of web society by relevant traveling, increasing and decreasing of neurons, combining the selection of neurons with the pruning strategy.At the end, we also proposed a performance criteria of variable topic focused Hopfield crawler, and realized variable topic focused crawlers relevant model. It has been tested that the accuracy, efficiency of the system are all better than traditional crawlers, reaching the basical demand. Above all, the reference model in this paper is feasible, and the key technology is efficient.更多还原