主题爬行器的研究与实现

【作者】 薛艳珠；

【导师】 徐洁；

【作者基本信息】 电子科技大学 ， 计算机应用技术， 2008， 硕士

【摘要】 随着Web的快速式发展,传统的通用搜索引擎技术暴露出了覆盖率低,资源占用多,更新时间长,结果的针对性低等一系列问题。为了克服通用搜索引擎的不足,满足特定用户针对特定领域的查询需求,以面向主题发现的爬行技术应运而生。主题爬行技术是在传统的搜索引擎面向整个网络爬行的基础上,应用机器学习等智能方法,对爬行做主题性的指导,使爬行程序只下载与某一主题相关的页面。主题爬行技术从90年代发展至今,以高度的目标化和专业化在下一代搜索引擎中占据了一席之地。目前对主题爬行器的研究集中在两个热点——文本分类技术和爬行策略。本文对主题爬行的关键技术——文本分类进行了研究。主题爬行器在进行文本分类时,通常用向量空间法的TF-IDF方法来计算特征项的权重,该方法只考虑了特征项在文档中出现的频率(TF)和以及特征项出现的文档数(DF),而没有考虑到特征项在不同位置出现对文档的重要程度是不同的。针对这一缺陷,本文提出了“基于特征项位置信息的权重算法”,对文档中不同位置的特征项赋予不同的加权因子,从而使特征项的权重能够较客观地反映特征项的重要性,提高了分类准确性。在具体实现中,还可以对权重因子进行调整,使之达到更好的效果。根据最佳优先爬行算法的不足,以及人在浏览网页判断一个URL链接页面的内容所利用的信息,本文提出了“基于URL综合信息的爬行策略”:利用当前页面的主题相关度、当前页面中的超链接所包含的URL的目录层次信息以及链接的锚文本信息对待爬行URL的主题相关性进行预估,根据结果将该URL放入不同优先级的待爬行队列中。对于预测主题相关度低的URL也不丢弃,而是放入等待队列中,待其他爬行队列空闲时,爬行这些URL以发现新的主题相关网页。使用这种爬行策略,可以提高主题相关度预测的准确性并能快速下载主题相关的网页。该方法计算简单,提高了爬行的速度和召回率。最后在上述“基于特征项位置信息的权重算法”和“基于URL综合信息的爬行策略”的基础上,设计和实现了主题爬行器,并介绍了系统的体系结构和具体实现技术。更多还原

【Abstract】 With the rapid development of the web, a series of questions have exposed in the traditional generic search engine, such as low coverage, large resource occupancy rate , long time to update, low pertinence and others. To tackle those problems of generic search engine and to satisfy specifically user subject-oriented query, the topic focused crawling technology came into being. The focused crawling, based on the generic search engine’s technology, applies machine learning and other intelligence methods, to download more related page with low cost. Topic focused crawling technology has occupied a position with its high degree of specialization and objectives in next development of search engine from 1990s.Currently, the researches of topic focused crawling mainly concentrate on two hotspots: Document Categoration and Crawling Strategy.Document Categoration technology is studied in this thesis. Topic focused crawling usually classify documents by the algorithm of TF-IDF of SVM. However, TF-IDF algorithm only takes the term’s frequency (TF) and the document frequency (DF) into consideration. It neglects the term’s positional information and different positional term has different importance to classify the document. To solve this problem,“A algorithm of term weighting based on information of term position”is proposed in this paper: assign different weighting factors to different positional term, therefore the weighting values of the term can reflect the importance of the term objectively. This algorithm improves the veracity of document categorizaation. The weighting factors can be adjusted to get good result.According the disadvantage of Best-First search, and the information people used when they judge whether the hyperlinks are useful to him or not,“A crawling strategy based on comprehensive information of URL”is proposed in this thesis. It calculates the predicted correlation of the URL by analyzing the similarity of current Web page, URL directory information and anchor text of hyperlink, then it put the URL to different priority crawling queue according the URL’s predicted correlation. For the URL having low correlation value, the system doesn’t throw it away, but put it to the waiting queue. When other crawling queues are free, system crawls these URLs for finding new topic Web pages. This crawling strategy is simple, but it improves the efficiency and recall rate.Finally, based on theory of“A algorithm of term weighting based on information of term position”and“A crawling strategy based on comprehensive information of URL”, the design and the implementation of the topic focused crawer, including the system structure and method are detailed in this thesis.更多还原