【作者】 张瑞雪；

【导师】 宋明秋；

【作者基本信息】 大连理工大学 ， 管理科学与工程， 2011， 硕士

【摘要】 随着Web信息资源的爆炸式增长,如何从海量数据中筛选出人们想要的数据就成为了一个富有挑战性的课题。传统的网页信息提取工具大都基于文本信息的匹配,并不能对复杂的结构化网页信息进行准确地比较和取舍。通过挖掘Web网页中的结构特性,也就是DOM树结构,来衡量目标信息和样本信息之间的相似度,并确认所需信息,已成为一种准确迅速提取信息的有效方法。基于DOM树结构的网页相似度度量理论主要有基于节点统计特征,基于链路匹配,基于最少编辑距离,基于简单树匹配等多种度量方法。然而这些方法都或多或少的存在一些问题,节点统计不够系统,链路匹配比较分散,最少编辑距离缺乏层次性,简单树匹配对顺序要求严格,不适合DOM信息的匹配,而且速度慢。针对以上问题,本文提出了新的解析DOM算法,基于DOM树的网页相似性度量算法以及基于DOM相似性的网页信息提取算法。具体研究工作如下：(1)基于数据预提取的DOM树解析算法解析DOM树是计算网页相似性的基础,也是提取网页信息的前提。本文主要提出了基于部分数据预提取的顺序DOM树解析算法以及逆序DOM树解析算法,可以有效地提取大部分网页的DOM树结构。(2)基于DOM树的网页结构相似度度量方法网页的结构相似度,不仅可以衡量两个网页之间的相似性,而且能量化同一个网页内部不同位置信息之间的相似性,进而根据这种相似性提取目标信息。与传统方法不同,本文提出了两种相似度度量算法：基于子树最优自由匹配的递归算法和基于链路简单树匹配的递归算法。(3)基于DOM树相似度理论的网页正文提取网页正文信息在DOM树中具有一定的结构相似性。这种相似性为我们提取正文信息提供了一条思路。通过正文信息在网页中的某些特征确定部分正文,并通过结构相似性寻找其他正文,进而提取所有正文信息。本文以前面的两种相似度度量方法为基础对网页正文信息进行提取。更多还原

【Abstract】 With rapid development of web information sources, how to select datas we need from large data sets is a challenging problem. Traditional tools of extracting web information are based on text matching, and they can not make precise comparisoin or selection. There has been a precise and effective method that information is extracted by mining web structural features. We can measure the similiarity between target information and sample informations, and then confirm the right ones by the similiarity.Generally, the theories of computing web similiarity based on DOM includes theory based on node statistic feature, theory based on root-to-leaf chain matching, theory based on minimal editing distance, theory based on simple tree matching and so on. However, there are some problems or flaws in all of these methods. Node statistic is not systemic, chain matching scattered, minimal editing distance lacking of understanding hierarchy, and simple tree matching strict in order. They are unsuitable for DOM information, and running with slow speed.In order to solve the above problems, this paper supposes a new DOM parsing method, algorithms of calculating the web similiarities based on DOM tree, and algorithms of extracting web information based on the similiarities. Detailed research work is as followed.(1) DOM parsing algorithm based on extracting partial dataParsing DOM tree is not only the basis of calculating web similiarities but also the premise of extracting web information. This paper proposes two DOM parsing algorithms. One is parsing DOM with normal order based on extracting partial data, and the other is parsing DOM with reverse order. Both can parse DOM from most of web pages.(2) Algorithms of web structual similiarities based on DOM treeWeb structural similiarity can not only measure similiarity between two web pages, but also quantize similiarity between informations in different parts of one web page. In this way we extract target informations. Different from trodational methods, this paper proposes two algorithms of measuring the web similiarities. One is a recursive algorithm based on optimal free matching for children trees, and the other is based on path pressed tree.(3) Web main texts extracting based on web structual similiaritiesGenerally, main texts in the same web page are similar in structure. And the similiarities provide us an idea for extracting main texts. We confirm one or two parts of main texts by some text feature, find the other parts by this similiarity, and then extract all of them. This paper uses the two above algorithms of measuring similiarities to extract web main texts.更多还原