基于主题划分的Web文档自动摘要研究

【作者】 陈志敏；

【导师】 沈洁；

【作者基本信息】 扬州大学 ， 计算机应用技术， 2006， 硕士

【摘要】 随着Internet的迅速发展,Web上的各种数据急剧增加,网络已经成为数据的仓库和知识的潜在源泉。用户如何才能快速有效地获取和利用这些资源,已经成为迫切需要解决的一个问题,而信息检索和自动摘要是目前解决这一难题的关键技术。自动摘要技术将文档内容以简洁精练的形式呈现出来,是信息检索技术发展到一定程度的自然延伸。目前,搜索引擎作为信息检索的一个重要工具,往往只是简单截取关键字所在的部分句子或段落作为摘要提供给用户,造成用户在浏览搜索结果时无法把握整个网页的内容提要,进而影响检索效率。而传统基于词频统计的摘要方法又主要针对纯文本文档,且单纯考虑文本的表层特征,缺乏对文档的语义分析,生成的文摘质量不高,因此也难以直接适用于Web文档摘要的自动提取。本课题针对传统摘要技术的不足,提出了一种基于主题划分的Web文档自动摘要方法。该方法以非受限领域的Web文档为处理对象,在机械文摘基础上融合了理解文摘中的相关语义分析技术,充分挖掘Web文档的结构特征来提取摘要。首先,利用HTML文档的标记信息划分网页主题,以划分的主题为单位抽取摘要,使得最终文摘覆盖的内容更加全面。其次,利用WordNet中的语义关系以概念统计代替传统的词频统计以消除同义词的影响,使得提取的主题特征更加准确。最后,根据句子之间的语义相似度,提出一种新的文摘句动态抽取算法,有效避免了文摘内容的重复。在具体研究过程中,本论文所作的主要工作如下:网页主题划分:利用Web文档的标记信息构建相应的DOM树,设计了两级不同粒度的过滤器对网页进行预处理。在此基础上,通过树中节点的自然分割功能和节点间的语义相似性比较,将文档划分成不同的主题块。主题概念提取:利用WordNet词典中的同义和上下位关系,将同义词归结为它们的祖先概念,对同义词的统计就归结为对它们上位词的统计,从而有效降低向量空间的维数。并针对网页特点,通过对自身频率、覆盖度等指标的衡量来选取主题概念。摘要动态生成:基于上述工作提取的主题概念,为文档中的每个句子构建向量空间模型,利用网页标签对摘要的重要辅助作用,对传统的句子权重计算方法进行了改进,并在相似度理论的基础上设计了一种动态的文摘句抽取算法。实验结果及评价:简要介绍了基于主题划分的Web文档自动摘要的原型系统,包括系统结构、用户界面、关键步骤及其中一些重要的功能模块。然后利用收集的各类网页,将本文提出的摘要方法和传统的基于统计的摘要方法进行对照实更多还原

【Abstract】 With the rapid development of Internet, the network has already become the warehouse of the data and potential source of knowledge. How could users obtain and utilize these resources fast and effectively become a problem needing to solve urgently, the information retrieval and automatic summary is the key technology. Automatic summary technology presents document content compactly, it is the result of the information retrieval technique which develops to certain extent. But now, the search engine only return sentences or paragraphs including the key words as the abstract, which makes users hard to grasp the whole content of the Web document. And the traditional technology based on the Statistic frequency of the word only focuses on the external characteristic of text and is lack of semantic analysis. So, it can’t be totally suitable for the summary of Web document.In the view of the deficiency of traditional summary technology, this text has put forward a new summary technology of webpage document based on the topic segmentation in the unlimited field. The method has added the semantic analysis technology of understanding digest on the found of mechanical digest and utilized fully the Web structure which has the auxiliary function to the summary. First, we segment the Web topics using the structure of HTML document and extract the digest in the unit of the divided topic blocks, which can increase the coverage of the digest to the document. Secondly, while extracting topic character, we utilize WordNet to extract topic concept instead of counting traditional frequency of the word, which can dispel the influence of the synonym and improve the accuracy of drawing topic words. Finally, according to similar degree among of sentence weight, we put forward a algorithms to extract digest sentences dynamically, which can reduce redundancy greatly. During the course of research in this paper, the main work is as follows:Webpage topic Segmentation: On the basis of constructed DOM tree of HTML document, carry on the pretreatment to the collected Webpage after designing two grades of filters, and then divide the document into different topics through the natural segmentation function and the semantic similarity comparison of nodes.Theme concept Drawing: Utilizing the relation of synonym and up/down in WordNet, sum up the synonym as their ancestor’s concept instead of the statistics of each synonymous word, which can reduce the dimension of the vector space更多还原