【作者】 李莉；

【导师】 吴共庆； 胡学钢；

【作者基本信息】 合肥工业大学 ， 计算机软件与理论， 2014， 硕士

【摘要】 Internet技术的发展、移动设备的普及以及微博、微信等应用的兴起促进了网民对Web新闻的接触度,碎片化时间阅读Web新闻已经成为了Web用户的主要活动之一。然而,Web新闻网页中除了主要内容外,还包含着大量与网页主题无关的“噪音”信息。这些噪音信息增大了Web新闻聚合、Web信息检索等应用的计算量和存储量,降低了Web服务的质量,同时也影响了手机等移动终端设备浏览网页的体验效果。因此,Web新闻内容抽取具有重要的研究意义和应用价值。为了解决面向开放式环境下的Web新闻内容的精确抽取问题,观察和分析了大量的Web新闻网页实例,发现Web新闻网页的内容分布与其对应的解析树中标签路径有着潜在的关联。基于以上发现,本文探索了基于标签路径特征的Web新闻内容抽取问题,并开展了以下研究：(1)根据Web新闻网页的内容分布与标签路径之间的潜在联系和Web新闻网页内容的文本信息,设计一种文本标签路径特征：文本标签路径比特征。通过深入分析文本标签路径比特征的不足和新闻内容的文本特征,对文本标签路径比特征进行了扩展;设计一种基于标签路径编辑距离的高斯平滑方法,有效地解决新闻内容中内链等短文本的精确抽取问题。在CleanEval等数据集上的抽取结果验证了CEPR方法是一种无监督的、通用的、高效的Web新闻内容抽取方法。(2)为提高标签路径特征的多样性,从不同的角度观察和分析新闻内容与标签路径特征之间的联系,设计了一个标签路径特征系,并分析和验证了每个标签路径特征的优缺点；为了将多个标签路径特征有机的融合为一个最终的决策特征,设计了一个基于DS证据理论的标签路径特征融合方法。大量真实Web网页数据集上的实验结果表明基于融合特征的Web新闻内容抽取方法CEPC的抽取性能优于基于任一标签路径特征的抽取性能,且其平均抽取性能优于CEPR方法。(3)为解决在抽取某一特定Web新闻网页时,标签路径特征系中存在的特征冗余问题,研究了一种基于Pearson相关系数的特征相关性度量方法,并根据该方法设计了一种组合特征选择策略。验结果表明：基于组合特征选择的Web新闻内容抽取方法CEPF的平均抽取性能可达到92.75%,优于CEPC、CEPR等其它抽取算法。(4)设计并实现了一个基于标签路径特征系的Web新闻内容抽取系统,系统集成了以上的抽取方法,介绍了系统的组成、实现以及系统的用户界面。更多还原

【Abstract】 The degree of exposure to Web news have been promoted by the development of the Internet technology, as well as the popularity of mobile devices and the rise of applications such as micrologging and micro-channel. Reading web news in fragmented time has become one of the main activities of Web users. However, in addition to the main content, web news pages also contain lots of "noise", which increases the computation and storage capacity of web applications like Web news aggregation and Web information retrieval, as well as influnces the experience result of pocket-size devices with small screens like mobile phones or PADs. Therefor, web news content extraction has both research and application values.In order to solve the problem of accurate extraction of web news in an open environment, our extensive case studies have indicated that there exists potential relevance between web content layouts and their tag paths. Motivated by this observation, this thesis explores a novel research topic on Web news extraction based on tag path features. The main contributions of this thesis are as follows:(1) Based on the potential relevance between web content layouts and their tag paths, a extraction feature based on tag path:Text to tag Path Ratio (TPR) is designed. With a deep analysis of the deficiency of the TPR feature, it is extended to be a Extend Text to tag Path Ratio (ETPR) feature. A Gaussian smoothing method weighted by tag path edit distances is designed to extract the short text accurately. Extraction results on CleanEval datasets show that the CEPR method is an unsupervised, generic, and efficient Web news content extraction methods.(2) In order to improve the diversity of tag path features, a tag path feature series was proposed by analyzing the relevance of the web content and their tag paths from different perspectives. After analyzing and verifying the advantages and disadvantages of each tag path feature, all tag path features are combined to a comprehensive feature by the DS theory and designed a Web news extraction method based on the comprehensive feature. Experimental results on real datasets demonstrate that the performance of CEPC outperform any one tag path feature and the average performance of CEPC is better than CEPR.(3) To solve the redundancy of tag path features of the CEPC method, a correlation measurement method between tag path features based on the Pearson correlation coefficient is studied, and on this basis, a group features selection strategy is designed. Experimental results show that:CEPF, a Web news extraction method based on the group features selection, can achieve better average performance by 92.75% than other existing methods like CEPC.(4) A Web news extraction system via tag path features is designed and implemented. All the extraction methods and tag path features which are researched in the dissertation are combined in the system. The implementation and the user interface of the system are introduced later.更多还原