【作者】 高峰；

【导师】 刘震；

【作者基本信息】 电子科技大学 ， 工程硕士（专业学位）， 2018， 硕士

【摘要】 由于互联网的高速发展和普及,互联网已经成为了一个十分重要的信息源。而许多互联网用户也越来越迫切的希望能够在浩瀚的互联网中高效准确的找到目标主题页面并对主题页面实现定制化实体信息提取。传统的搜索引擎领域中主题爬虫和垂直爬虫是获取特定主题和特定网站数据的较为普遍流行的方法,但是主题爬虫更注重于主题页面的搜索,往往忽视对页面信息精确抽取的深入研究,而垂直爬虫虽然可以对一个网站实现精准信息抽取,但是其一大弊端就是可移植性较差,无法对不同网站实现通用的爬取,自动化程度较低。经典的WEB信息提取方法,虽然在各自适应的领域取得一定的成果,但是也都有其适应范围的局限性和提取算法效率低的问题;同时,这些方法基本都只侧重于对目标WEB页面实体信息抽取的研究,而忽略了对目标页面的搜索策略的研究;因此,现有的经典WEB实体信息的抽取方法都有其适用范围和研究范围的局限性。本文针对垂直爬虫无法直接移植到其他网站并且程序设计需要大量人工干预的弊端,以及经典WEB实体信息抽取方法的局限性,提出了一种高效的,且具有高可移植性的WEB实体信息提取算法,而提取算法的研究又包括了主题页面搜索定位和页面信息提取两个部分:(1)在主题页面搜索定位部分,提出了有监督的广度优先网页带权搜索策略,自动识别主题目标和目录页面URL以及利用URL聚类生成URL正则表达式过滤器,利用正则表达式过滤器广度优先大范围搜索相关页面,同时辅以基于隧道技术的网页权值计算达到有监督的最佳优先的效果,通过实验证明本文设计的搜索策略能够保证爬虫充分、快速、准确的对于主题相关页面的定位及下载,具有较高的搜索效率和准确率。(2)在页面信息提取部分,结合多种经典WEB信息提取技术的优势提出了基于配置信息的数据解析路径模板自动生成方法,基于配置信息可以让爬虫实现对于定制化WEB实体信息的准确完整提取,而自动生成数据解析路径模板则可以充分保证信息抽取的高效性和准确性,并能够提高自动化程度。利用本文提出的WEB实体信息提取算法的原理,本文设计实现了一种通用型垂直爬虫系统,该系统的实现是WEB数据采集器方面的一个具体应用,并且该系统可以在便捷的配置信息后,实现对不同网站的高效、快速、准确的定制化数据爬取,可移植性高且通用性强。同时,也证明了本文提出的WEB实体信息提取算法的合理性和有效性,具有较高的应用价值,也丰富了WEB信息抽取领域的理论和应用研究。更多还原

【Abstract】 Due to the rapid development and popularity of the Internet,the Internet has become a very important source of information.Many Internet users are also increasingly eager to be able to effectively and accurately find the target topic page in the vast Internet and implement customized entity information extraction on the topic page.In the traditional search engine domain,topic crawlers and vertical crawlers are popular methods for obtaining specific topics and specific website data.However,the topic crawlers pay more attention to the search of topic pages,and often neglect the in-depth study of accurate extraction of page information.Although reptiles can extract accurate information from a website,their main disadvantage is their poor portability,their inability to achieve common crawling of different websites,and their low degree of automation.Made some achievements as cassical WEB methods of information extraction in respective fields they adapt to,certain problems also exist,limited scope of adaptation and low efficiency of the extraction algorithm included.In the meanwhile,these methods only focus on studying the entity information extraction of the target WEB page,while the topic crawler concentrates more on the search and location of the subject target page.Therefore,the existing classical WEB methods of entity information extraction is of limited scope of application and research.Aiming at the disadvantage that vertical crawler can not be directly transplanted to other websites and the program design needing a lot of manual intervention,and the limitation of classical WEB entity information extraction method,a highly efficient and highly portable WEB entity information extraction algorithm is proposed in this paper,and the research of the extraction algorithm which includes the search and location of topic page as well as the extraction of page information.(1)In the searching and locating part,a supervised weighted searching strategy for breadth-first web pages is proposed,which can automatically identify topic targets and directory pages URL,and generate URL regular expression filters by URL clustering.The regular expression filter is used to search the relevant pages in a wide range of areas first,and the effect of the best and priority can be accomplished by means of the value calculation based on tunnel technology.Experiments demonstrate that the search strategy designed in this paper can ensure the crawler to locate and download the topic related pages sufficiently,quickly and accurately with high efficiency and accuracy.(2)In the section of page information extraction,the crawler can realize the accurate and complete extraction of the customized WEB entity information based on configured information taking advantage of Wrapper.The automatic generation of data parsing path template can fully ensure the efficiency and accuracy of information extraction.Using the principle of the WEB entity information extraction algorithm proposed in this paper,this paper designs and implements a general-purpose vertical crawler system.The implementation of the system is a specific application of the WEB data collector,The system can realize efficient,fast and accurate customized data crawling on different websites after convenient configuration information,high portability and strong universality.At the same time,it also proves the rationality and effectiveness of WEB entity information extraction algorithm proposed in this paper,has a high value of application,and also enriches the theoretical and applied research of WEB information extraction.更多还原