【作者】 王鹏；

【导师】 周彩兰；

【作者基本信息】 武汉理工大学 ， 计算机应用技术， 2010， 硕士

【摘要】 随着网页信息量越来越大,采用通用搜索引擎搜索某一个方面的信息时覆盖率和准确率低,回馈的内容不够详细,噪音过多,维护庞大的网页索引库特别困难,在信息的采集和存储面临着严峻的挑战。由于通用搜索引擎存在以上的缺点垂直搜索引擎正好弥补了这些不足,它最大的特点是精,准,深。论文的主要工作如下：1.介绍了垂直搜索引擎,包括网络蜘蛛,索引器,检索器,用户接口的体系构架和主题页面分布特征、倒排序索引的建立和中文分词等基本技术。2.分析并研究了页面的解析和抓取、与主题相关性判定、网页收集和净化等基本工作原理,优化并且实现了一种页面消重算法。3.利用lucene开发包实现了一个小型垂直搜索引擎。用网络蜘蛛实现对各种类型的文档的解析,包括文本、html、Word、pdf等格式,通过解析文档来提取与主题有关的信息,同时实现了中文分词、索引器和检索器等模块。4.论文对内容重复的网页消重算法进行了改进和实现,它是对传统基于特征串的页面消重算法的一种改进。由于转载导致网址不同而网页内容相同,这样会有大量的重复页面出现,论文页面消重算法改进算法利用了主码和辅码来实现,特征码能够较好反映网页内容和方便计算。主码表示网页文本的段落结构信息,辅码用来标识网页文本的内容信息,这样就能够利用网页文本的结构和内容信息消除重复网页,在很大程度上提高了消重的效率。更多还原

【Abstract】 With the growing amount of information pages when searching for a particular information on general search engines accuracy and coverage is low, paid back the content is not detailed enough and too much noise, maintain a huge index library of web pages is especially difficult, in information collection and storage is facing severe challenges. Because there is more than general search engines disadvantages of vertical search engines offset the sortage and its greatest feature is the precise, accurate, deep.Main work of page as follows:1.Describing completily vertical search engine, including system architecture which including a web spider, indexer, crawler, and user interface and themes distributing features, inverted sort index creation and basic techniques such as chinese segmentating word.2.Analysising and studying the parsing and crawling web, and the subject of determination, collection and purification of the basic web and also so operating principle, optimize and implement an algorithm for elimination of duplicated pages.3.Implementing a small vertical search engine, Mainly implementing lucene development kit, Web spider to achieve resolution of the various types of documents, including text, html, Word, pdf and other formats, by parsing the document to extract the topic-related information, and the page achieve modules including the Chinese word segmentation, the indexer and the seacher.4.This page eliminate improving and implementeing web including duplicating content.it is a an improvement.for traditional feature-based words algorithm.As the result pages, reproduced led to such as the emergence of the same content at different web url, so there will be a lot of duplicate content, The improved algorithm of this paper use the main code and secondary code to achieve, the feature expression signature can reflect the page content and the convenience of calculation. Master code paragraph of text signature web page structure information, secondary code identify the contents of the web page, so that the text can use the structure and content of information to eliminate duplicate pages. The algorithm has greatly improved the efficiency of eliminating duplicate pages.更多还原