【作者】 张芳；

【导师】 高永兵；

【作者基本信息】 内蒙古科技大学 ， 计算机应用技术， 2012， 硕士

【摘要】 随着校园网建设的迅速发展，校园网信息资源迅速增加，这使得全校师生迅速定位有价值的信息难度较大，浪费时间而且效率低下。基于校园网自身的特点，发展较成熟的通用搜索引擎不能完全适用于校园网，并且大量转载网页的存在造成检索结果重复页过多。通过分析校园网网页的特点和现有去重技术，以解决校园网搜索引擎检索结果重复网页过多问题，针对不同类型的重复网页，采用在索引和实时检索时分别去重的策略，构建了校园网搜索引擎，完成了如下几项工作：第一，对网页去重的准备工作进行了研究和分析。首先，分析网页噪音产生的原因、噪音的定义及类型，采用合并内容块技术对原始网页集进行噪音去除和正文抽取，以获得网页的正文内容。其次，研究中文分词技术，对比现有中文分词技术，最终采用庖丁解牛分词软件，对Nutch进行二次开发——修改Nutch源码，实现中文分词。第二，对索引时网页去重算法进行研究和改进。分析现有算法，针对完全重复或部分重复的网页，采用最长段落签名的网页去重算法。首先对整篇文档签名后去重，其次对去重过滤后的文档分段，对分好的段落排序，再取前N个段落对其进行指纹签名，将其作为文档的特征，当这两个文档中相同段落数超过系统给定的一个阈值时，就判定这两个文档为相互重复的文档。提取前N段并进行指纹排序大大降低了计算的复杂度。实验证明，该方法有较高的去重准确率。第三，针对网页转载时对原网页进行微小修改而产生的重复网页，在实时检索时采用优化傅立叶变换去重算法。该算法把每篇文档的每个词映射成一个数值Fingerprint，那么每篇文档就可以表示成一个离散数值序列。对该数值序列进行傅立叶变换得到傅立叶系数，比较系数的前若干项即可大致比较出两个数列的相似性。实验证明，基于优化傅立叶变换的去重算法能够在网页发生修改的时候兼顾查全率和去重率。以Nutch作为系统的开发工具，通过对Nutch源码进行修改实现索引时的去重算法，并采用插件形式实现检索时的网页去重算法，在Nutch的基础上设计实现校园网搜索引擎，并详细说明了校园网搜索引擎系统开发过程和方法。最后对提出的去重策略进行实验性能测试，采用Nutch爬取校园网网页作为实验的数据集，结果表明将两种算法结合的去重策略提高了搜索结果的精确度和去重的准确率，并且搭建的校园网搜索引擎系统能够有效的、正常的运行。更多还原

【Abstract】 With the rapid development of the campus network construction, campus network informationresources increase quickly, which makes difficult for teachers and students to locate valuableinformation quickly, and it is also a waste of time and efficiency. Based on the characteristics of thecampus network itself, mature general search engines cannot be applicable to the campus networkcompletely, and retrieval results have too many duplicate web pages due to the copy messages fromother websites. By analysis of the characteristics of the pages of the campus network and theexisting duplicate detection technology, the problem that the retrieval results have to many duplicatepages is solved. For different types of duplicate pages, the index and real-time search duplicatedetection strategies are used to build a campus network search engine. The work is finished asfollows:Firstly, web page duplicate detection algorithms are researched and analyzed. Foremost thecauses of web page noise, the definition and types of noise are analyzed, and correlation contentmerger technology is used to get rid of noise and extract content in order to obtain the content of theweb pages. Then, after the study and analysis of Chinese word segmentation technologies, Paodingword segmentation analysis is applied to Nutch secondary development--Nutch source is modifiedand Chinese word segmentation is realized.Secondly, duplication detection algorithm is researched and improved in the index phase. Afterthe analysis of existing algorithms, the duplicate detection algorithm based on the longest paragraphand fingerprint is proposed for the complete or part duplication of web pages. Firstly, duplicatepages are removed from the entire documents. Secondly, the document from which duplicate pageshave been removed and filleted is segmented. Paragraphs sorted are sorted, and then the first Nparagraphs are taken for fingerprint signatures which are used for the characteristics of the document.When the number of paragraphs which are the same in the two documents exceeds a given threshold,which determines whether these two documents are duplicate documents. The first N segments havebeen extracted and the fingerprint is sorted, which greatly reduces the computational complexity.Experiments show that this method has higher duplicate detection accuracy.Thirdly, duplicate pages modified minimum from the original pages use the optimized Fouriertransform duplicate detection algorithm. The algorithm maps each word of each document into anumerical fingerprint, and each document can be expressed as a discrete sequence of numbers. TheFourier coefficients are obtained by use of the Fourier transform to process the numerical sequence,and the similarity of the two series can be obtained by comparing the first several coefficients. The experiments show that recall rate and duplicate detection rate both can be taken into account basedon the optimized Fourier transform algorithm for the web pages which have been modified.Nutch is used as a system development tool and through the modification of the Nutch sourcecode，duplicate detection algorithm is achieved when the web pages are indexed. Web pageduplicate detection algorithm is achieved by the form of plug-in when the pages are being retrieved.Campus network search engine is designed and implemented on the basis of the Nutch. The campusnetwork search engine system development process and methods are described in detail. Finally,using Nutch to crawl the pages of the campus network as the experimental data sets, duplicatedetection strategy proposed is tested experimentally. The results show that the accuracy rate ofsearch results and the duplicate detection are improved by the combination of the two algorithms,and the campus network search engine system built can be runned effectively and steady.更多还原