基于Hadoop的分布式藏文新闻网站垂直搜索引擎设计与实现

【作者】 韩志强；

【导师】 胥桂仙；

【作者基本信息】 中央民族大学 ， 计算机科学与技术， 2016， 硕士

【摘要】 在网络发展日新月异的今天,各种形式的信息呈现爆炸式增长的态势,这给传统搜索引擎快速去除无关内容、及时索引、准确检索带来了严峻的挑战。同时,面对传统搜索引擎返回的海量检索结果,我们常常需要花费很长时间来寻找真正需要的内容,可是结果仍然未必能如我们所愿,这一点无论是国外还是国内的搜索引擎都无法给出完美的解决方案。而垂直搜索引擎因其针对某一特定领域或人群的特性,有效地改善了这一现状,过滤了海量无关、冗余信息,提高了检索效率,最重要的是帮助用户快速、精准地锁定目标信息。随着藏族网民数量的逐年增加,人们获取藏文新闻的需求也越来越普遍。然而藏文的新闻网站却存在分布的比较分散,访问难度大；新闻更新不及时、新闻质量没有保证；网站之间新闻内容区分度不高,并且含有大量的无效网址、乱码新闻页面的特点,这给藏文新闻的传播带来了巨大阻碍。为了帮助人们高效、精确地浏览藏文新闻,本文设计和实现了一个基于Hadoop的分布式藏文新闻网站垂直搜索引擎,将垂直搜索引擎首次应用在藏文新闻领域。本文的设计方案在保证垂直搜索引擎精准性的前提下,有效地解决了上述问题,本文主要贡献如下：(1)设计并实现了基于Hadoop的分布式藏文新闻网站垂直搜索引擎系统。本系统严格按照传统软件项目管理流程设计。首先,从功能、界面、性能需求三个方向对软件进行细致的需求分析,并完成系统总体架构分析；其次,通过研究语料的选取来确定研究的领域,并具体分析系统研究数据的各项指标；最后,实现系统的功能模块,包括网络爬虫模块、信息预处理模块及索引和检索模块三大部分。(2)提出了藏文网络爬虫解决方案。本文分析了常见网络爬虫的工作原理,在此基础上开发了适合藏文新闻网站的爬虫解决方案,此解决方案实现了跨平台性,增加了系统可靠性。(3)提出了高频文章推荐算法。结合藏文新闻网页数量较少的特点,本文开发了高频文章推荐算法,将时下用户关注的热点新闻定向推荐给用户。(4)基于Hadoop的分布式藏文新闻网站垂直搜索引擎系统软件测试。系统开发结束,即进行了细致的软件测试。通过黑盒测试对系统界面、用户交互和性能三方面进行测试发现,本系统符合设计预期,可以满足用户使用需求。总之,本文的创新点在于,设计、实现了基于Hadoop的分布式藏文新闻网站垂直搜索引擎系统,通过了全面而系统的软件测试,并提出了藏文网络爬虫解决方案和高频文章推荐算法。更多还原

【Abstract】 With the rapidly changing development of network, various forms of information have been growing explosively, which poses a serious challenge to traditional search engine in quickly removing irrelevant content, indexing timely and accurately retrieving. At the same time, facing the massive conventional search results returned by search engines, we often take a long time to find the content that is really needed, but the result still may not be as we wish. It makes both foreign and domestic search engines unable to give the perfect solution. The vertical search engine effectively improves the situation because of its properties for a particular area or people. Also, it improves the search efficiency by filtering massive extraneous, redundant information. Thus, it helps users quickly and precisely get the target information.With the increase of the number of Tibetans Internet users year by year, people has the more and more common demand to obtain Tibetans news. However, the Tibetan news website has a series of problems, such as the dispersion of distribution, difficult to access, not timely to update news, no guarantee of the quality of the news, low discrimination between news content of websites and lots of invalid URLs and garbled news pages, which brings a great obstacle to the spread of Tibetan news. To help people efficiently and accurately browse Tibetan News, this paper designs and implements a distributed hadoop-based Tibetan news sites vertical search engine to apply vertical search engine technology in the field of Tibetan News.In this paper, the design ensures the accuracy of the vertical search engine search results. In this premise, it effectively solves the above problems. The main contributions of this paper are as follows:(1) The design and implementation of a distributed hadoop-based Tibetan news sites vertical search engine.The system is designed in strict accordance with the traditional software project management processes. Firstly, from three directions, function, interface, performance requirements, the software needs analysis is conducted detailed, and the overall system architecture analysis is completed. Secondly, through the study of the selection of the corpus to determine the field of research, and specific analysis of the indicators of the system research data. Finally, the realization of the system function module is finished, including the web crawler module, information preprocessing module and index and retrieval module three parts.(2) The implementation of Tibetan web crawler solutions.This paper analyzes the common web crawler works. On this basis, a crawler solution suitable for Tibetan news sites is developed which can achieve cross platforms and increase the reliability of the system.(3) A high frequency article recommendation algorithm is proposed.Combined with a small number of Tibetan news page features, this paper develops a high-frequency article recommendation algorithm. Hot news people focus on will be recommended to them(4) The distributed Hadoop-based Tibetan news sites vertical search engine testing.A detailed software testing is conducted as soon as the end of the system development. With the black box testing for the system interface, user interaction and performance, the system is found to be qualified and it complies with the user requirements.In short, the innovation of this paper is that, it designs and implements a distributed Hadoop-based Tibetan news sites vertical search engine system, passes a overall software testing, proposes Tibetan web crawlers solution and a high frequency article recommendation algorithm更多还原