【作者】 李欢；

【导师】 刘怀兰； 周源；

【作者基本信息】 华中科技大学 ， 工业工程， 2016， 硕士

【摘要】 随着互联网的发展,互联网上积累了越来越多的专业信息。然而,由于Web信息资源的爆炸式增长,传统的搜索引擎已满足不了人们对信息的个性化检索需求。针对不同领域,利用主题爬虫技术可以获取更细致、更专业的数据。本文在WebMagic框架的基础上设计并实现了面向机器人领域的主题爬虫,并且改进了主题识别模块中的朴素贝叶斯分类算法。主要完成了以下三个方面的工作。第一,通过对爬虫框架的大量调研,选择WebMagic作为本文的爬虫框架,对其进行二次开发,并增加了主题识别模块,实现了Web信息抽取、中文分词、去除停用词和特征选择等功能,最后构建了面向机器人领域的主题爬虫。第二,比较了不同的文本分类算法,选择朴素贝叶斯分类算法作为主题识别模块中分类器的算法,并分析了该算法的优势和劣势。第三,为提高朴素贝叶斯算法的二分类性能,结合了三种改进策略对算法进行改进：增加放大系数、对类别属性进行加权、增加主题范围约束参数。最后,通过对主题爬虫与通用爬虫进行对比实验,验证了主题爬虫在精确度上明显优于通用爬虫。通过对比传统朴素贝叶斯算法与本文改进的朴素贝叶斯算法,验证了改进后的算法在正确率、召回率和精确度上有了一定提升。更多还原

【Abstract】 With the development of the Internet, more and more professional information is accumulated to the Internet. However, due to the rapid growth of the Web information resources, traditional search engines have failed to meet people’s demand fo customized information retrieval. Focused crawler can crawl more detailed and professional data for different areas.In order gather to professional information, the focused crawler oriented to the robot industry is designed and implemented on the basis of WebMagic framework with an added theme identification module, where the naive bayesian classification algorithm is improved. The achievements of this thesis are as follows. Firstly, by comparing crawler frameworks, WebMagic is selected as the basic crawler framework of this thesis. Then, the secondary application development is discussed. In order to achieve the focused crawler, the theme identification module is added. This module includes Web information extraction, Chinese word segmentation, feature selection and removing stop words, etc. Secondly, by comparing different text classification algorithms, naive bayesian classification algorithm is chosen as the classification algorithm of the theme identification module. And the advantages and disadvantages of the algorithm are analyzed. Thirdly, in order to improve the performance of the naive bayes, three parameters are added. These parameters are the magnification factor, the attribute weights and the constraint factor of the theme.At last, comparing the experiment results obtained from focused crawler and general crawler, it is validated that focused crawler is better than general crawler on the accuracy. Comparing the experiment results classified by naive bayes and improved naive bayes, it is validated that the improved naive bayes contributes better accuracy, recall and precision.更多还原