【作者】 李伟；

【导师】 张建锋；

【作者基本信息】 西北农林科技大学 ， 计算机应用技术， 2018， 硕士

【摘要】 传统网页敏感词过滤系统大多是通过对比基于数据库的文本文件实现的,这种方式实时性较差,敏感词语过滤效率低,需要耗费网管人员大量的时间和精力。本文以主动匹配并过滤网页中的敏感词语为目标,实现了文本在上传到服务器之前对网页中的敏感词进行主动匹配与过滤处理,采用决策树方法对含有敏感词的网页文本进行了分类。论文的主要内容有:(1)设计实现了一种基于字典树的网页敏感词匹配及过滤方法。采用Python中的Beautiful soup模块将网页解析为DOM(Document Object Model)文档对象模型,进而抽取出网页中的文本内容。研究了文本中敏感词的检索与匹配方法,设计出了一种基于字典树的网页敏感词匹配及过滤方法,提高了网页文本中的敏感词过滤的正确率和召回率。(2)设计了基于决策树的敏感文本分类器。通过文本的预处理构建敏感文本分类的训练集与测试集,利用中文分词系统构建敏感文本向量空间模型,计算文本集向量空间中词向量的TF-IDF值得到训练集与测试集的权重矩阵,采用了决策树C4.5算法构建敏感文本分类器。(3)使用Python语言实现了网页文本内容抽取,文本预处理和敏感文本的分类。针对网页文本中存在干扰项过多的问题,采用正则表达式去除文本中的特殊字符、将繁体中文转化为简体中文的方法,对网页文本进行了预处理。通过在不同类别的文本中添加敏感词的方法提供了敏感文本训练样本,提取其特征值构建决策树,设置剪枝条件,对样本进行了分类。更多还原

【Abstract】 The traditional system of webpage sensitive word filtering is mostly realized by comparing text files based on the database.This method has poor real-time performance and low efficiency in filtering sensitive words.It takes a lot of time and effort for network management personnel.This paper takes the initiative to match and filter the sensitive words in the webpage.It realizes the active matching and filtering of the sensitive words in the webpage before uploading it to the server,and uses the decision tree method to classify the webpage text containing sensitive words.The main contents of the paper as follows:(1)Design and implement a dictionary tree based webpage sensitive word matching and filtering method.Using the Beautiful soup module in Python to parse the web page into a DOM(Document Object Model)document object model,and then extract the text content of the web page.The method of retrieval and matching of sensitive words in texts is studied.A method of matching and filtering webpage sensitive words based on dictionary tree is designed,which improves the correctness and recall rate of sensitive word filtering in webpage texts.(2)A sensitive text classifier based on decision tree is designed.Through text preprocessing,the training set and test set of sensitive text classification were constructed.The vector space model of sensitive texts was constructed by using the Chinese word segmentation system.The TF-IDF value of the word vector in the text set vector space was calculated to obtain the weight matrix of the training set and the test set.A decision tree C4.5 algorithm was used to build a sensitive text classifier.(3)Using Python language to achieve the text content extraction,text preprocessing and sensitive text classification.For the problem that there are too many disturbing items in the webpage text,regular expressions are used to remove special characters in the text and convert traditional Chinese into Simplified Chinese.the webpage text is preprocessed.By adding sensitive words in different categories of texts,sensitive text training samples are provided,feature values are extracted,a decision tree is constructed,pruning conditions are set,and samples are classified.更多还原