【作者】 刘阳；

【导师】 张化祥；

【作者基本信息】 山东师范大学 ， 计算机软件与理论， 2014， 硕士

【摘要】 研究表明，80%的搜索引擎使用者查看返回结果不超过三页，因此在搜索引擎返回结果中，排名越高带来的利润越大，很多网页通过欺骗搜索引擎的手段获得较高的排名，这类网页被称为垃圾网页。垃圾网页在利益的驱使下采用作弊手段欺骗搜索引擎获得更高的排名，干扰了用户对信息的获取。垃圾网页损害搜索引擎的声誉，削弱了其用户对搜索引擎的信任，检测垃圾网页已是搜索引擎面临的重大的挑战之一。为了有效的检测出垃圾网页，分析了数据集中垃圾网页与正常的网页内容特征与链接特征分布，通过结合网页的内容特征与链接特征结合机器学习方法及其他排序算法对垃圾网页进行检测。具体工作如下：1. TrustRank算法是著名的基于链接信息的网页排序算法。传统的TrustRank算法基于链接特征的垃圾网页检测方法，这种方法不是对所有作弊网页都有效。譬如一组网页提供一些有用资源，吸引其他网站的链接，但这组网页包含了许多指向目标作弊页面的链接，这些链接可能是隐蔽的，目标作弊网页的TrustRank值就有可能很高。而且有的垃圾网页的拓扑结构和正常网页的拓扑结构相似，这时基于内容特征的垃圾网页检测方法就比较有效了。因此，我们在基于链接分析的TrustRank算法基础上对网页内容特征进行提取，并分析网页内容特征的分布，根据正常网页与垃圾网页内容特征分布的不同结合网页链接特征对Web spam进行检测。2.基于内容特征的垃圾网页检测方法只考虑了网页的文本内容特征，没有考虑网页的链接结构，很难适应不断发展的网页作弊技术，而基于链接结构的垃圾网页检测方法则忽略了网页的内容信息，如果只考虑网页的拓扑结构，很难检测出那些拓扑结构与正常网页十分相似的垃圾网页。我们在分析数据集网页特征分布的基础上，用各种分布函数拟合网页的内容特征分布和链接特征分布，由于正常网页的特征分布比较有规律，而垃圾网页的特征分布混乱，因此用分布函数拟合之后求差值，正常网页差值较小而垃圾网页差值较大。根据网页特征值与拟合函数的差值利用决策树检测垃圾网页。3.目前大多数研究把垃圾网页检测看作是分类问题，并且使用机器学习方法譬如SVM、决策树等对垃圾网页进行检测。我们把垃圾网页检测问题看作排序问题，结合网页的内容特征和链接特征建立排序模型对网页进行排序。排序模型的基本要求是能够保证正常网页排名靠前，垃圾网页排名靠后，从而使得用户使用搜索引擎时不会被垃圾网页干扰。我们结合网页的链接信息和网页的内容特征信息建立数学模型。首先，我们通过分析网页的内容特征分布，用分布函数拟合计算得到内容特征向量，用内容特征向量逼近网页的链接信息值求得权值向量，计算内容特征向量与权值向量的乘积作为网页值对网页排序。垃圾网页不仅影响了用户通过搜索引擎寻找有用的信息，而且严重浪费了搜索引擎的资源，搜索引擎在根据用户的请求索引网页时，需要处理大量的垃圾的网页，所以检测垃圾网页的相关研究有切实的意义。更多还原

【Abstract】 Research shows that eighty percent of search engine users browsed the results returned bysearch engine no more than three pages. Therefore, in the results returned by search engine, thehigher the rank the greater the profits. Meanwhile, many web pages get higher rankings bydeceiving search engines which are called web spam. Driven by the profit, web spam get higherranking by deceive the search engines which interfere with the user access to information. Webspams damaged the reputation of search engines and weaken users trust to search engines. So webspam detection is one of the major challenges faced by the search engines. In order to detect spampages effectively, we analyzed content and link features distribution of normal pages and spampages. We combined web content features and link features as well as machine learning methodsand other sorting algorithms to detect spam pages. Details are as follows:1. Trustrank algorithm is the ranking algorithms which based on the link structure. Thetraditional TrustRank algorithm detected spam pages only with information of link feature, butthis method is not effective for all spam pages. Such as a set of web pages provide some usefulresources to attract other links, but this group of web pages contains many links to the target pagewhich cheating the search engines, and those links may be hidden. In this way, The TrustRankvalue of target page could be very high. It is difficult to detect web spam which topology structureis very similar to the normal page, so the spam detection method based on content features iseffective. Therefore, we extracted and analyzed the distribution of the web content features,according to the different of the distribution of the normal web content features and spam pagescontent features, we combined the difference with the pages links feature to detect spam pages.2. Spam page detection methods based on the Content features only consider the content ofthe web page features difficult to adapt to the evolving web cheating technology, spam pagedetection method based on the link structure ignored the content information of pages. It isdifficult to detect web spam which topology structure is very similar to the normal page if we onlyconsider the topology structure of the page. We analyzed the distribution of web content featuresand linked features and indicate that normal web features distribute regular but spam web featuresdistribute scattered. So we employ function to fit the distribution of normal web features thencalculate the difference between web proportion and the distribution function. The difference ofnormal web pages is small but the difference of spam pages is large. Finally, we use decision treesto detect spam pages with difference as threshold.3. Many people refused Web spam detection as a classification problem and use of thevarious machine learning classifiers to detect spam pages, such as SVM and decision tree such ascan be trained to detection spam pages. In our opinion, web spam detection also can be regardedas a ranking problem. the basic requirement of ranking model is that normal pages rank higher andspam pages rank later, allowing users will not be disturbed by spam pages when using searchengines. First of all, we obtained content feature vector by analyzed the distribution of the contentfeatures, and then we trained a mathematical model with content feature vector represent valueswhich determined by the link information. Finally, we used the model ranked pages.Web spam pages not only affects the user find useful information through the search engine, but also waste a serious of resources for the search engine, search engines index pages accordingto the user’s request need to deal with a lot of spam pages, so studies detect spam pages havepractical significance.更多还原