【摘要】 为充分利用爬行主题与待访问页面之间的语义关系,提高主题爬虫的整体性能,在研究概念背景图(CCG)主题爬行算法的基础上,提出改进的CCG主题爬行算法。利用HITS算法选取高质量的主题背景知识,根据形式概念分析理论构建概念格模型,将概念格生成CCG用以存储用户查询意向;利用CCG综合父网页、锚文本、链接上下文以及URL自身预测链接主题相关度,过滤不相关页面。实验结果表明,改进的爬行算法有效提高了网页抓取的精度和召回率,具有较强的可行性。更多还原

【Abstract】 To make full use of the semantic relationship between the crawling topic and the unvisited page and to improve the performance of topic-crawler,the concept context graph(CCG)focused crawling algorithm was analyzed and an optimization algorithm based on it was put forward.The high-quality topic context knowledge was selected using HITS algorithm,the concept lattice was constructed with formal concept analysis theory and the CCG was produced to store the query intention of user.The predict similarity of unvisited page was computed using CCG and the similarity of parents page,anchor text,link context and URL information were all taken into consideration synthetically.Experimental results show that the precision and recall rate of the optimization crawling algorithm are better,and it possesses higher availability.更多还原