【作者】 冉熙璐； 段磊； 吕广奕； 陈珂； 李钟麒； 黄东兰； 唐常杰；

【Author】 Ran Xilu,Duan Lei,Lv Guangyi,Chen Ke,Li Zhongqi,Huang Donglan,and Tang Changjie (School of Computer Science,Sichuan University,Chengdu 610065)

【机构】 四川大学计算机学院；

【摘要】 随着Web 2.0相关技术的发展,传统爬虫无法适于动态网页中用户评论的获取.主要工作包括:1)分析了动态网页用户评论的挑战;2)设计了一种新的用户评论获取方法ReviewCrawler,ReviewCrawler根据网页的DOM树,利用对比学习思想挖掘包含用户评论的节点,并在获取评论中学习新特征词;3)利用真实商品评论验证了ReviewCrawler准确性、有效性.实验表明ReviewCrawler的查全率及查准率大于98%.同时ReviewCrawler具有良好的伸缩性,能够满足获取海量用户评论的要求.更多还原

【Abstract】 With the development of Web 2.0 related techniques,traditional crawler cannot effectively crawl the customer reviews from dynamic Web pages.The main contributions of this paper include: 1) analyzing the challenges of Customer Reviews Crawling from Dynamic Web Pages;2) developing a novel approach named ReviewCrawler for Customer Reviews Crawling.By the DOM tree of a Web page,ReviewCrawler applies the thought of contrast learning to discovering the nodes containing customer reviews,and learning the new feature words from the crawled reviews;3) applying ReviewCrawler to crawl cross-domain customer reviews in the real-world online shopping Web sites. The experimental study shows that the recall of ReviewCrawler and the precision is larger than 98%. Moreover,ReviewCrawler has good flexibility,which can satisfy the requirement of crawling massive customer reviews.更多还原