【摘要】 网络上存在大量考研资讯,如何对这些考研信息进行有效采集、分析、筛选,对后续数据挖掘、数据分析有着举足轻重的作用。在分析Scrapy基础上,针对Scrapy框架无法下载AJAX动态页面的缺陷,提出一种采用Scrapy-Splash模块使Scrapy也能爬取AJAX数据的信息采集方案。通过对Scrapy框架Request的构造方法和Response跟进方法覆写,使Scrapy Engine能够向Splash发起渲染请求,接收渲染返回的SplashResponse对象并对其进行调度。利用Scrapy-Redis框架设计了一套分布式网络爬虫系统,用于高效、稳定地获取中国研究生招生考试网中的数据。测试结果显示,获取的数据具有实时性和可靠性。更多还原

【Abstract】 A large amount of information about graduate entrance examination is published on the Internet. How to effectively collect,analyze,and screen these information plays a significant role in the data mining and data analysis. Based on the analysis of Scrapy,in view of the defects that the Scrapy framework cannot download dynamic pages which using AJAX(Asynchronous JavaScript And XML),a strategy that imports Scrapy-Splash module to allow Scrapy to crawl AJAX data is proposed. By overriding the Scrapy framework Request construction method and Response follow-up method,the Scrapy Engine can initiate a rendering request to Splash,and receive and dispatch the returned SplashResponse object. Based on the Scrapy-Redis framework,a distributed web crawler system is designed to efficiently and stably obtain data from the China Graduate Entrance Examination Website. The obtained data is real-time and reliable.更多还原