【摘要】 随着大数据时代的到来,网络爬虫已经成为很普遍的技术,无论是做项目、科研、创业或者写论文,获得大量数据并且对数据进行分析都是必不可少的。但是目前存在深层网(Deep Web)的数据量是表层网(Surface Web)数据量的数百倍,乃至上千倍。传统的爬虫对表层网数据进行获取已经无法满足我们的需求,同时因为深层网数据通常没有各种复杂的标签结构等,使得其本身更加清晰,干净,故而我们深入研究深层网络爬虫是非常有必要的。本文将会通过Python的Scrapy爬虫框架,对深层网络爬虫进行研究,通过分析深层网络特点制定合适的Scrapy爬虫策略,最后通过实际操作,对指定的爬虫策略进行验证。更多还原

【Abstract】 With the advent of large data age, web crawler has become a very popular technology, whether it is doing projects, scientific research, entrepreneurship or writing papers, accessto a large number of data and analysis of the data is essential. However, the amount of data in the deep Network(deep web) is hundreds of times more than the amount of surface network(surface web) data, even thousands of times. The traditional crawler to obtain the surface network data has not been able to meet our needs, at the same time, because the deep network data is usually not a variety of complex tag structure, making itself more clear and clean, so it is very necessary for us to deeply study deep web crawler. This paper will study the deep web crawler through Python’s Scrapy crawler framework, make suitable Scrapy crawler strategy by analyzing the characteristics of deep network, and finally validate the specified crawler strategy through actual operation.更多还原