基于Web对象的分布式抓取及存储的设计与研究

【作者】 罗理；

【导师】 王锋；

【作者基本信息】 昆明理工大学 ， 计算机应用技术， 2008， 硕士

【摘要】 本文主要研究的内容分为网页消重技术研究、Web对象分布式抓取技术、Web对象分布式存储技术。全文将重点对这几个部分进行了详细的论述。在网页消重技术方面,分别针对基于网页URL消重和网页内容消重进行了研究。在基于网页URL消重方面引入了Bloom Filter技术,通过采用Bloom Filter技术可以将庞大数量的URL全部映射到较小的内存中,从而实现快速URL匹配消重处理。针对网页消重的特点,本文还给出了基于Bloom Filter的改进方法,其改进思路是,将原算法中对URL映射空间位的更新操作改为记录该位在映射空间中的索引下标,并将该位的更新操作放到最后统一处理。采用这种内存映射位推迟更新的方法,使得可以将数量庞大的URL平均分配到多台消重服务器上创建相应的内存映射位域,这将大大减轻消重服务器的内存负担并极大地提高消重效率。在基于网页内容消重方面,本文根据新闻网页的固有特点,提出了采用新闻标题与新闻内容结构相结合来作为新闻网页的消重特征值。将新闻标题作为网页消重重要特征,对其进行MD5散列,得到的散列值进行匹配,在匹配成功的情况下,再对新闻内容结构进行相似度计算,利用其相似度来判断两篇新闻网页是否重复;如果在新闻标题匹配失败的情况下,那么就直接认为两篇新闻网页是不重复的。在Web对象分布式抓取方面,本文设计了一套基于任务分配队列的分布式网络爬虫系统,并设计了分布式网页抓取任务的分配算法。整个爬虫系统是由爬虫控制服务器和爬虫服务器组成的,爬虫控制服务器通过任务队列组管理和监控着各个爬虫服务器的工作状态,通过对各个爬虫服务器负载情况的分析,调整各服务器之间的任务分配,从而达到负载均衡。同时由控制服务器向爬虫提供获得抓取任务的服务。爬虫从控制服务器的任务队列组中获得URL抓取任务,并开始从互联网上抓取网页,从爬取下来的网页中抽取出新的URL,进行消重处理后入库。在Web对象分布式存储方面,本文设计了一套基于共享存储分配表的分布式网络存储系统。整个系统主要是由存储控制服务器和存储服务器组成,存储控制服务器向爬虫服务器提供共享存储分配表,爬虫只需要按照获取的存储分配表对Web对象进行存储即可,而在爬虫与存储服务器进行存储的过程中是不需要控制服务器参与的。存储控制服务器通过心跳机制管理和监控着各个存储服务器的工作状态。在存储服务器端以存储组为单位进行存储,而存储组是由主从服务器组成,其中从服务器同步备份主服务器上的数据,实现了存储服务的高可靠性。更多还原

【Abstract】 In this paper, the content of the mainly study include with research of detecting near-duplicate in web pages, technology of distributed crawling on web objects and technology of distributed storage on web objects. The entire thesis will concentrate on having carried out detailed discussion on this several parts.In the respect of technology of detecting near-duplicate web pages, to address detecting near-duplicate documents based on web URL and web content, we will research respectively. Led into the Bloom Filter technology in the respect of detecting near-duplicate based on web URL, a huge number of URL mapping into less memory by adopt the Bloom Filter technology, realized URL matching fast. To address features of detecting near-duplicate documents, this dissertation presents a improvement method based on Bloom Filter, its thinking is that recording index of place in mapping space replace operation of updating the place in the previous algorithm, and the updating operation will be executed put the final. Adopt the improved algorithm can make a huge number of URL distributed averagely many detecting servers and create the corresponding memory mapping space. And this will greatly reduce the heavy burden of memory in single detecting server and greatly improve the detecting efficient. In the respect of detecting near-duplicate based on web content, according to the inherent characteristics of news web pages, this dissertation presents a method that using headlines and structure of news content as a combination of the news web pages detecting fingerprints. News headline will be as an important characteristic of detecting, after MD5 hashed to it and got hash value, matching the hash value. In matching successful circumstances, to calculate similarity of the news structure, and use of its similarity to judge whether the two news pages are near-duplicate documents; In the failure of matching circumstances, then two news pages directly aren’t regarded as near-duplicate.In the field of distributed crawling on web objects, in this paper, we have designed a distributed network crawling system based on assignment-queue, and have designed allocation algorithm for distributed web crawling tasks. The entire crawling system consists of two main components: crawling control server, crawling server. Crawling control server manages and monitors the working state of all crawling servers by the task’s queue array. The crawlers can keep load-balance by analyzing the load and adjusting the assignment between every server. And the control server provides an acquisition of tasks for crawlers. Crawler get URL crawling tasks from the group of task queue in control server, and begin to capturing web pages from the Internet. The crawler parses new URL from the captured web pages, and inserts database after detecting near-duplicate.In the respect of distributed storage on web objects, in this paper, we have designed a distributed network storage system based on sharing storage table. The entire system consists of storage control server and storage server. The control server provides the sharing storage table for crawling servers. Crawler will store the web objects according to the storage table. In the process of crawler storing data, participation of the control server is not necessary. Storage control server manages and monitors the working state of each storage servers by the heartbeat mechanism. In the storage server side, storage group as storage unit, and storage group consists of master server and slave server. Through slave server backups the data on the master server, achieved the high reliability storage services.更多还原