【作者】 陈飞；

【导师】 周东清；

【作者基本信息】 大连理工大学 ， 计算机应用技术， 2011， 硕士

【摘要】 随着互联网的迅猛发展,特别是Web 2.0时代的到来,网站在格式和内容上都发生了比较大的变化,越来越多的网站开始采用动态网页。这类网页摆脱了传统的只能由服务器发布网站内容的束缚,允许普通的用户创建,修改和发布网站内容。为了创建动态网页,Ajax技术应运而生。Ajax技术大大的改善了用户的使用体验和网页的交互响应速度。除此之外,Web 2.0网站的网页内容信息来源更加的多样化,信息在时效性的要求上也比传统的方式更加严格。为了解决这些问题,传统的爬虫技术必须在抓取动态内容和时效性两个方面进行改进。在抓取动态内容方面,由于在动态网页中,页面的内容需要执行动态脚本才能得到,页面的跳转不再仅仅通过标签<a>来实现,URL也不再是页面的唯一标识。本文提出了一种支持Ajax的爬虫模型,该模型使用嵌入式浏览器实现对动态脚本的解析以及提出了一种高效的获取动态脚本网站有效页面的方法,首先通过训练得到触发哪些页面元素的哪些事件能跳转至有效页面,总结出这些页面元素的XPath特征以及相应的事件类型,在应用阶段只触发这些页面元素上特定的事件。最后通过实验证明了本文方法的效率和性能。在时效性方面,本文着重研究了实时搜索中数据抓取的基本问题,并提出了一种通过分析数据对象的历史变化过程对对象的变化频率进行预测的方法。针对实时垂直搜索领域中,对象的热门程度的变化与传统搜索相比更加敏感的问题,提出了一种基于对象属性关联的热门预测模型,对对象的热门程度进行预测。在综合考虑对象权重和变化频率的基础上,提出了一种基于泊松过程的抓取策略,实现了针对对象变化的抓取策略,提高了抓取资源的利用率和和数据的新鲜度。更多还原

【Abstract】 With the rapid development of Internet, especially in times of Web 2.0, both format and content have great changes in websites. There are more and more sites begin to use dynamic pages, and these new pages allow ordinary users to create, modify and publish the content, which get rid of the shackles that only published by server. To create dynamic pages, the technology of Ajax emerged. This technology has great improved user’s experience and reduced response time. In addition, web pages in Web 2.0 website access information are more diverse, and timeliness of information is also more stringent than the traditional way. To solve these problems, the traditional crawler technology must be improved in two aspects—crawl dynamic web page and timeliness.In aspect of crawling dynamic web page, the crawler need to perform dynamic script to get the page content, the switches between pages are no longer all based on the "<a>" tags and URL is no longer the unique identification of a web page. This paper presents a model which uses the embedded browser to achieve the resolution of the dynamic script, and propose an efficient approach to crawl valid page for websites with dynamic scripts. First, by training we can get the elements and triggered the events which can jump to the effective page. Then we summed up the XPath features of these elements and the events we have to trigger, and in the application stage, we only trigger these specific events. Finally, we proved the efficiency and performance of this model through experiments.On the other hand, we focus on the basic problems of data crawl, and predicting the frequency of data changes by analyzing the historical process. In real-time vertical search areas, object changes more sensitive than the traditional search engines, so we propose a model to predict the tendency of object distribution. Considering the weight and the changing frequency of objects, this paper proposes a crawl strategy based on Poisson process which improves the resources utilization and data freshness.更多还原