【作者】 刘凡；

【导师】 郑纬民；

【作者基本信息】 清华大学 ， 计算机科学与技术， 2003， 硕士

【摘要】 网页信息的更新是网络一个非常重要的性质。随着Web信息内容的迅速增长,如何有效的跟踪特定网站和页面的更新情况日渐成为人们关心的课题。Web页面跟踪系统是一种专门关注于页面动态更新的信息服务系统。本文对页面跟踪系统的有关技术进行了深入的研究。主要研究以下各方面的内容:1,利用泊松过程作为网页更新的数学模型,在研究模型的基础上通过预测页面的更新频率,自动调节对各个页面的跟踪频率与页面的更新匹配,从而达到提高系统的整体效率的目标。我们首先讨论页面更新的特性及其数学模型,推导出捕获率和完备率的平衡点,以此来评价跟踪频率动态调整的效果。接下来,根据泊松过程原理设计并实现了ASA预测器算法,同时提出一个改进的QFA算法。最后,开发一套新的实验平台来提高实验的效率,并在此平台上对各种算法实现进行测试。2,研究用于页面跟踪的采集器时,主要对两个方面进行改进:一是深入研究HTTP协议相关部分,寻找提高状态检查效率的途径;二是应用线程池技术实现多线程并行采集,提高采集的性能,并给出性能测试结果。3,页面更新检测着重提高检测效率。对更新检测的功能进行简化,将重点放在关注不同版本间链接的变化上。为了方便高效的存取页面文件,设计一套分布式的文件存储方案。变更如何通知用户也是网页信息跟踪的一个重要的方面,我们给出一个变更通告服务的原型。在上述研究的基础上,我们设计与实现了页面跟踪系统“ChangeSpider”。ChangeSpider能够自动的跟踪用户输入的网页,并且自适应调节跟踪频率以配合网页的变化情况,及时把网页变化的信息通知给用户。基于ChangeSpider实现的多个应用实例表明ChangeSpider有很高的实用价值,性能效率良好。更多还原

【Abstract】 The information changing of web pages is a very important property of the current Internet. With the fast changing content on the web, similar to some other web applications, how to trace the update of the specified pages effectively is very important. Tracing system is a kind of information service which focuses on the dynamic changes of web pages. In this thesis, we study the key technologies of web tracing system. The main content of this thesis includes: 1, The Poisson process model is a good mathematical model for the web page changing. Based on this model, the changing frequency of a specific web page is predicted. The tracing frequency will be adjusted according to the changing frequency of the target web page. This can greatly improve the tracking efficiency. Firstly, the Poisson process model for the web page changing is well studied. We make the balance between the catching ratio and the recall ratio. The ASA algorithm is then designed to get the best performance together with an improved QFA method for comparison. An experiment platform is designed to test those different algorithms.2, The web crawler which is used to collect the source web pages is important in the tracking system. We improve the performance of a crawler by two means and make it suitable for a web tracking system. One is to find out the best way to check the web page status the other is to use the multi-thread technology which can improve the efficiency of web page downloading. 3, The efficiency if web changing detection is improved. The function of difference detection is simplified and we focus on the links changing among different versions of a specific web page. A distributed storage system is designed to improve the file accessing. The updating information delivery is also important and a prototype of the information delivery is also proposed in this thesis. We designed our <WP=5>own web tracking system: ChangeSpider. ChangeSpider can monitor the changes of the user specified web pages automatically as well as its ability of self-adjust frequency according to the changing frequency of the target pages. Experiments show that ChangeSpider turns to be useful and have a good performance.更多还原