【作者】 朱玉丽；

【导师】 徐全生；

【作者基本信息】 沈阳工业大学 ， 计算机应用技术， 2007， 硕士

【摘要】 网格技术是近年来得到广泛应用的技术，它解决了资源的真正共享，使各结点大量资源得到统一支配和使用，信息网格是在计算网格的基础之上，利用数据挖掘、信息融合和搜索引擎等技术而构建的，便于网格资源的搜集与共享，目标是创建一种构建在OS和Web之上的基于Internet的新一代信息平台，在这个平台上，信息的处理是分布的、协作和智能化的。主题爬虫是根据目标主题，以智能化的方式从Web上收集主题相关的页面集，然后采用机器学习或信息检索的方法对收集到的信息进行智能处理和分析，最后以方便、有效的检索方式满足用户的信息检索需求。本文阐述了网格和主题爬虫的基本概念和国内外的发展现状，说明了研究基于网格技术的主题爬虫算法的意义。详细论述了网格技术以及网格的体系结构，同时阐述了Globus的积木式体系结构、OGSA体系结构和基于Web Services的OGSA架构，进而介绍了开放式网格服务的基础结构。重点分析了主题爬虫的概念、体系结构以及存在的主要问题，提出一种主题爬虫算法——ZTSpider算法，通过对超文本分类器和超链接评价器的研究，改进了现有主题爬虫算法在线学习能力差的问题，同时应用反馈机制，优化了网页父子结点之间的信息传递，增加爬行收获率。设计并实现了主题爬虫ZTSpider模拟系统，改进了现有的主题爬虫的体系结构，该系统应用JAVA语言开发，具有跨平台、高可扩展等特性，实现了Web信息的智能化收集，通过该爬虫算法的收获率，验证了ZTSpider算法的有效性。设计并实现了基于服务网格的分布式主题爬虫系统，应用SOAP、WSDL、UDDI完成了描述服务，定义消息服务和端口类型，在Globus Toolkit中完成了主题爬虫ZTSpider的分布、协作和智能化工作。更多还原

【Abstract】 In recent years, grid technology has been widely used to fulfill authentic sharing of resources and to maneuver uniformly and use large amount of resources in each node. Information grid is built on computing grid, using technologies such as data mining, information infusion, search engine and so on. It facilitates searching and sharing of grid resources and is intended to build a new-generation information platform based on OS and web. On this platform, Information processing is distributed, corporate, and intelligent. Information can be accessed through one entrance.Focus spider collects pages set according to theme of the subject in an intelligent way. The pages set collected are then processed and analyzed using methods like machine learning and information searching. Finally, requests of information searching from users are fulfilled in an efficient way of searching.Basic concepts and current development of grid and focus spider in our country and aboard are discussed. Also, the meaning of research in focus spider algorithm based on grid technology and main work of this paper are fully illustrated.Grid technology and architecture are discussed in detail. Globus architecture, OGSA architecture and OGSA architecture based on Web Services are also illustrated and further basic architecture of open grid services are also mentioned.The concept, architecture and current problems concerning focus spider are analyzed in great depth. A focus spider algorithm-ZTSpider algorithm is brought about. The algorithm settles the problems of deficiency of online-learning of current focus spider algorithms through research of hypertext categorization and hyperlink analysis, and at the same uses increment feed-back mechanisms, and optimizes information delivery between parent node and children node, and further enhanced crawling efficiency.A ZTSpider simulating system is developed and implemented. The system improved architecture of current focus spider. The system is developed using Java programming language and possess features like cross-platform and high extensibility. It also fulfills intelligent web information collecting and validates the availability of ZTSpider algorithm through gain rate of the algorithm.A distributed focus spider system based on information grid are also devised and implemented. The system harnesses SOAP, WSDL, and UDDI to accomplish description services, interface definition, and releasing ZTSpider to Globus Toolkit. It also accomplishes design of OGSA architecture and distribution, association and intelligent work of focus spider.更多还原