【作者】 李杰；

【导师】 冯径； 黄瑞芳；

【作者基本信息】 国防科技大学 ， 工程硕士（专业学位）， 2018， 硕士

【摘要】 随着国际上气象水文交换和共享技术及互联网数据定向爬取等技术的发展,根据我国对气象水文观探测资料需求,充分利用互联网气象水文数据资源,弥补相关热点地区气象水文资料不足,显得尤为重要。本文在国内外先进技术的基础上,开展了互联网气象水文数据定向采集系统方案设计,并研制了原型软件。系统拓宽了气象水文资料收集渠道,解决了相关地区气象水文资料的有无问题,为发展互联网气象水文数据业务化采集提供了重要支撑。系统采用自动化采集传输流程,将互联网下载区的数据摆渡到内网供分发共享。系统由互联网数据定向采集综合设置、互联网数据定向采集任务调度、网页解析与数据获取、互联网数据预处理和网络摆渡等五个分系统组成。针对每个分系统和功能点,详细分析了相关技术的使用,总结归纳出基于坐标树的页面结构分析技术、基于WEB页面特征的页面内容提取技术和基于IP地址池和代理地址的数据采集技术等三项关键技术,为系统的成功研制打下了坚实的基础。原型软件设计成功后,进行了采集实例检验,检验结果表明,信息网络飞速发展的大背景下,利用本系统可以很好的根据用户需求实现气象水文数据的互联网定向采集。更多还原

【Abstract】 With the development of international meteorological and hydrological exchange and sharing technology and Internet data oriented crawling technology,and according to our country’s demand for meteorological and hydrological observation data,it is particularly important to make full use of the Internet meteorological and hydrological data resources to make up for the lack of meteorological and hydrological data in the relevant hot spots.Based on the advanced technology at home and abroad,this paper has designed the system scheme of the oriented acquisition of internet meteorological and hydrological data,and developed the prototype software.The system has widened the collecting channels of meteorological and hydrological data,solved the problem of lack of meteorological and hydrological data in the related areas,and provided an important support for the development of the operational collection of the meteorological hydrological data of the Internet.The system adopts the automatic acquisition and transmission process to ferry data from the Internet download area to the internal network for distribution and sharing.The System consists of five subsystems,which are Comprehensive setting of Internet data oriented acquisition subsystem,Task scheduling of Internet data oriented acquisition subsystem,Web page analysis and Data acquisition subsystem,Internet data preprocessing subsystem and Network ferry subsystem.For each sub-system and function,this paper analyzes the use of related technologies in detail and summarizes three key technologies,which contain the page structure analysis technology based on coordinate tree,the page content extraction technology based on WEB page feature and the data acquisition technology based on IP address pool and proxy pool.These key technologies lay a solid foundation for the successful development of the system.After the successful design of the prototype software,a instance test for data acquisition is carried out,and the test results show that,the system can be used to realize the Internet Oriented collection of meteorological hydrological data,under the background of the rapid development of information network.更多还原