【作者】 吕鹏；

【导师】 蒋志方；

【作者基本信息】 山东大学 ， 软件工程（专业学位）， 2017， 硕士

【摘要】 按照国家税务总局“互联网+税务”行动的战略部署,随着数据集中、大数据应用等项目的逐渐开展,各省对于数据质量提升的需求,越来越迫切。为了满足总局对数据质量提升的需求,为了指导数据质量工作的开展,建立一个完整、准确的数据质量管理体系,架设一个专业、高效的数据质量监控平台,成为一种必然的趋势。本文所描述的是采用B/S网络架构、Sword框架、J2EE技术与java语言实现数据质量管理系统。该系统同时运用了成熟的阿里云技术和产品,包括对象存储(0SS)、大数据计算服务(ODPS)、云数据库(RDS)和云服务器(ECS),等。系统设计时采用了包含展示层、服务层、中间层、数据层的四层架构,同时采用云数据库(RDS)进行数据存储。本系统所使用的关键技术除了阿里云相关的一系列组件外,还运用了 JSqlParser插件解析SQL语句,运用Quartz组件进行任务调度。经过设计与实现环节,数据质量管理系统所实现的功能有质量需求管理功能、质量规则管理功能、质量任务管理功能、质量问题管理功能和质量问题监控功能。其中,质量需求管理最终实现质量需求定义和质量需求维护。质量规则管理最终实现质量规则定义和质量规则维护两个功能菜单。质量任务管理最终实现质量任务定义、质量任务维护、质量任务审批、质量任务运行监控和质量任务结果查看五个功能菜单。质量问题管理最终实现质量问题定义、质量问题维护和质量问题推送三个功能菜单。质量问题监控最终实现问题监控总览、质量整改监控和质量问题统计查询三个功能菜单。数据质量管理系统的应用,在一定程度上规范数据质量管理的流程,使得数据质量的分工和责任更加的清晰明确。系统的整个流程实现从上到下,再从下到上的一个环路,以规则为基础循环前进,实现数据质量的不断提高。在循环中建立一个统一的规范的质量规则库,可供全国各个省份参考使用。系统利用先进的大数据技术,实现对海量数据的分析和监控;充分发挥云计算的优势,突破了以往ORACLE数据库针对海量数据进行分析时效率低下甚至出现无法得到最终结果的窘境。更多还原

【Abstract】 With the development of the data collection of the State Administration of Taxation and the application of big data,the demand for data quality in the provinces has become more and more urgent.In order to meet the needs of administration to improve the quality of data,quality of data in order to guide the work,establish a complete and accurate data quality management system,data quality monitoring platform to set up a professional and efficient,has become an inevitable trend.This paper describes the use of B/S network architecture,Sword framework,J2EE technology and Java language to realize data quality management system,at the same time,the use of cloud technology and products Ali mature,including the object storage(OSS),big data,MaxCompute services(ODPS).Relational Database Service(RDS)and cloud server ECS,etc..Using four layers of Architecture:display layer,service layer,middle layer and data layer,the cloud database(RDS)is used for data storage.The system adopts the key technology,in addition to using Ali cloud components,but also uses the JSqlParser plug-in to resolve SQL statements,using Quartz components for task scheduling.Through the design and implementation aspects,the realization of data quality management system has the function of quality demand management function,quality rules management function,task management,quality management and quality quality monitoring function.Among them,quality requirements management ultimately achieves quality requirements definition and quality requirements maintenance.Quality rule management,final implementation,quality rule definition and quality rule maintenance,two function menus.Quality task management ultimately implements quality function definition,quality task maintenance,quality task approval,quality task monitoring and quality task outcome,and five function menus.Quality problem management,ultimately achieving quality issues,defining quality issues,maintaining and quality issues,and pushing three feature menus.Statistical quality monitoring and ultimately monitoring overview,quality monitoring and quality rectification problem of query three function menu.The application of data quality management system,to a certain extent,standardize the process of data quality management,and make the division of quality and responsibility of data more clear and definite.The whole process of the system is realized from top to bottom,and then from bottom to top,a loop,continuous cycle forward,based on the rules,to achieve continuous improvement of data quality.In the loop,a unified and standard quality rule base has been established,which can be used for reference by all provinces of the country.Using a large data system with advanced technology,to realize the analysis and monitoring of the massive data;give full play to the advantages of cloud computing,breaking the previous ORACLE database for the massive data analysis of low efficiency and even unable to get the final result.更多还原