【作者】 付维权；

【导师】 孙志仁； 曹奇英；

【作者基本信息】 东华大学 ， 计算机应用技术， 2005， 硕士

【摘要】 随着企业信息化的进程逐步加快,企业经营数据的管理呈现越来越多的困难。根据“进去的是垃圾,出来的也是垃圾”这条原理,为了支持正确决策,就要求管理的数据必须可靠,没有错误,准确反映企业的实际情况。因此,企业数据质量的管理受到越来越多的关注,本文主要从数据清洗的角度进行探讨。 宝钢集团实施的计划值系统通过对基础数据(包括历史值和当前值)的整理分析,结合实际预测未来值(即计划值的预计值),在跟踪分析实际值与预计值的差异原因的基础上,找出改进管理的方向,实施管理循环。因此,计划值管理为各项管理提供了完善管理的手段,是提高各项基础管理的有效方法,也是实施例外管理的很好平台。但其所依赖的基础数据存在精度过粗的问题,影响系统的正常运行,必须应用数据清洗技术对其基础数据进行清洗处理,以解决大量脏数据存在的问题。 本文主要从以下四个方面对数据清洗技术及其在宝钢计划值系统中的应用做了详细的分析和研究。 第一是对数据清洗技术进行了概述,阐述了数据清洗的更多还原

【Abstract】 Along with the speeding up of the information-based progress in enterprises, the management of the enterprises’ working data is getting more and more difficult. According to the rule of "garbage in, garbage out", in order to provide the support for the decision-maker, the data of management must be accurate and represent the real status of the enterprise actually, so more and more people begin to pay attention to the management of enterprise data. This paper mainly dealt with the management of enterprise data from a data cleaning perspective.The Planning Value System implemented in Baosteel Group Corporation depends on the analysis of the foundation data including the history data and the real-time data, the cleaning up of garbage data and the actual estimated value (the Planning Value’s estimated value). Based on analyzing the reason of the difference between the actual value and the estimated value, it can find out the direction to improve the management and help to implement the management cycle. Therefore, the Planning Value management provides an effective means for various managements, a valid method to raise the efficiencies of various foundation managements and a good platform to implement exception management.Because the foundation data have so many problems in accuracy which will affect the normal function of the Planning Value System, it is necessary to use data cleaning techniques to resolve the problem of data before it put into use.This paper had a detailed analysis and research on the technique of data cleaning and its application in the Planning Value System of Baosteel Group Corporation and was divided into the following four parts.The first part just gave us a summarization of thetechnique of data cleaning. It explained the origin and definition of data cleaning. Then it gave us a complete introduction to the data quality problems which we could meet with and the common steps of data cleaning. In the end, an algorithm of clustering, based on the N- Gram, was explained in detail.The second part discussed the requirement analysis and part of related development design of the Planning Value System in Baosteel Group Corporation. It discussed the difficulties、 the key techniques and the characteristics of inner dirty data in the system. It also explained the superiority of related function support in the SAS software whose data warehousing system had a specific mechanism to support the checking of the outer data and the integrating of various source data. The SAS software is a good helper to the work of data cleaning which required by the Planning Value System in Baosteel Group Corporation.The third part expounded a series of work which using the SAS software to implement the operation of data cleaning. Then, based on the Euclidean distance, it realized an algorithm of clustering. The result of the work proved that this algorithm could meet the basic requirements of the Planning Value System implemented in Baosteel Group Corporation, and could well resolve the problem of data duplicate, which badly existed in the foundation data.In the end, it put forward several directions on data cleaning worthy of our research, including the easy operability, the validity, the compatibility and the generalization of data cleaning.更多还原