【作者】 汪锐；

【导师】 李国强； 龙德清；

【作者基本信息】 上海交通大学 ， 软件工程， 2012， 硕士

【摘要】 随着信息化水平不断提升，在公安部门内部，各种数据信息、文件、视频资料都需要存储。各个部门都购买存储设备用于存储这些数据资源，昂贵的存储设备的购买加大了运营成本，而且又无法满足存储资源的共享需求。因为存储设备的地理位置分散，要实现数据资源共享，就必然要利用网络实现数据传输，网络信息存储系统已经逐渐成为公安部门高效运作的一个核心技术平台。但是由于网络的各种不安全、不稳定因素和公安系统部门的工作特性，网络信息存储系统就需要具有很高的工作性能，其中包括安全的数据访问、稳定的数据传输和存储扩展能力。因此如何对已有的存储设备进行整合利用，提供一个节约资源又安全可靠的信息存储平台成为公安系统面临的一大难题。云存储由分布式文件系统、网络传输、集群应用、虚拟化存储等技术构成，把分布在网络上的各种存储设备整合成为一个资源池，作为一个服务体系对外提供统一的接口。采用云存储技术以后，对于用户来说，就不单是使用某个具体的存储设备，而是分布在网络上的多个存储设备的集合体。这样不仅可以有效整合存储设备资源，还可以提高存储的便捷性和资源共享。Hadoop就是一个能够对大量数据进行分布式处理的云存储软件框架，提供了一种安全可靠、高效、可伸缩的方式存储数据。本文分析了公安系统部门的资源存储状况，同时分析了云存储的现状和Hadoop的架构，然后深入的研究了系统内部的工作原理和功能特征，在此基础上，提出了一个基于Hadoop的信息存储系统方案。信息存储系统的底层是Hadoop分布式文件系统，这个文件系统将一个大文件分割成为了若干个数据块，分别存储在保存数据的数据节点上。由于每个数据块都有若干个相同的副本数据块，所以当数据块发生错误的时候，系统会自动的用另外一个完整的数据块副本取代当前出错的数据块，同时将出错的数据块上报给名字节点进行处理，这种机制保证了系统中数据的完整性。与传统的信息存储系统不同，基于Hadoop的信息存储平台的用户是分散在各个不同地方、不同部门的，用户在上传数据资源的时候，都储存在了一个共同的海量的存储池里面。当用户需要查找需要的资源时，可以自由的从存储平台中下载下来，就省去了传统的信息沟通不畅和数据时效性问题。信息系统部署在了基于Linux操作系统的机器集群上，默认的操作管理接口是命令行方式的，如果不是专业的人员，就很难掌握操作。该系统主要利用Hadoop实现存储层的技术支撑，用户层采用Web开发相关技术实现用户与存储层进行交互。最终为工作人员提供友好的接口来访问系统，工作人员在不需要了解底层技术细节的情况下，就可以使用一个基于云存储的信息存储系统。更多还原

【Abstract】 Nowadays, huge data, documents and video materials need to be stored in publicsecurity department due to the rapid informationization construction progress. Topurchase those expensive storage equipments and resources individually by eachsub-departments increases the construct and operation cost, while there is noconvenient way to share resources. Network is mandatory to implement datatransmission for the data resources share purpose, because those storage equipmentsare located in different places. But considering the security risk, network instabilityand the specificity of Public Security Department, it’s very critical to make thenetwork storage system to be with high performance, including safe access, stabledata connection and expandability of storage capacity. Hence one big challenge forPublic Security System is how to integrate existed storage equipments, and provideone cost-saving and reliable platform for Information Storage.Cloud Storage Technique comprises distributed file system, network transfers,application clusters, virtual storage and etc, which can combine variant distributedstorage equipments into one resource pool, and make it as one service system withone uniform interface. By using Cloud Storage, user can access the combination ofthose multiple storage equipments but not one specified storage equipment. It canintegrate the storage equipment and resource efficiently, improve the convenience ofusage, and share the resources. Hadoop is one of Cloud Storage Software Framework, which can distributely handle huge data and provide one reliable,efficient and flexible way to store data.This paper analyzes the resource storage status of Public Security Department,current situation of Cloud Storage and architecture of Hadhoop. Then the systemmechanism and features are studied, based on which, one information storage systemsolution is provided. The lower layer of the Information Storage System is HadoopDistributed File System.This file system put a large file split into a number of datablocks,these data blocks are stored in the data nodes.Since each block of data has anumber of copies of data blocks, so when the error occurred, the system willautomatically use a copy of data block to replace the current block.At the sametime,data error will be reported to the name node.This mechanism ensures theintegrity of the data in the system.Different from traditional information storage system,The users of the systemare scattered in different places, different departments,The user to upload dataresources are stored in the common and massive storage pool.When users need to findthe resources needed,they can be freely downloaded from the storage platform,eliminating the need for poor communication and data timeliness.The informationsystem is deployed in the Linux operating system-based machine on the cluster, thedefault operating management interface is a command line, if not a professional staff,it is difficult to master the operation.This system implements the storage layer withHadoop, and implements user interaction layer with Web technology. Hence onefriendly interface is provided to users to access system, by which operators canoperate one Cloud Storage based information storage system without knowing thetechnical details of lower layers.更多还原