【摘要】 为了避免造成数据库瞬时存储压力过大和系统存储负载不平衡等问题，针对海量并发数据流存储请求，本文提出一种面向并发数据流处理的存储机制。通过消息队列的异步传递缓解高并发数据阻塞问题，针对均衡存储并发数据流数据，将数据流进行数据预处理为任务单元，提出基于任务分配的一致性哈希算法实现任务单元的动态分配及处理节点间的负载均衡；同时使用HBase集群进行数据存储和持久化，为系统提供了可靠的存储基础。实验结果表明，该存储方法能够在持续的并发数据流下实现存储系统的负载均衡，保证了系统存储的稳定性，能够提高系统存储吞吐量。更多还原

【Abstract】 Aiming at the massive concurrent data stream storage requests, in order to avoid the problems of excessive instantaneous storage pressure of the database and system storage load imbalance, this paper proposed a storage method for concurrent data stream processing. Through the asynchronous transmission of message queue, the problem of high concurrent data blocking is alleviated. In the system, the data preprocessing is taken as the task unit in the efficient processing of concurrent data stream data unit, and the consistent hash algorithm based on task allocation is proposed to realize the dynamic allocation of task units and realize the load balance between processing nodes. At the same time, the HBase cluster is used for data storage and persistence, which provides a reliable storage foundation for the system. Through experimental analysis, this storage method can realize the load balancing of the storage system under the continuous concurrent data flow, ensure the stability of the system storage, and improve the throughput of the system storage compared with the traditional method.更多还原