【作者】 陆鑫；

【导师】 戴元顺；

【作者基本信息】 电子科技大学 ， 计算机应用技术， 2018， 硕士

【摘要】 随着各种互联网应用的业务系统复杂度以及用户数据量的增长,数据的存储与读写压力随之不断提升,采用非关系型数据库,可以在一定程度上解决这个问题。常用的非关系型数据库Cassandra具有很好的横向可扩展性以及较好的写操作性能,但是,在实际使用过程中,发现Cassandra的读操作速度不是非常理想,并且并发性能也有着一定的瓶颈,而缓存和索引可以高效地提升数据系统的读操作速度。所以,本文将系统并且全面地介绍基于Cassandra数据库的缓存和索引数据系统的设计与实现。本文在保证了 Cassandra分布式特性的前提下,对于Cassandra数据库的并发读、写性能进行了优化。首先,读操作缓存以及索引系统可以很好地降低对于磁盘的直接访问,但是由于近两年内存价格居高不下,为了增大缓存的容量,提高缓存命中率,本文将近些年投入使用的固态硬盘作为内存的后备缓存使用,利用分代式算法思想对于内存和固态硬盘中的缓存数据进行分别处理,不仅可以对内存缓存进行扩容,还能够增强缓存的容错能力,实现缓存的持久化。其次,结合Bloom Filter的一种优化方法来作为索引系统,和Cassandra数据系统本身采用的LSM数据存储思想结合,通过存储寻址速度的加快,来提升整个系统的读性能。并且,为LSM数据存储方式增加数据缓冲层,来保证数据批量刷入磁盘,可以在一定程度上增强系统的写操作性能。最后,结合近些年非常热门的Node.js服务端框架以及对象关系映射思想,实现了 Node.js环境下的Cassandra数据库驱动,并且结合异步I/O以及多进程并发思想,实现了一个高效的Node.js数据服务端。除此之外,为了增强系统的可维护性,本文使用了数据可视化的方法将上述各个模块的运行状态实时反馈,保证了每个模块都能够稳定运行。通过上述几点优化,来实现一个可以很好地和现有的云计算平台结合的数据系统。本文的最后对于实现的Cassandra数据系统进行部署以及测试,分别测试系统最初预设的功能目标以及实现性能。测试结果表明,改进后的Cassandra数据系统的读性能有着较大幅度的提升,而并发性能和写性能也有小幅增长。更多还原

【Abstract】 With the increasing of the complexity of the business systems of various Internet applications and volume of user data,the storage and read/write pressures of data are continuously increasing.Using non-relational databases can solve this problem to some extent.Commonly used non-relational database Cassandra has good horizontal scalability and better write performance,but in actual use,found that Cassandra’s read operation speed is not satisfactory enough,and concurrent performance has a certain degree of bottleneck.The cache and index can effectively improve the read speed of the data system.Therefore,this thesis will systematically and comprehensively introduce the design and implementation of the cache and index data system based on Cassandra database.This thesis optimizes concurrent read and write performance of Cassandra database on the premise of guaranteeing the distributed character of Cassandra.First of all,the read cache can reduce the direct access to the disk,but due to the high memory prices in recent two years,we will put cache into solid state disk as the backup cache of memory in order to increase the cache capacity and improve the cache hit rate.The use of generational algorithm to separately process the cached data in the memory and the SSD can not only expand the memory cache,but also enhance the caching system’s fault-tolerant capability and achieve persistent cache.Secondly,combined with an optimization method of Bloom Filter as an indexing system,combined with the LSM data storage idea adopted by the Cassandra data system itself,the read performance of the entire system is improved through the acceleration of storage addressing speed.In addition,adding a data buffer layer to the LSM data storage mode to ensure that the data is flushed into the disk in batches can enhance the system’s write performance to some extent.Finally,combined with the popular Node.js server-side framework and object-relational mapping in recent years,we will implement a data server system which combined asynchronous I/O and multi-process concurrency.In addition,in order to enhance the maintainability of the system,this thesis uses a data visualization method to provide real-time feedback on the operating status of the above modules,ensuring that each module can operate stably.Through the above several optimizations,a data system that can integrate well with existing cloud computing platforms can be realized.At the end of this thesis,we deploy and test the Cassandra data system that we implemented,and test the initial function target and performance of the system respectively.The test results show that the read performance of the improved Cassandra data system has been greatly improved,while the concurrent performance and the write performance have also slightly increased.更多还原