【摘要】 在大数据时代中,网络增量数据自动分片是统计理论与数据库结合的产物,针对当前方法网络增量数据自动分片准确率和效率低的问题,提出基于知识获取的网络增量数据自动分片方法。为了完成对网络增量数据自动分片,需要先对数据做降维处理,利用数据样本中心计算数据样本点类内的平均距离,得到数据样本点重构误差的重构系数,利用该系数完成对网络增量数据的降维处理。在此基础上,分析数据观察变量和潜在变量的概率分布情况,并计算其后验概率,网络是根据数据节点之间的边所组成的,可以通过数据节点间的边数等条件衡量数据分片参数的估计量,利用参数的估计量来描述网络增量数据自动分片的过程,得到邻节点数据分片在传播中的分量加权乘积,并对其迭代计算,最终实现了网络增量数据的自动分片。实验结果表明,提出方法在对网络增量数据自动分片时,具有较高的准确率,并且数据自动分片耗时短,效率高,均验证了提出方法的有效性。更多还原

【Abstract】 In the era of big data, the automatic sharding of network incremental data is the product of combining statistical theory with database. Due to low accuracy and efficiency of current method, this article presented an automatic sharding method of network incremental data based on knowledge acquisition. In order to complete automatic sharding of network incremental data, it was necessary to reduce the dimension of data at first. Then, the data sample center was used to calculate the within-class average distance of data sample points and obtain the reconstruction coefficient of reconstruction error of data sample points. In addition, this coefficient was used to complete the dimension reduction of network incremental data. On this basis, the probability distribution of observed variables and potential variables was analyzed, and then the posterior probability was calculated. The network was composed of edges between data nodes. The estimator of data sharding parameters could be measured by the number of edges between data nodes. The automatic sharding process of network incremental data was described by the estimator of parameters, so that the component weighted product of neighbor node data sharding in propagation was obtained. After iteration calculation, the automatic shrding of network incremental data was realized. Simulation results show that the proposed method has high accuracy in automatic sharding of network incremental data. Meanwhile, the data automatic sharding has low time consumption and high efficiency. The effectiveness of the proposed method can be proved.更多还原