【摘要】 针对幂律判断方式存在的问题,本文基于早期对幂律分布的研究,结合最大拟然拟合方法及词频统计算法,对中英文词频分布进行研究。给出了双对数坐标系下幂律分布的判断,并对词频统计与幂律分布进行拟合。研究结果表明,在双对数坐标系下,分布图像为近似直线是判断幂律分布的必要条件,而非充分条件;在自然语言的词频统计分布模型上,对观测数据进行幂律分布的拟合,得出的p-value分别为0.14和0.19,均大于0.1,且泊松分布、指数分布、广延指数分布的p-value值都为0,即可排除满足其他分布的假设,因此对观测数据拟合效果最好的是幂律分布。说明自然语言的词频分布满足幂律,且中英文同样适用。该研究对人们认识语言的发展过程具有重要意义。更多还原

【Abstract】 Aiming at the problems existing in the power law judgment method,this paper studies the frequency distribution of Chinese and English words based on the previous research on power law distribution,combined with the maximum likelihood fitting method and word frequency statistics algorithm.The judgment of power law distribution in double logarithmic coordinate system is given,and the word frequency statistics and power law distribution are fitted.The results show that in the double logarithmic coordinate system,the distribution image being an approximate straight line is a necessary condition for judging the power law distribution,but not a sufficient condition.On the word frequency statistical distribution model of natural language,the power law distribution of the observation data is proposed.The p-values obtained are 0.14 and 0.19,respectively,both greater than 0.1,and the p-values of the Poisson distribution,the exponential distribution,and the extensive exponential distribution are all 0,that is,the assumptions satisfying other distributions can be directly excluded.The best fit for the observation data is the power law distribution.It shows that the word frequency distribution of natural language satisfies the power law,and both Chinese and English are applicable.This research is of great significance for people to understand the development process of language.更多还原