【作者】 李星；

【导师】 牛奉高；

【作者基本信息】 山西大学 ， 统计学， 2018， 硕士

【摘要】 文本挖掘是实现知识发现、自然语言处理最为关键的一步.文本挖掘与一般数据挖掘的本质的区别在于,文本的数据形式是半结构化的,因之对文本数据进行挖掘的首要任务是文本数据的结构化表示.然而,现有的文本表示方法存在语义信息提取不充分、表示模型的维数很高、模型构建复杂度大等问题.从而对文本表示模型进行深入研究并对模型中现有的问题着手改进是十分必要的.围绕文本挖掘的核心文本表示问题,本文在梳理、归纳现有文本向量表示模型的基础上,以共现分析理论、长尾理论、布尔运算法则为支撑,对文本向量表示模型中的几个核心问题展开了较为深入的研究并作出相应改进.本文首先介绍了文本向量表示模型的研究背景,目的,意义及国内外研究现状,并指出文章的主要研究内容、研究方法及创新之处.其次是对文本向量表示模型的相关理论研究.对本文的所用到的共现分析理论、长尾理论、布尔代数运算法则作了介绍;详细梳理、归纳了文本表示模型的发展路径.对发展过程中的重要关键模型构建的数学思想作了对比分析并指出各模型的适用范围和不足之处,厘清文本向量表示模型的构建原理和发展脉络;第三,是对文本向量表示模型的关键技术的改进研究.针对文本向量表示的新模型-共现潜在语义向量空间模型中的权重设置、特征维数约简、语义信息的提取等方面进行深入研究并作出改进.首先针对权重设置问题提出基于多种估计方法的共现潜在语义向量空间模型(CLSVSM);其次为了减小模型构建的复杂度结合长尾理论进行特征维数约简提出截尾共现潜在语义向量空间模型(TCLSVSM);最重要的是对潜在语义信息的提取结合布尔运算法则提出广义潜在语义向量空间模型(GCLSVSM),消除了潜在语义向量空间模型的局限性.第四,对潜在语义向量空间模型的改进模型进行聚类检验与评价.一系列聚类实验表明了改进模型在文本聚类中的有效性.最后进行总结和展望.总结本文的主要内容,并提出今后的研究与改进方向.更多还原

【Abstract】 Text mining is the most important step in Knowledge Discovery of Database and Natural Language Processing.The difference between text mining and general data mining is that the text data is semi-structured,and the primary task of mining text data is the structured representation of text data.However,the existing text representation methods have several problems such as insufficient semantic information extraction,high dimension of representation model,and large complexity of model construction.Therefore,it is necessary to study the text representation model and improve the existing problems in the model.Focusing on the core content of text mining to text representation problem,this article is based on combing and summarizing the existing text vector representation models,based on co-occurrence analysis theory,long tail theory,and boolean algorithm rule supporting.The core issues have carried out more in-depth research and made corresponding improvements to the text vector representation model.This paper first introduces the research background,purpose,significance and research status of text vector representation model,and points out the main research contents,research methods and technical routes and innovations of the article.This was followed by the introduction of the relevant theory and text vector model and its history.The co-occurrence analysis theory,long-tail theory,and Boolean algebra algorithm used in this paper are introduced;the development path of text vector representation model is combed and summarized in detail.The development path of the text vector representation model was reviewed,and the construction principle of the text vector representation model and development context was clarified.Thirdly,it is an improvement research on the key technology of the text vector representation model.It’s studied in-depth and improved to the core problem of the new model of text vector representation-co-occurrence latent semantic vector space model in terms of weight setting,feature dimension reduction,semantic information extraction and so on;Firstly,co-occurrence latent semantic vector space model(CLSVSM)based on multiple estimation methods is proposed for the weight setting problem;Secondly,in order to reduce the complexity of model combined with the long tail theory,we proposed to truncated co-occurrence latent semantic vector spacemodel(TCLSVSM);The most important is to propose the generalized latent semantic vector space model(GCLSVSM)for the extraction of latent semantic information combined with the Boolean algebraic product idea,and to broaden the scope of topic aggregation of the latent semantic vector space model.Fourthly,clustering and evaluation of improved models of latent semantic vector space models.A series of experiments show the advantages of improved models in text clustering.Finally,the summary and prospect are carried out.The main content of this article is summarized,and the future research and improvement direction is put forward.更多还原