【作者】 周志杰；

【导师】 苏波；

【作者基本信息】 上海交通大学 ， 信息与通信工程， 2015， 硕士

【摘要】 在现今这个信息高速膨胀的时代,网络中充斥着太多有价值但却没有得到合理利用的信息,对大量文本进行批量的分类处理逐渐成为一种非常迫切的需要。其中对于用户通过互联网发表的自己的观点这种主观性文本进行分析和挖掘,识别出其中所蕴含的主客观性和情感倾向性,对于电子商务、舆情监控等各种领域有着重要的应用价值。本文研究的是微博文本的主客观分类。本文设计了基于投票机制的主客观分类系统,集成了包括基于线索密度、基于N-POS和基于条件随机场的三种方法。因此本课题的研究内容重点在于各个基分类器的优化和集成学习融合。本文首先介绍了文本分类系统的结构和流程,然后依次介绍了文本预处理相关技术、文本表示模型、文本特征表示方法中的特征空间选择方法等关键技术,为后续继续进行文本主客观分类研究提供了基础的技术支持。在基于线索密度的主客观分类模块中,本文提出了主观线索的概念,并对主观线索进行统计验证,采用了新型的线索权重,引入了线索密度的概念来衡量文本的主观性程度,提出了两种特征权重计算方法,并比较它们对于主客观分类性能的影响。在基于N-POS的主客观分类模块中,本文提出标准N-POS主观模式库的概念和基于N-POS主观模式的主客观分析算法,探究了决策用经验主观程度阀值和主观模式选取比例对分类效果的影响。在基于条件随机场的主客观分类模块中,本文提出了适合的从标注序列中抽离出主客观分类结果的算法,通过增量学习的思想完备了特征模板项,探究了不同特征模板以及分类结果抽离算法中主观权重对分类效果的影响。在系统集成实验中,实验结果显示集成学习的引入使得整体主客观分类性能提高2%,表明了集成学习在文本主客观分类领域的可行性。更多还原

【Abstract】 In this day of information expansion, the network is filled with too much valuable information which isn’t used reasonably. Classification of massive texts in large batches has gradually become a very urgent need. More people are willing to express their views and comments via Internet. Analysis and mining to these subjective texts can identify the inherent subjectivity and emotion tendency which has important application value in many fields such as E-commerce, public opinion monitoring and so on. In this paper, we mainly focus on subjectivity classification of microblog.This paper designs a subjective and objective classification system based on voting mechanism. The system adopts the integrated study, integrating three methods including the method based on density of clues, the method based on N-POS and the method based on conditional random field. So the research content is focused on the optimization of the base classifiers and the integration of learning.This paper introduces the structure and classification process of the traditional text classification system, and then respectively introduces the text pretreatment technology, text representation, feature selection method, feature dimension reduction method, text features’ weight calculation method, text categorization algorithm and assessment methods on text classification results and so on. It provides the fundamental technical support for the follow-up text classification of subjectivity and objectivity.In the subjectivity classification module based on the density of clues, this paper puts forward the concept of subjective clues and carries out statistical validation of subjective clues. At the same time, this paper adopted the new clues’ weight calculation method. For the first time, it introduces the concept of the density of clues to measure the subjectivity of text. It compares the two different weight calculation methods for the classification effect.In the subjectivity classification module based on N-POS, this paper puts forward the concept of the standard N-POS subjective pattern library and the subjective and objective analysis algorithm based on subjective N-POS model. It explores the subjective threshold and selection of proportion for the classification effect.In the subjectivity classification module based on conditional random field, this paper puts forward the suitable algorithm to get the results of subjective and objective classification from sequence. By the thought of the incremental learning, it completes feature template. It explores different templates and subjective weights for the classification effect.In integration experiments, the results show that the introduction of integrated learning makes the overall subjectivity classification performance increased by 2%. It indicates the feasibility of integrated study in the field of text classification of subjectivity.更多还原