【作者】 张静；

【导师】 吴信东；

【作者基本信息】 合肥工业大学 ， 计算机应用技术， 2015， 博士

【摘要】 近年来随着诸如Amazon Mechanical Turk等众包系统的出现,很多无法通过机器智能完成的任务可以通过这种微型外包的方式交由在线用户手工完成。机器学习和数据挖掘领域也从中受益,大量的样本标注任务由传统的耗时费力的专家标注方式转变为众包标注,加快了数据更新和模型迭代的速度。然而,由于众包标注者的标注质量无法保证,使得利用众包标注数据进行机器学习面临巨大挑战。因此,如何利用多噪声标签进行机器学习具有重要的研究与应用价值。本文从广泛存在的偏置标注问题着手,主要研究内容如下：(1)从众包系统数据质量和模型质量的概念以及偏置标注问题的定义出发,理论上分析了偏置标注对大多数众包系统中使用的多数投票策略所产生的影响。然后,对大量来自于众包系统的实际数据进行分析,揭示出偏置标注问题的普遍性,并分析产生偏置标注的原因。最后,在几个典型的偏置标注数据集上进行实验,分析了基于期望最大化的算法的真值推理性能,从而得出偏置标注问题使得基于期望最大化算法性能退化的结论。(2)针对二分类偏置标注问题,提出一种基于自动阈值估计的正标签频率阈值算法PLAT。该算法只依赖于每个样本的多噪声标签集中正负标签的分布,无需标注者质量、底层数据分布、以及偏置水平等先验知识。它能够自动估计正负类划分的阈值并进行真值推理。在人工合成数据集上的模拟和真实众包数据集上的实验显示PLAT算法不但能在偏置标注环境下进行真值推理并构建质量较高的学习模型,在非偏置标注环境下该算法也能够具备和当前算法相当的性能。(3)利用众包系统进行样本标注是一个动态过程,学习模型的构建符合主动学习的视角。针对偏置标注中主动学习问题,提出一种新型的主动学习框架。该框架包含两个核心过程：标签真值推理和样本选择。在标签真值推理过程中,使用PLAT算法推理训练数据集中每个带有多噪声标注集的样本的集成标签。在样本选择过程中,提出三种基于不确定度量的样本选择策略来提升模型学习的性能。这三种不确定度度量标准分别基于多噪声标签集和偏置程度(MLSI),基于学习模型和偏置程度(CMPI),以及前两者的混合策略(CFI)。在具有不同底层数据分布的多个数据集上的模拟和真实数据集上的实验显示CFI具有最好的性能。(4)针对多分类标注环境下偏置标注问题提出一种模糊化处理偏置标注的新算法GTIC。该算法通过聚类来提升从多个不确定标注者获得集成标签的性能。对于一个K分类标注问题,GTIC利用每个样本的多噪声标签集生成其概念层聚类特征。然后,使用经典的K-Means聚类算法将所有样本聚成K个不同的簇。每个簇对应一个类别,该簇中所有样本将被赋予相应的类别标签。实验结果显示在低质量数据集上,GTIC在准确度和M-AUC指标上明显优于现有算法。除此之外,GTIC比基于最大似然估计和期望最大化的真值推理算法运行时间缩短10倍以上,并且GTIC易于并行化,适合大数据环境应用。更多还原

【Abstract】 With the emergence of crowdsourcing systems, such as Amazon Mechanical Turk, many tasks that cannot be tackled by machine intelligence can now be posted to these systems and manually completed by online users via a micro outsourcing manner. Machine learning and data mining communities obtain the benifits from the crowdsouring, where a lot of traditional time comsuming and highly costly labeling tasks that used to be conducted by human experts are now transited to crowdsourced labeling, which accelerates the data updating and model refinement. However, without a guarantee of labeling qualities of the crowdsourced annotators, machine learning with the crowdsourced labeled data faces greate challenges. Thus, it is valuable to study machine learning algorithms which ultilize multiple noisy labels. This dissertation starts with the biased labeling problem, and its main research contributions are as follows.(1) We discuss data quality and model quality issues in crowdsouring systems as well as the definition of the biased labeling problem. We theoretically analyze the impact of baised labeling on the majority voting strategy that is commonly used in many crowdsourcing systems. Then, we analyze a great number of real-world data sets to reveal the existence of biased labeling and its causes. At last, we conduct a set of experiments on the data sets with biased labeling and investigate the performance of EM-based ground truth inference algorithms. Our findings confirm that biased labeling indeed deteriorates the performance of the EM-based ground truth inference algorithms.(2) For binary labeling, we propose a novel algorithm named positive label frequency threshold (PLAT) which can automatically estimate the decision threshold. The algorithm only depends on positive and negative labels in the multiple noisy label set of each example without the necessity of any prior knowledge such as the qualities of labelers, underlying class distributions and the level of bias. It can automatically estimate the threshold that divides the examples into positive and negative. Experimental results on both synthetic and real-world data sets show that PLAT not only infers the integrated labels and builds a high quality learning model under biased labeling circumstances, but also works well in comparison with other state-of-the-art algorithms under non-biased labeling circumstances.(3) Labeling using crowdsourcing is a dynamic process that follows the perspective of active learning. We propose a novel active learning framework with biased labeling. The framework includes two core procedures:ground truth inference and instance selection. During the ground truth inference, the PLAT algorithm is used to infer integrated labels of examples with multiple noisy label sets. During the instance selection, we propose three strategies based on uncertainty measures to improve the learning performance. These three uncertainty measures are based on multiple noisy labels and the level of bias (MLSI), on the currently learned model and the level of bias (CMPI), and on hybrid strategies (CFI). Experimental results on both synthetic and real-world data sets with different underlying class distributions show that the CFI strategy has the best performance.(4) Under multi-class labeling circumstances, we propose a novel algorithm GTIC which can fuzzily handle the biased labeling problem. The algorithm can improve the performance of inferring integrated labels form multiple uncertainty labelers. For the K-class labeling problem, GTIC generates concept-level features from the multiple noisy label set of each example. Then, all examples are clustered into K groups using the classic K-Means algorithm. Each group is mapped into a class and all examples in this group are assigned with the same class label. Experimental results on low quality data sets show that GTIC is superior to the state-of-the-art algorithms both in accuracy and M-AUC metrics. Besides, GTIC is about ten times faster than the MLE-and-EM-based ground truth inference algorithms. GTIC is also easy to parallelize and suitable for big data applications.更多还原