【作者】 张艳丽；

【导师】 林路；

【作者基本信息】 山东大学 ， 概率论与数理统计， 2015， 博士

【摘要】 判别分析是利用已知类别的样本建立判别模型,对未知类别的样本进行判别归类的一种统计方法。近年来,判别分析在医学、自然科学、社会学及经济管理学科中都有广泛的应用。判别分析的特点是根据已有的每个类别的若干样本的数据信息,以概率统计理论为基础,建立判别准则。对于新的样本点,利用得到的判别准则,就能判别该样本点所属的类别。常用的判别分析方法有Bayes判别和Fisher判别。Bayes判别是一种概率型的判别分析方法,在分析过程开始时需要获得各个类别的分布密度函数,同时也需要知道样本点属于各个类别的先验概率,以建立一个合适的判别规则。但是实际中往往不知道各类别的分布密度,而且当判别变量较多时,计算也较为复杂,此时,Fisher线性判别分析方法占有优势。Fisher判别的基本思想是投影降维,使多维问题简化为一维问题来处理,实际上就是把特征空间压缩到一维。问题的关键是投影之后原来可以分的开的样本可能变得混杂在一起而无法区分开。所以在Fisher判别分析中,寻找投影方向是关键。在一般情况下,总可以找到某个最好的方向,使样本投影到这个方向的直线上是最容易分得开的,即每一类内的投影值越集中越好(投影后类内离差尽可能小),而不同类间的投影值差别越大越好(类间离差尽可能大)。基于Fisher判别基本思想,在两总体,协方差阵相等的条件下,而且p<n时,Fisher判别方向为β=∑-1(μ1-μ)。当∑、μ1、μ2未知时,分别用样本协方差阵S,两总体的样本均值X1,X2来估计。已有的文献证明,当n→∞时,Fisher判别是渐近最优的。随着科学技术的迅速发展,人们开始越来越关注如何从大数据中获取信息,比如基因组学,功能磁共振成像、风险管理、信号处理、气候和Web搜索问题。在这些问题中变量维数p可能比样本数n大很多或者样本协方差阵S的逆不存在。这种情况下,Fisher判别准则的错判概率接近1/2,也就是说此时线性判别分析相当于随机猜测。针对这一问题作者在第二章和第三章分别提出了所谓的Dantzig判别方法和Lasso判别方法,用来在高维数据下,直接估计Fisher线性判别方向β。第二章提出的Dantzig判别方法基于Fisher判别的解正是线性回归最小二乘解这一结论,考虑惩罚最小二乘估计来估计判别方向β。Dantzig判别方法通过最小化目标函数|β|1+λn|X(Y-XTβ)|∞,得到估计β的估计,进而得到相应的线性判别准则。在β渐近稀疏条件下,第二章证明了Dantzig判别方法的错判率是渐近最优的,同时我们用数值模拟和实例验证把我们的方法和其它已有的方法进行了比较,验证了方法的优良性。相比于大部分已有的方法,我们的方法不用去分别估计∑-1(或者∑)和μ1-μ2,从而也不要求∑’和μ1-μ2的稀疏性,提高了判别效率。第三章针对线性判别方向声,提出了一种Lasso估计方法,得到的判别方法我们称为Lasso判别方法,其想法基于2012年Wang Tao的关于广义指标模型方向估计方法,估计如下其中Fn(Y)是Y的经验分布函数,得到是一种非参数判别方法。和已有的大多数高维判别方法相比,有以下优点：(1)判别方向β与最小二乘具有等价关系,但Y和XTβ不一定满足经典的线性回归模型可加结构,所以用经典的线性回归模型变量选择方法去估计判别方向β是不合适的；(2)因为在经典的线性回归模型,要求{Yi,Xi}i=1n是独立同分布的,但判别分析问题中,{Yi,Xi}n=1n不是独立同分布的,因为{Xi}i=1n来自不同的总体,具有不同的分布,从这点来说性质和经典的回归模型下回归系数估计的性质有很大差别；(3)我们选取h(y)为F(y),用Fn(y)来估计F(y),计算简单。在§3.3中,给出了Lasso判别分析的相合性,在§3.4中用数值模拟和实际数据把我们的方法和其它已有的方法进行了模拟比较,验证了方法的优良性。第四章我们研究了基于测量误差模型选择似然函数,在p>n的情况下对线性回归中的变量进行选择与对应系数估计的性质。我们知道关于高维数据下的变量选择方法在过去几十年,理论和方法都得到了巨大的发展。常用的传统的变量选择方法是子集选择法,比如AIC、KIC、GIC、BIG和Cp准则等。但是子集选择方法可能会因为庞大的计算量或其他原因失效,而且子集选择方法不稳定,也就是说变量选择结果可能会因为数据的微小变化而发生很大的变化。所以针对高维数据的变量选择问题,近年来提出的系数压缩法因为能同时达到变量选择和参数估计的目的而备受关注。系数压缩方法以Tibshirani(1996)提出的基于惩罚思想的Lasso变量选择方法为标志,能够在选择变量的同时得到参数的估计,而且计算量比传统的变量选择方法小很多,因此基于惩罚思想的变量选择方法受到了广大统计学者的广泛关注。除Lasso方法外,还有SCAD、Dantzig、Least angle regressions election、Elastic net等方法。在线性回归模型下,通过选择非零系数βj来选择变量,但是如果变量不是以参数的形式进入模型,就不能再以直观地通过选择参数来选择变量了,比如非参数模型和算法拟合模型等。显然,不管什么模型,变量中包含误差是很直观的。如果一个变量加上误差对回归函数没有影响,说明这个变量不重要,舍弃这个变量,因此,可以利用变量包含的误差的大小选择变量。第四章基于测量误差模型,提出估计的目标函数,在惩罚总的测量误差的条件下让测量误差选择似然函数达到最大。具体估计如下：其中σλ2=βTD(λ/1)β+σ2,D(λ/1)=diag{1/λ1,…1/λp),λj=1/σU,j2,τn是惩罚系数。得到的估计β(λ)是path相合的。第五章,我们考虑非参数回归模型选择问题。设(Xi,Yi),i=1:…,n是独立同分布样本,具有联合密度函数p(x,y)=f(x)g(y|x),条件均值函数为m(x)=E(Y|X=x),目的是估计条件回归函数m(x)。回归模型的形式通常地说有参数模型,非参数模型和半参数模型。参数模型就是假设n(x)有参数结构形式m(x,β),函数m(.)形式已知,估计m(x,β)只需估计未知参数β。如果回归函数确实具有参数形式m(x,β),此时可以得到精度很高的估计。但是如果对回归函数假设的参数模型形式不对,估计的偏差会很大,估计精度也很差。非参数模型是对回归函数形式不作参数假设,就产生了非参数估计方法。统计学者提出了很多非参数估计方法,比如核估计、局部线性估计、局部多项式估计等等。其中局部线性估计在高于一维的情况下是最小最大化估计。然而,随着协变量维数的增加,局部线性估计也不再是一个好的估计,这就是非参数问题中存在的“维数灾难”问题。针对非参数估计的“维数灾难”问题,很多学者提出了有模型限制的非参数模型和半参数模型,比如可加模型,部分线性模型等,在一定程度上解决了”维数祸根”问题。可加模型和半参数模型实际上还是对模型作了假定,如果回归函数不具有假定的模型形式,估计精度同样会很差。此外,还有一种半参数模型不用假定模型形式(称为全模型),其指导思想是一些先验或经验信息有时可以为回归函数的形式提供一些信息,把这些先验信息应用到建模过程中去,这样得到估计既不会想参数估计那样有很大的偏,又不会像非参数估计那样不稳定。如何把这些先验信息应用到建模过程中去,为构造模型形式提供帮助是关键问题。Studer(2005)提出的一种惩罚非可加结构的非参数模型和Lin(2013)提出的局部线性可加估计为这个问题奠定了基础。第五章基于任何一个函数都可以分解成关于自变量的线性部分和线性剩余部分,提出了一种惩罚模型选择方法,把模型的线性部分和非线性部分结合起来考虑,得到一族连续的半参数模型rλ。当参数λ=0时对应全模型,即是局部线性估计,当λ=∞时,对应于线性模型。此估计避免了“维数灾难”问题,是线性估计和局部线性估计的一个组合。当回归函数真的具有线性结构时,估计的收敛速度达到参数速度((?))。我们给出了估计的渐近性质,并用数值模拟例子验证了我们的方法。更多还原

【Abstract】 Discriminant analysis is a kind of statistical method which uses the histor-ical data of clear classification to establish discriminant model and then classify the unknown samples. In recent years, discriminant analysis is widely used in medical science, natural science, sociology and economics and management science. The key point of discriminant analysis is to establish discriminant function based on the existing data information of each category by using the theory of probability and statistics.Then for the new sample, we can judge which category it belongs to by the discriminant criterion.Two of the most popular methods in discriminant analysis are the Bayes discriminant analysis and Fisher discriminant analysis. Bayes discriminant analysis is a method based on the information of the probabilities of the data. We need to know the probability density function of each category and the prior probability of the data belongs to each category. However, in practice it’s difficult to know the distribution density of each class, and the calculation is more complex when the number of discriminant variables is large. At this time, fisher linear discriminant analysis method is much better than the Bayes discriminant analysis.The basic idea of Fisher discriminant is to reduce dimension by projec-tion in order to simplify the multidimensional problem to a one-dimensional problem which means to compress the feature space to the one-dimension. The trouble is the sample which can be separated originally may be mixed together and can not be separated after the projection. Thus in Fisher discrimination, it’s quite important to find the projection direction. In general, we can always find the best direction which makes samples are easy to separate. That is to say, the projected values are concentrated in the same category and scattered between different categories by this direction.When β=∑-1(μ1-μ2), the rule is Fisher discriminant when the two populations have same covariance in the case p< n based on the idea of Fisher discriminant. When and ∑、μ1、μ2 are unknown, the Fisher rule, respectively, use S, X1 and X-2 to estimate them,where It has been shown that the rule is asymptotically optimal when n→∞.With the rapid development of science and technology, people begin to pay more and more attention to how to obtain the information from big data in many fields such as genomies, functional magnetic resonance imaging, risk management, signal processing, the climate and the Web search problem. In these problems, the number of variables p may be much larger than the number of samples n or the inverse sample co-variance matrix S does not exist. In this case, the error probability of Fisher criterion will be close to 1/2, that is to say the linear discriminant analysis is equal to Random guesses at this moment.To solve this problem, the author put forward the so-called Dantzig dis-criminant method and the Lasso discriminant method in second and third chapter which estimates the discriminant direction (3 directly in high dimen-sional data. The Dantzig discriminant method in chapter 2 derives from the idea that Fisher discriminant direction is just the least square solution of linear regression. We consider the penalized least squares to estimate the discrim-inant direction β. Specifically, we get the corresponding linear discriminant, criteria by minimizing the objective function |β|1+λn|X(Y-XTβ)|∞,The miselassification rate of Dantzig discriminant method is asymptoti-cally optimal if β is asymptotic sparse. We confirmed the optimal nature of the method in both the numerical simulation and real data compared to other existing methods. As our method doesu’t need to estimate ∑-1 and μ1-μ2, the sparsity of ∑-1 and μ1-μ2 is not needed and then improve the efficiency for discriminant.In the third chapter, we proposed a Lasso estimation method for linear discriminant direction β. The method is based on the idea of a generalized single index model by Wang Tao 2012,the estimation is as follows: Where Fn(Y) is the empirical distribution function of Y. Compared with most of the existing high dimensional discriminant methods, our method has the following difference:(1) Though the discriminant direction β and least squares are equivalent. Yand XTβdo not necessarily satisfy the classical additive structure in the linear model. Thus it is not appropriate to use the variable selection method in the classical linear regression model to estimate the discriminant direction β;(2) There is a big difference between the properties of regression model estimator and discriminant direction. In the classical linear regression model, {Yi,Xi}in=1 are independent and identically distributed, but in discriminant analysis,{Yi, XI}=in=1 are not independent and {Xi}in=1 come from different, pop-ulations;(3) We choose h(y) as F(y) and estimate F(y) by Fn(y) in order to make the calculation simple.We show the consistency of Lasso discriminant analysis in section §3.3 and the simulated results in §3.4. Our method performed well compared with the existing methods.In the fourth chapter, we study the properties for the method of selecting the important variables and estimating the corresponding coefficient in linear regression model when p> n via measure error model selection likelihood. As we know, the theory and method about the choice of variables in high dimensional data have been developed greatly in the past few decades. The commonly used method of variables selection is subset selection such as the AIC, KIC, GIC. BIC and Cp information criterion.But the method may be ineffective due to the large amount of computation or other reasons. Besides, the method is not stable meaning that variable selection results may change a lot when there is little change in the data. So a new method about the variable selection and estimation has been concerned widely, which is called the coefficient compression method.The compressed coefficient method is based on the idea of punishment that proposed by Tibshirani (1996) and has obtained great development in recent years, such as SCAD, Dantzig, Least angle regressions election, Elastic net and so on. These methods are used in the linear parametric regression model and select, variables through the nonzero coefficients of βj. However, if the variable is not contacted with parameter in the model, the methods can’t be used by choosing parameters, such as the non parametric model and algorithm fitting model.It’s intuitive that the variables contain the error in the model. If a variable with the error has no effect on the regression function, it means that this variable is not important so we can abandon it. Therefore, we can select the variable by the error contained in the variables. The fourth chapter build the objective function which aims to maximize the likelihood function under the condition of the punishment of the total measurement error based on the measurement error model. The specific estimates are as follows: Where σλ2= βιD(λ/1)β+σ2, D(λ/1)= diag{1/λ1,…,1/λP},λj,=1/σU2,τn is tuning parametric.The estimates obtained by β(λ) path is consistent while the consistency can only be confirmed under strong conditions in Lasso and other methods. We show the good properties of the estimator using numerical simulation.In the fifth chapter, we consider the model selection of nonparametric regression. Suppose (Xi, Yi),i.= 1,…,n are independent and identically dis-tributed samples with the joint density function p(x,y)= f(x)g(y|x), the conditional mean function is m(x). The aim is to estimate the conditional re-gression function m(x). The form of the regression model includes parametric model, non-parametric model and semi-parametric model. Parametric model assume that the form of m(x) is already known and we only need to estimate the unknown parameters of β. If the assumption is true then the precision of the estimator is high. However, the bias of the estimation would be large and the accuracy would also be poor if the rue regression function is not parametric form.Non-parametric model doesn’t make any assumptions about the form of the regression function. Statisticians put forward many non-parameter estima-tion methods such as kernel estimation, local linear estimation, local polynomi-al estimation etc. The local linear estimation is the mini-maximum estimation in the multi-dimensional. However, with the increase of dimension, local lin-ear estimation is no longer a good estimation. So the "dimension disaster" problem arises in non-parameter estimation.In view of the "dimension disaster" problem, many scholars have put for-ward restricted non-parametric model and semi-parametric models, such as the additive model, partial linear model. They solve the "curse of dimension" prob-lems to a certain extent. In fact, the additive model and the semi-parametric model still make an assumption on the regression model and the accuracy would be very poor if the model assumption is wrong.In addition, there is a kind of semi-parametric model without assuming the model form (known as the full model). Its basic idea is that some prior or experiential information may provide useful information for the form of the regression function. Then the estimator would not have large bias as the para-metric estimation and would not be unstable as non-parametric estimation.How to put these prior information into the modeling process is the key. The non-parametric model penalizing nonadditivity proposed by Studer (2005) and local-linear additive estimator proposed by Lin (2013) have built the foun-dation for this problem.We put forward a kind of punishment model selection method based on the fact that any function can be decomposed into a linear part and the remaining parts in the fifth chapter. It combined linear parts with nonlinear parts and get a family of continuous semi-parametric model rλ. When the parameter λ= 0 it becomes the full model and when λ=∞ it is the linear model. This estimator avoids the "dimension disaster" and it’s a combination of linear model and full model. When the true regression function is linear, the rate of convergence reach parameter convergence rate((?)n). We give the asymptotic properties of the estimator and exam our method by numerical examples.更多还原