【作者】 张明亮；

【导师】 樊鑫； 叶昕辰；

【作者基本信息】 大连理工大学 ， 计算数学， 2020， 博士

【摘要】 获取稠密的高质量场景深度图像是感知和重建三维场景的一个关键环节,也是计算机视觉领域的一个研究热点。利用单幅图像来获得高质量场景深度信息的方法即为单目场景深度推断。由于该类方法本身是一个不适定问题且可利用的图像信息较少,因而难度更大、更具有挑战性。目前单目场景推断方法根据不同相机数据类型:退化深度图像、彩色图像,可以进一步分为单目深度恢复和单目深度估计两种。其中单目深度恢复方法往往存在模型刻画不精确、目标函数非凸等问题,而单目深度估计方法由于缺乏图像几何约束往往需要大量的深度标签。因此,为了解决基于异源相机数据的单目场景深度推断存在的问题,本文提出了基于稀疏诱导先验的图像分解模型以及结合场景先验知识和网络结构优化的无监督单目深度估计方法,来获取高质量的场景深度图像。其主要研究内容如下:1.基于稀疏诱导先验图像分解模型的单目深度恢复方法。现有的单目深度恢复方法往往不能刻画深度图像在不同区域的本质属性,并且往往不能保证模型目标函数的凸性,从而导致复杂的求解过程。因此,本文提出了一种基于稀疏诱导先验的图像分解模型,该模型从信号分解的角度出发,将深度图像分成平滑区域和深度跳跃区域。然后对于不同的区域分别使用最小二乘多项式和稀疏诱导先验进行拟合,来更加本质地刻画二维深度信号特征。本文提出的基于Moreau包络(envelope)的稀疏诱导先验虽然是非凸函数,但可以证明当满足0<α≤μ条件时,模型的整个目标函数关于每个变量的凸性。提出了基于近端梯度(proximal gradient,PG)与交替方向乘子法(alternating direction method of multipliers,ADMM)混合的模型优化算法,并给出相应的收敛性证明。最后,针对于算法计算量问题,提供了一种加速算法,使得程序测试时间缩短为原来的1/20。在Middleburry数据库上,关于处理深度图像噪声、低分辨率和深度值缺失退化问题的实验结果表明所提出的方法可以达到比其他方法更好的效果,相应的误差指标最高分别减少了 30.5%,21.3%,22.0%,验证了模型和算法的有效性。2.结合场景先验知识的无监督单目深度估计问题研究。目前基于监督学习的单目深度估计方法通常需要大量的真实深度标签作为训练数据。相对地,无监督单目深度估计方法通过利用立体匹配对(双目数据)或者单目视频作为训练数据,然后进行单目深度推断,克服了上述方法的不足。但由于缺乏真实深度标签的监督,其深度推断的数值结果和视觉效果往往不如基于监督学习的单目深度估计方法。为此,本文提出通过结合基于自然场景的先验知识,包括人工设计的先验和可学习的先验,利用注意力机制和矩形卷积分别从不同维度和方向获取特征相关性信息。同时设计了融入几何关系的损失函数来更好地指导网络的训练,并提出了一种可学习近端算子模块通过模仿基于变分模型的近端算法来增强网络初始估计深度图像的边缘和细节。在KITTI数据库上的定性和定量实验表明所提方法取得比现有无监督学习方法更优的深度估计结果。其中3.3实验部分的误差指标最高分别减少了 8.7%和10.5%;而3.4实验部分的误差指标最高减少了29.4%,并且在Make3D数据库上的实验结果的误差指标最高减少了 21.5%,从而表明所提方法拥有更好的泛化性能。3.基于无监督单目深度估计的深度图像精细化以及网络结构优化问题研究。由于目前的网络结构中往往存在大量的池化和下采样操作,导致特征信息的丢失,使得当前基于单个网络得到的深度推断结果通常不能令人满意。并且目前的无监督单目网络其结构设计往往存在局限性,即通常在训练时无法充分利用双目信息。为此,本章节主要包括两个方面的工作,一方面提出了一种基于级联网络结构的图像精细化方法,这种由粗到细的级联网络结构能够充分获得各个水平的互补特征,提高网络的特征表达能力。另一方面,提出了一种单目网络和双目网络相结合的新的网络结构,它可以同时允许单目或者双目图像测试。并设计了一种新的蒸馏机制通过借助于双目图像训练数据的信息来辅助单目图像深度估计,使得单目无监督深度估计网络能够学习到更加精确的几何知识。在KITTI数据库上,通过大量的定性和定量实验表明所提方法比现有的无监督学习方法的效果更好,并且优于一些基于监督学习的深度估计方法。其中,和监督学习方法相比,4.2和4.3实验部分的误差指标最高分别减少了 10.6%和20.4%,从而验证了模型的有效性。更多还原

【Abstract】 Capturing high-quality dense depth maps is a crucial link for scene perception and 3D recon-struction,and it is a popular direction in the field of computer vision.The method of obtaining high-quality depth information from single image is monocular scene depth inference.Since the method itself is an ill-posed problem and has less available image information,it is more diffi-cult and challenging.According to different camera data types,i.e.,degraded depth image and color image,monocular scene inference methods can be further divided into monocular depth recovery and monocular depth estimation.The methods for monocular depth recovery usually have some problems,such as imprecise model description and non-convex objective function.In contrast,the methods for monocular depth estimation often need a large number of depth labels due to lack of geometric constraints.Thus,in order to solve the problem of monocu-lar scene depth inference based on heterogeneous camera data,this thesis proposes an image decomposition model based on sparsity-promoting prior and an unsupervised monocular depth estimation method combined with prior knowledge and network structure optimization to obtain high-quality scene depth maps.The main contents include:1.Image decomposition model based on sparsity-promoting prior for monocular depth re-covery.The existing methods for monocular depth recovery are often insufficient to depict the essential characteristics of depth image in different regions,and can not guarantee the convexity of objective function,leading to complex solving process.Thus,from the perspective of signal decomposition,depth image can be divided into smooth regions and step discontinuities regions,and then the least square polynomial and sparsity-promoting prior are utilized to fit there two re-gions to establish a more accurate optimization model.Though the proposed sparsity-promoting prior based on the Moreau envelope is non-convex,this thesis can prove the convexity of the whole objective function for each variable under some mild conditions.The proposed model is solved by the proximal gradient(PG)method combined with the alternating direction method of multipliers(ADMM)algorithm and the corresponding convergence analysis of algorithm is also given.At last,an accelerated algorithm is provided to reduce running time in testing phase.The extensive experiments on noise,depth value missing and low resolution in Middleburry dataset demonstrate that the proposed method achieves better results than other methods,which verifies the effectiveness of the proposed model and algorithm.2.Research on unsupervised monocular depth estimation combined with prior knowledge.At present,the supervised monocular depth estimation methods usually need a large number of real ground truth depth label as training data.In contrast,the unsupervised monocular depth estimation methods overcome the shortcoming of the above methods by exploiting stereo image pairs and monocular videos during training,and infer depth map from monocular input image in testing phase.However,due to lack of the supervision of real depth label,the numerical results and visual effects for depth estimation are usually inferior to those of supervised monocular depth estimation methods.Therefore,this thesis proposes to combine the prior knowledge based on natural scene,including hand-crafted and learnable prior,in order to improve the accuracy and vi-sual effects.The proposed method exploits the attention mechanism and rectangular convolution to capture the information of feature dependencies respectively and designs a geometric-aware loss function to construct the relationship between color image and predicted depth map.The learned composite proximal operator is proposed to refine the obtained initial depth maps by sim-ulating the proximal operator based on the variational model.The qualitative and quantitative experiments on KITTI dataset show that the proposed method outperforms the existing unsuper-vised methods.The experimental results on Make3D dataset show that the proposed method has better generalization performance.3.Research on depth refinement and network structure optimization based on unsupervised monocular depth estimation.Due to the large number of pooling and down-sampling operations in the network structure,leading to the loss of feature information,the current depth inference results based on a single network are usually not satisfactory.Moreover,the structure design of the existing unsupervised monocular network often has limitations,i.e.,it can not fully explore the stereo information during training.To deal with the problems,firstly,this thesis proposes a depth refinement method based cascaded network structure.This kind of cascaded network structure can fully capture complementary features at all levels and improve the ability of feature representation in a coarse-to-fine manner.After that,a novel network structure which combines monocular network and stereo network is proposed.It allows monocular image or stereo image pairs as input in testing phase depending on the input mode.In order to further improve the performance of monocular network,this thesis proposes a new distillation mechanism to assist monocular depth prediction by using the information of stereo network during training,so that the unsupervised monocular depth estimation network can learn more accurate geometry knowledge.Meanwhile,a recursive network strategy and a feature-driven adaptive refinement module are used in the stereo network to enhance its inference performance and the ability to assist the learning of monocular network.A large number of qualitative and quantitative experiments on KITTI dataset demonstrate that the proposed method outperforms the existing unsupervised learning methods,and beats some supervised learning methods for monocular depth estimation,which verifies the effectiveness of the proposed method.更多还原