【作者】 李英杰；

【导师】 黄刘生； 杨威；

【作者基本信息】 中国科学技术大学 ， 信息安全， 2021， 硕士

【摘要】 去除前景遮挡任务是目前计算机视觉领域中一项非常十分重要的任务,是计算机视觉领域中很多常用任务的基础。例如如果能够高效的分离重建相互重叠的多个物体,那么检测识别任务的准确率将极大的提升,在自动驾驶任务中,也需要去除树木和马路隔离栏造成的遮挡,以便更好的识别远处的车辆、行人。分离重叠物体、去除前景遮挡能够有效提升很多广泛应用的计算机视觉任务的算法效果。因此,如何有效去除前景遮挡、重建清晰的背景图像成为了一个极为重要并且富有意义的研究内容。相比于单视图图像,光场图像包含丰富的视角信息,因此光场图像在去除前景遮挡任务上有着巨大的优势,但是由于光场图像为四维信息,传统图像处理方法以及现有的深度学习方法均难以充分利用光场图像中的信息。目前现有的光场图像去遮挡算法通常仅选择部分子孔径图像,或者只是简单地堆叠所有子孔径图像,以重建中心视图,这会破坏子孔径图像的空间布局。本文提出了两种简单有效的光场图像去遮挡算法,解决了上述光场图像难以处理的问题,使得光场图像能够得到更加充分有效的利用。本文主要贡献如下:1、提出使用四维卷积搭建神经网络来应对高维光场图像特征难以充分利用的问题。保留光场图像的全部子孔径图像并保持全部子孔径图像的空间布局,舍弃以往算法采用的的极平面思路,将光场图像构造为五维张量作为输入。通过采用四维卷积、反卷积设计网络,使得在神经网络中能够保持光场的四维空间结构。网络框架采用Autoencoder-decoder框架提取全局语义信息,并加入跳层链接以保留低层局部特征。2、添加遮挡掩膜识解码器分支,同时进行无遮挡中心子孔径图像预测和遮挡掩膜识别。避免直接使用预测得到的无遮挡中心子孔径图像重建结果,则会对图像中未被遮挡区域造成破坏,影响重建图像的清晰度与平滑度。本文使用两个解码器分支同时进行遮挡物对应的像素的标记和无遮挡图像的还原,最终输出为输入图像中未被遮挡部分和被遮挡部分的遮挡去除结果的合成。3、引入感知损失函数用以衡量去除遮挡结果与真实值Groundtruth之间的感知差异。传统的均方损失函数仅能够衡量两种图像像素之间的差异,无法衡量人类对两张图像感知差异,难以反映图像质量的好坏。而感知损失函数最早用于图像风格迁移,能够很好衡量图像之间的感知相似性。本文使用感知损失函数与均方误差损失函数共同评估恢复结果,并使用交叉熵损失函数用来评估遮挡遮罩预测的结果,获得了色彩更加准确丰富,过度更加平滑的重建结果。4、提出使用Transformer结构与传统卷积神经网络相融合构建网络,引入注意力机制来进一步应对高维光场图像特征难以充分利用的问题。本文将光场图像中全部子孔径图像沿颜色通道进行堆叠,采用二维卷积减少计算量。网络结构仍采用Autoencoder-decoder框架,在瓶颈处加入Bottleneck Transformers结构,加入MHSA层引入注意力机制。本文提出了两个光场图像去遮挡算法,使用了两种方法来更加充分利用光场图像中空间和角度信息,并且提出了共同进行无遮挡中心子孔径图像预测和遮挡掩膜识别、使用感知损失函数等光场图像去遮挡优化方法。通过消融实验的定量结果,证实了遮挡掩膜识别分支的有效性。实验采用公开的具有可移除遮挡物的光场图像进行训练和测试,两个算法在三个评价指标上均超越了目前最先进的光场图像去遮挡算法。实验结果表明,本文提出的两个算法均能有效光场图像中的信息,能够生成正确清晰的中心无遮挡子孔径图像重建结果。更多还原

【Abstract】 At present,the foreground occlusion removal task is very important in the computer vision field,because occlusion removal is the basis of many common tasks in the computer vision field.For example,in the task of target detection and recognition,if overlapping objects can be separated and reconstructed efficiently,the accuracy of detection and recognition can be effectively improved.Moreover in the task of Autopilot,occlusion removal is also necessary,the algorithm can better identify distant vehicles and pedestrians if the occlusion caused by trees or road barriers can be removed.Overlapping objects separation or foreground occlusion removal can effectively improve the effect of many widely used computer vision algorithms.Therefore,how to effectively remove foreground occlusion and reconstruct a clear background image has become an extremely important and meaningful research content.Compared with the single-view image,the light field image contains numerous sub-aperture images(SAIs)at different positions and angles,so it has a huge advantage in foreground occlusion removal task.However,since the light field image is four-dimensional data,traditional image processing methods and existing deep learning methods did not take advantage of the light field image.Current light field occlusion removal approaches usually select only a part of sub-aperture images or simply stack all SAIs to reconstruct the center view,which destroys the spatial layout of SAIs.In this dissertation,we proposes two simple yet effective light field image de-occlusion methods to better use the light field,so that the light field images can be used more fully and effectively.The main contributions of this dissertation are as follows:1.We proposed to use four-dimensional convolution to build the neural network to solve the problem that the high-dimensional light field is difficult to make full use of.Keep all the sub-aperture images of the light field image and keep the spatial layout of all the sub-aperture images,discard the polar plane idea used by previous algorithms,and construct the light field image as five-dimensional tensor as input.By using four-dimensional convolution and deconvolution to design the network,the four-dimensional spatial structure of the light field can be maintained in the network.The network framework uses the Autoencoder-decoder framework to extract global semantic information and use skip-connections to preserve low-level local features.2.Add the occlusion mask decoder branch,use two decoder branches to jointly learn the occlusion mask with the center view reconstruction.If we uses the center view reconstruction result directly,the unoccluded area in the image will be affected,and will affect the clarity and smoothness of the reconstructed image.In this dissertation,two decoder branches are used to jointly mark the pixels corresponding to the occluded object and rebuild the occluded background,and the final result consists of the two parts together.3.Perceptual loss is used to measure the perceptual difference between the de-occlusion result and the groundtruth.The traditional mean square loss function can only measure the pixels difference between two images,it cannot measure human perception difference of two images,and it is difficult to reflect the quality of the image.The perceptual loss function was first used for image style transfer and can well measure the perceptual similarity between images.This dissertation uses a combination of perceptual loss function,mean square loss function and cross-entropy loss function to make the de-occlusion result more real and clear.4.This dissertation proposes to use Transformer structure and traditional convolutional neural network to build a network,and use attention to deal with high-dimensional light field images.We stacked all sub-aperture images in the light field image along color channel,and use 2D convolution to reduce the calculation amount.The network structure still uses the Autoencoder-decoder framework,adding the Bottleneck Transformers structure(MHSA layer)at the bottleneck to bring attention mechanism.In this dissertation,we proposes two light field image de-occlusion algorithms,using two different methods to make full use of the space and angle information in the light field image,and proposes to jointly learn the occlusion mask with the center view reconstruction.Extensive evaluations demonstrate that our methods surpass the state-of-the-art approaches across different datasets.Moreover,visualizations show that our methods predicts the occlusion mask precisely and the reconstructed center view looks more realistic than other approaches.更多还原