【作者】 赵杰；

【导师】 韦世奎；

【作者基本信息】 北京交通大学 ， 信号与信息处理， 2024， 博士

【摘要】 以鱼眼相机为代表的超广角相机在安全监控、虚拟现实、自动驾驶等领域有广泛应用。然而,由于鱼眼相机的非相似成像特性,导致所获取的鱼眼图像存在严重的空间几何畸变,其不仅不利于人眼观察,还严重影响诸多下游任务的性能。因此,鱼眼图像的畸变校正理论与方法的研究,受到学术界和工业界的广泛关注。近年来,随着深度学习的发展,鱼眼图像畸变校正也逐渐从传统方法向深度方法演进。为了获取深度学习所需的大规模标签,现有方法通常采用人工合成的方式构造大规模鱼眼畸变图像训练数据集。然而,在数据合成过程中,受畸变参数空间、畸变模型种类以及畸变感知范围的限制,合成的鱼眼畸变图像训练数据集具有较大局限性,导致合成-真实域泛化能力受限。针对上述问题,本文拟在鱼眼图像的渐进式校正、相机畸变模型增强、统一感知的畸变表征和合成-真实域泛化等方面开展研究,提出一系列具有针对性的鱼眼图像畸变校正方案,实现更精确更可靠的畸变校正。本文的主要研究工作如下:(1)针对基于参数估计的畸变校正方法畸变参数空间受限的问题,提出了一种基于深度强化学习的渐进式畸变校正算法。首先,将鱼眼图像的畸变校正任务建模为马尔可夫决策过程,将原本的单步校正替代为多步渐进式校正,从而扩展了畸变参数空间;其次,将畸变参数的估计转化为校正智能体动作的选取,并设计强化学习算法对校正智能体进行多轮训练,提升了参数估计的合理性;最后,结合图像质量评价指标和损失函数,设计并选取奖励函数,提升了畸变校正的精度。实验表明,在多步渐进式校正的思路下,该算法在面对轻微畸变数据时达到具有竞争力的性能,在面对严重畸变数据时达到最佳的性能。(2)针对畸变校正方法普遍存在的畸变模型种类受限的问题,提出了一种基于级联畸变模型和增强后向流网络的畸变校正算法。首先,设计了一种可逆的级联畸变模型,弥合传统畸变模型间的分布鸿沟;进一步,设计了一种畸变表征方法,从像素层面显式地建立校正图像到畸变原图之间的映射关系,从而实现了对鱼眼图像畸变更有效的表示;最后,提出了一种增强后向流预测的畸变校正网络,结合像素级的前向映射图作为辅助,增强对后向映射图的学习,提升了畸变校正效果。在合成数据集和真实畸变图像上的实验结果表明,该算法在面对任意畸变模型数据时都取得最佳校正性能。(3)针对畸变校正方法存在的畸变感知范围受限的问题,提出了一种基于统一畸变感知的畸变校正算法。首先,设计了一种易于网络学习的径向畸变表征,并利用径向表征之间的畸变一致性互相增强各表征的畸变信息,实现了对不同形状的鱼眼图像畸变的统一表征;其次,设计了一种多尺度融合的畸变感知网络,将多层多尺度并行结构得到的畸变特征分析融合,生成最终的畸变表征,实现了统一畸变表征的有效学习;最后,采用大规模预训练联合参数微调的方式,对统一畸变感知的畸变校正方算法进行训练,提升了畸变校正的性能。在合成数据集和真实畸变图像上的实验结果表明,该算法在取得较高图像质量评价的基础上,对不同感知范围的畸变数据都具有良好的畸变感知能力。(4)针对畸变校正方法存在的合成-真实域泛化能力受限的问题,提出了一种面向真实场景的自监督畸变校正算法。首先,基于真实畸变图像构建大规模畸变数据集,避免了合成图像存在的视场受限问题;其次,引入畸变参数估计网络作为进一步预测的基准,解决了真实图像训练过程中难以获取标签的问题;最后,基于真实图像的视场范围,采用内在一致性约束的自监督方法对网络进行训练,提升了畸变校正的泛化性能。在真实畸变图像数据集上的实验结果表明,该算法处理真实畸变图像时更具鲁棒性,并能得到精确的校正结果。综上所述,本文重点解决鱼眼图像的畸变校正问题,提出了基于深度强化学习的渐进式畸变校正算法、基于级联畸变模型的任意模型畸变校正算法、统一畸变感知的任意形状畸变校正算法以及面向真实场景的自监督畸变校正算法,提升了畸变校正任务的准确性、鲁棒性和泛化能力。更多还原

【Abstract】 The fisheye camera,representing ultra-wide-angle cameras,has widespread applications in security monitoring,virtual reality,and autonomous driving.However,the non-similar imaging characteristics of fisheye cameras lead to significant spatial geometric distortions in the acquired images,which not only hinder human observation but also severely affect the performance of many downstream tasks.As a result,the research on distortion rectification theories and methods for fisheye images has garnered considerable attention from both academia and industry.In recent years,with the development of deep learning,fisheye image rectification has gradually shifted from traditional methods to deep learning approaches.To obtain the large-scale labeled data required for deep learning,existing methods typically rely on artificially synthesized datasets of fisheye distorted images.However,due to limitations in the distortion parameter space,types of distortion models,and the perceptual range of distortions,the synthesized training datasets have significant limitations,resulting in restricted synthetic-to-real domain generalization capabilities.To address these issues,this dissertation proposes research in areas such as gradual rectification of fisheye images,enhancement of camera distortion models,unified perception of distortion representation,and synthetic-to-real domain generalization,aiming to develop a series of targeted distortion rectification solutions for fisheye images to achieve more accurate and reliable rectification.The research in this dissertation are as follows:(1)Deep reinforcement learning based gradual distortion rectification method.First,the distortion rectification task for fisheye images is modeled as a Markov decision process,replacing the original single-step rectification with a multi-step progressive rectification,thereby expanding the distortion parameter space.Second,the estimation of distortion parameters is transformed into the selection of actions for the rectification agent,and a reinforcement learning algorithm is designed to conduct multiple rounds of training for the rectification agent,enhancing the rationality of parameter estimation.Finally,by combining image quality evaluation metrics and loss functions,a reward function is designed and selected to improve the accuracy of distortion rectification.Experiments show that under the multi-step progressive rectification approach,the algorithm achieves competitive performance when dealing with mildly distorted data and optimal performance when handling severely distorted data.(2)Model-free distortion rectification method based on cascaded distortion model and enhanced backward flow networks.First,an invertible cascading distortion model is designed to bridge the distribution gap between traditional distortion models.Furthermore,a distortion representation method is developed to explicitly establish a mapping relationship from corrected images to distorted originals at the pixel level,allowing for a more effective representation of fisheye image distortions.Finally,an enhanced backward flow network distortion rectification method is proposed,combining pixel-level forward mapping as an auxiliary to improve the prediction of backward mapping,thus enhancing distortion rectification results.Experimental results on synthetic datasets and real distorted images show that this algorithm achieves optimal rectification performance when dealing with any distortion model data.(3)Shape-free distortion rectification method based on unified distortion perception.First,a radial distortion representation that is easy for the network to learn is designed,enhancing the distortion information of each representation through the consistency of radial representations,thereby achieving a unified representation of distortions in fisheye images of different shapes.Second,a multi-scale fusion distortion perception network is designed,which integrates distortion features obtained from a multi-layer,multi-scale parallel structure to generate the final distortion representation,enabling effective learning of the unified distortion representation.Finally,a large-scale pre-training and joint parameter fine-tuning approach is adopted to train the unified distortion perception rectification method,enhancing its performance.Experimental results on synthetic datasets and real distorted images show that this algorithm exhibits good distortion perception capabilities across different distortion data while achieving high image quality evaluation scores.(4)Self-supervised distortion rectification method enhanced by real images.First,a large-scale distortion dataset is constructed based on real distorted images,avoiding the field-of-view limitations present in synthetic images.Second,a distortion parameter estimation network is introduced as a baseline for further predictions,addressing the challenge of obtaining labels during the training process with real images.Finally,based on the field-of-view range of real images,a self-supervised training method with intrinsic consistency constraints is employed to improve the generalization performance of the distortion rectification network.Experimental results on real distorted image datasets demonstrate that this algorithm is more robust when processing real distorted images and can achieve accurate rectification results.In summary,this dissertation focuses on the distortion rectification problem for fisheye images and proposes a gradual distortion rectification method based on deep reinforcement learning,a model-free distortion rectification method based on cascaded distortion model,a shape-free distortion rectification method based on unified distortion perception,and a self-supervised distortion rectification method enhanced by real images.These approaches improve the accuracy,robustness,and generalization ability of distortion rectification tasks.更多还原