【作者】 李青云；

【导师】 张小梅；

【作者基本信息】 武汉理工大学 ， 信息与通信工程， 2022， 硕士

【摘要】 作为重要的信息交流与传播的载体,视频具有直观性,蕴含丰富的信息量,与计算机视觉任务和个人生活息息相关。然而由于相机设备受到外部和内部的干扰,视频帧会表现出不受控制的振荡,并可能伴随着模糊效应。现阶段对于视频去抖的算法研究大致可分为两类,一类是基于传统算法的视频去抖研究,但其在面对没有纹理或遮挡区域、大前景运动物体、光照变化的场景下会出现性能急剧下降甚至失效的情况;另一类是基于深度卷积神经网络的视频去抖研究,这类方法倾向于将抖动帧图像变换回归到稳定帧图像,或者直接以端到端的学习方式对抖动视频进行插帧生成稳定的视频,这类方法缺乏可控性,性能非常容易极端化,且其训练过程中需要大量的抖动视频和稳定视频对,但这类数据在实际生活中是很难获取到的。本文针对上述挑战进行了深入的研究,提出一种基于光流与网格轨迹的全帧视频去抖方法,主要研究内容如下:(1)面向抖动视频的光流估计。针对现有的光流估计网络在面对复杂场景下,出现光流细节丢失的问题,提出结合空洞卷积的特征金字塔网络,利用空洞卷积提取包含更多运动信息的特征,探究从粗到细的相邻帧间光流估计;针对光流计算开销较大的问题,引入卷绕扭曲、匹配代价来减少计算开销;针对光流误差常常由于遮挡而引起的问题,使用正反一致性校验来对遮挡进行检测,对遮挡部分的像素点进行识别,剔除由于遮挡引起的光流误差,以无监督学习的方式实现光流估计。在基准数据集KITTI上实验的结果表明,本文的光流估计方法性能提升了5%左右。(2)结合光流的网格轨迹提取。首先本文基于关键点检测网络对视频帧与帧之间的关键点进行了提取,针对关键点的鲁棒性选择问题,采用光流进一步对关键点匹配进行校验,同时生成鲁棒的特征运动矢量场;针对视频中存在多平面运动的情况,对整个视频帧使用全局单应性变换会造成剪切伪影和大残差的现象,提出多平面单应性变换替代全局单应性变换的方法,并设计基于残差运动向量的网格轨迹提取网络,从特征点轨迹中恢复出网格顶点的运动轨迹,分别对每个子平面进行单应性变换,以“分治”的思想缓解剪切伪影和大残差的问题。实验结果表明,本文选择的关键点在不同的误差阈值上有不同程度的提升,最严格误差下的性能提升达到了7%左右,且网格轨迹提取网络能以“尽可能相似”选择特征点。(3)基于轨迹平滑与边界外视图合成的视频去抖。针对现有的自适应路径平滑算法内核权重固定,仅仅与相邻帧时间有关,无法保证空间一致性约束的问题,提出了基于3D卷积的帧间网格轨迹平滑网络,利用3D卷积对时空信息进行提取融合,根据轨迹的特性生成动态平滑权值,完成网格轨迹的平滑。针对现有的投影变换存在裁剪窗口,造成稳定视频帧画面过小的问题,使用即插即用的边界外视图合成算子进行进一步精对齐,利用正反一致性光流检测重叠区域,获取边界外视图像素,实现近乎全帧的视频去抖。最终在多类复杂场景的基准数据集下,使用裁减率、失真率和稳定性多种度量标准评估本研究方法的有效性与优越性,与当前最先进的视频去抖研究相比,本文模型性能有不同程度的提升。更多还原

【Abstract】 As an important tool for information exchange and dissemination,videos are intuitive and contain a wealth of information,which are closely related to computer vision tasks and personal life.However,due to the external and internal disturbances of the camera device,the video frames will show uncontrolled oscillations,possibly accompanied by blur effect.At present,the algorithm research on video stabilization can be roughly divided into two categories.One is based on traditional algorithms,but its performance will deteriorate sharply or even fail in the cases of less texture or occlusion,large foreground moving objects and changing lighting.The other is the research based on deep convolutional neural networks,which tends to transform the shaking frames back to the stable frame images,or directly insert the stable frames to shaking video by end-to-end learning.This kind of method is lack of controllability,the performance is very easy to be extreme,and it requires a large number of shaking video and stable video pairs in the training process.However,this kind of data is very difficult to obtain in real life.In this paper,the above challenges are deeply studied,and a full-frame video de-dithering method based on optical flow and grid trajectory is proposed.The main contents are as follows:(1)Optical flow estimation for shaking video.In view of the loss of optical flow details of the existing optical flow estimation network in complex scenes,a feature pyramid network combined with dilated convolution is proposed,which uses dilated convolution to extract features containing more motion information.Explore the optical flow estimation between adjacent frames from coarse-to-fine;Aiming at the high computational cost of optical flow,warping and cost volume are introduced to reduce the computational overhead.In order to solve the problem that the optical flow error is often caused by occlusion,the positive and negative consistency check is used to detect the occlusion,identify the pixels of the occluded part,and eliminate the optical flow error caused by occlusion.Realizing the optical flow estimation in the way of unsupervised learning.The experimental results on the benchmark data set KITTI show that the performance of the optical flow estimation method in this paper is improved by about 5%.(2)Mesh trajectory extraction combined with optical flow.Firstly,this paper extracts the key-points between video frames based on the key-points detection network.For the robust selection of key-points,the optical flow is used to further verify the key-points matching,and at the same time,a robust feature motion vector field is generated.In view of the existence of multi-plane motion in the video,the use of global homography transformation for the whole video frames will cause shear artifacts and large residuals.A method of multi-plane homography transformation is proposed.A mesh trajectory extraction network based on residual motion vector is designed to recover the motion trajectories of mesh vertices from the key-points trajectories,and each sub-plane is transformed by homography.The idea of "divide and conquer" is used to alleviate the problem of shear artifacts and large residuals.The experimental results show that the key-points selected in this paper are improved in varying degrees on different error thresholds,and the performance improvement is about 7% under the strictest standard,and the mesh trajectory extraction network can select key-points in the "as similar as possible" way.(3)Video stabilization based on trajectory smoothing and Out-of-bounds View Synthesis.Aiming at the problem that the kernel weight of the existing adaptive path smoothing algorithm is fixed,which is only related to the adjacent frame time.An inter-frame grid trajectory smoothing network based on 3D convolution is proposed.The spatiotemporal information is extracted and fused,and dynamic smoothing weights are generated according to the characteristics of the trajectory.Aiming at the problem that the existing projection transformation has a cropping window,which causes the stable video frames to be too small,the Out-of-bounds View Synthesis operator is used for further fine alignment,and the positive and negative consistent optical flow is used to detect the overlapping area and obtain the boundary.Finally,the effectiveness and superiority of our research are evaluated under the benchmark datasets of multi-complex scenes using various metrics of stability,croping rate and distortion rate.The performances have been improved in varying degrees.更多还原