【作者】 徐海；

【导师】 万毅；

【作者基本信息】 兰州大学 ， 工程·电子与通信工程（专业学位）， 2021， 硕士

【摘要】 在计算机视觉领域,通过摄像设备采集到的图像来合成未拍摄到的视角下的图像,称为虚拟视图合成技术。上个世纪80年代初期,以Marr为代表的一些学者提出了一整套视觉计算的理论,推动了虚拟视图技术的发展。在研究虚拟视图合成技术的最初阶段,原始图像由一个沿水平轴移动的相机拍摄。进入21世纪后,虚拟现实(Virtual Reality,VR)系统被开发并实现,远程手术、全景漫游等技术飞速发展。这些技术中都有虚拟视图合成的身影。虚拟视图合成技术已越来越贴近我们的生活,在摄影、3D-TV、自由导航系统和军事等方面也具有很好的前景。虚拟视图合成算法分为两大类:基于几何模型的渲染和基于图像的渲染。基于几何模型的渲染需要用专业设备采集物体的3D信息,再用软件进行3D建模和材质渲染,进而合成虚拟视角下的视图。基于图像的渲染是根据相机拍摄的图片,通过各种手段合成虚拟视角下的视图。由于第一种方式在场景复杂的情况下渲染速度过慢而且成本过高,因此基于图像的渲染技术是研究的主流方向。目前基于图像的渲染技术一般需要大量的原始图像数据,或者需要已知相机的外参或内参,再根据深度重建的方法计算出图像上点的深度,进而合成虚拟视图。在现实生活场景中,不可能每个普通相机和拍照设备在合成虚拟视图前都被标定过,标定多个相机也会费时费力。通常也不会拍摄特别多的原始图像。因此本文希望能减少原始图像的采集张数,并适用于普通相机和拍照设备,在相机未标定的情况下能合成虚拟视图。另外本文希望仅靠拍摄某个场景的部分原始图片,就能合成不同视角下的虚拟视图,从而构建全景漫游的效果。基于以上目的,本文的主要研究内容如下:1.在相机未标定,且原始图像没有图像深度信息的情况下,本文仅根据两张原始图像,提出并建立了一个虚拟多相机模型(Virtual Multi-Camera Model)。合成了相机在旋转和平移过程中视角下的虚拟视图。该方法相比于目前具有代表性的虚拟视图合成方法Softmax Splatting相比,在原始图像视差较小(图像重合区域较多)的情况下,合成的虚拟视图效果基本一致;在原始图像视差较大(图像重合区域较少)的情况下,本文的虚拟多相机模型能合成效果更好的虚拟视图;整体而言,MSSIM指标平均高出3.8%,PSNR指标平均高出2.6%。本文的方法无论是在室内还是室外场景、白天或者黑夜场景都能获得很好的合成效果。2.本文基于虚拟多相机模型,将虚拟视图合成应用到视频插帧中。将合成的视差较小的相邻虚拟视图作为视频帧,得到了视角连续变化的虚拟视频。并且本文针对合成的虚拟视频中出现的问题提出了改进的虚拟多相机模型。改进的虚拟多相机模型的MSSIM值比目前具有代表性的插帧技术RIFE高出2.5%,合成的虚拟视频流畅程度和视觉效果可被人眼接受。更多还原

【Abstract】 In the field of computer vision,the technology that the images captured by camera is used to synthesize the images in the perspective of not captured by camera is called virtual view synthesis.In the early 1980 s,some scholars represented by Marr put for-ward a set of visual computing theories,which built the basic framework of computer vision and promoted the development of virtual view synthesis technology.In the initial stage of virtual view synthesis,the original images were captured by a camera moving along the horizontal axis.After entering the 21 st century,Virtual Reality(VR)system has been developed and implemented,and remote conference,remote surgery tech-nologies have developed rapidly.All of these technologies need virtual view synthesis.Virtual view synthesis has become the key technology of 3D-TV,free navigation sys-tem and other applications.It also has a good prospect in photography and panoramic roaming.Virtual view synthesis algorithms are divided into two categories: Model-Based Rendering and Image-Based Rendering.Model-Based Rendering needs to collect 3D information of objects with professional equipment,and then uses software for 3D mod-eling and material rendering,so as to synthesize the views under the virtual perspectives.Image-Based Rendering needs to synthesize the views under the virtual perspectives by the pictures taken by the camera.Due to the slow rendering speed and high cost of the first method in the case of complex scenes,Image-Based Rendering technology is the mainstream direction of research.In real life,it is impossible for each ordinary camera and camera equipment to be calibrated before synthesizing virtual view,and it will take time and effort to calibrate multiple cameras.Therefore,this paper hopes to reduce the number of original images,and apply to ordinary cameras,and can synthesize virtual views without calibration of cameras.In addition,this paper hopes that only by taking some original pictures of a scene,virtual views from different perspectives can be syn-thesized,so as to construct the panoramic roaming effect.The research contents and innovations of this paper are as follows:1.When the camera is not calibrated and the original images have no depth infor-mation,a Virtual Multi-Camera Model is proposed and established based on only two original images.The virtual views in the camera’s moving trajectory can be synthe-sized.This method is compared with the representative virtual view synthesis method Softmax Splatting,the effect of the synthesized virtual views are basically the same when the disparity of the original images is small(there are many overlapping regions in original images)? when the disparity of the original image is large(there are few overlapping regions in original images),the Virtual Multi-Camera Model in this paper can synthesize the virtual views with better effect? overall,the MSSIM is 3.8% and the PSNR is 2.6% higher than Softmax Splatting,The Virtual Multi-Camera Model can achieve good synthesis effect in indoor or outdoor scenes,day or night scenes.2.Based on the proposed Virtual Multi-Camera Model,this paper applies virtual view synthesis to video interpolation.The virtual video with continuous changes in view angles is synthesized by the adjacent virtual views with small parallax which are regarded as video frames.In this paper,an improved Virtual multi-camera model is proposed to solve the problems in virtual video synthesis.The MSSIM value of the improved Virtual Multi-Camera Model is 2.5%higher than RIFE which is a representa-tive frame insertion technology.The fluency and visual effect of the synthesized virtual videos meet the requirements of human visual system.更多还原