【作者】 刘旭；

【导师】 高文； 范晓鹏；

【作者基本信息】 哈尔滨工业大学 ， 计算机科学与技术， 2025， 博士

【摘要】 单目深度估计是通过单个视觉传感器来推断场景的三维深度信息。与立体视觉深度估计方法有所不同,单目深度估计易于部署以及在数据采集上更为简单和经济,被广泛应用于自动驾驶、三维重建和机器人导航等领域。现有大多数图像帧相机的采样帧率和动态范围低,在高速运动场景会出现运动模糊或弱光照场景存在纹理丢失,从而极大影响单目深度估计精度。神经形态视觉传感器具有高时域分辨率、高动态范围和数据冗余小等特点,为解决极端场景感知难题提供了新路径。当前主流方法采用”先重建后估计”的级联框架,通过将异步脉冲事件流转换为标准图像再执行深度估计,但该范式存在四重缺陷:动态特征在插值重建过程中被低通滤波削弱,中间图像生成模块引入的运动模糊与噪声造成误差累积,重建图像帧率瓶颈制约高速目标几何变化的捕捉能力,以及两阶段处理导致的推理延迟倍增。因此,如何直接从事件流中构建端到端深度估计模型,并协同融合图像与事件流的异构视觉特征,成为突破现有技术瓶颈的关键挑战,研究神经形态视觉单目深度估计具有重大挑战与应用价值。本文围绕“面向神经形态相机的单目深度估计方法研究”这一前沿课题,聚焦于“事件流时空信息高效挖掘”、“多模态异构视觉流融合”、“多模态视觉流多任务学习”三个关键性问题,提出并实现了多项神经形态视觉单目深度估计方面的创新模型和方法。本文主要研究成果包括:第一,针对事件流时空信息挖掘的问题,提出了一种基于循环Transformer的事件流单目深度估计方法,实现了具有高效挖掘事件流全局空间信息和丰富时序信息的全新单目深度估计框架。本方法首先设计了一个基于Transformer的编码器-解码器,利用多尺度特征从事件中建模全局空间信息。然后,本方法提出了一个门控循环视觉Transformer,将递归机制引入Transformer,以利用来自事件的丰富时间线索。最后,本方法提出了一种跨注意引导的跳过连接,执行交叉注意融合多尺度特征,以提高全局空间建模能力。实验结果表明,本方法在合成和真实数据集上都比最先进的方法表现优异。第二,针对图像和事件多路异构视觉流融合的问题,提出了一种跨帧率协作的异构视觉流单目深度估计方法,实现了两路互补性视觉流异构协作的高帧率单目深度估计框架。本方法首先提出了一个跨帧率多模态融合模块,用于图像帧和事件的联合表征。通过采用隐式空间对齐和基于动态注意力的融合,它解决了不同时刻的图像帧和事件之间的错位问题,并在各种具有挑战性的场景中稳健地结合了两种模态的优势。在跨帧率多模态融合之后,本方法提出了一个采用循环结构的时间一致性建模模块,以保持融合后联合表征的时间一致性。实验结果表明,本方法在两个公开数据集上的深度估计精度优于现有的最先进方法,同时实现了高达100Hz的高帧率单目深度估计。第三,针对从图像和事件多路视觉流中进行多任务联合学习的问题,提出了一种基于多模态的联合单目深度及物体表面积估计方法,实现了从两路视觉流里精确估计场景内深度信息和物体表面积的联合估计框架。具体来说,本方法首先构建了一个数据采集平台,获得事件、RGB图像以及深度图,并提出了一个联合深度与物体表面积估计数据集。然后,本方法提出了一个单目深度和物体表面积联合估计框架,由深度估计和物体表面积估计分支组成。物体表面积估计分支通过设计专门的分类以及面积箱模块以实现精确估计物体的表面积。同时,深度估计分支除了预测场景深度信息外还负责辅助提升物体表面积估计的鲁棒性。实验结果表明,本方法能够准确地估计出不同类别和大小物体的可视表面积,在物体表面积估计数据集上的性能是最优的,同时也验证了深度估计分支的有效性。综上所述,本文提出了多项创新性的神经形态视觉单目深度估计方法,不仅进一步提升单目深度估计性能,实现鲁棒和高帧率的单目深度估计,还能够将其应用于辅助精确地估计物体表面积。本文研究工作为神经形态视觉单目深度估计领域的后续深入研究奠定了基础。更多还原

【Abstract】 Monocular depth estimation involves inferring the 3D depth information of a scene from a single camera sensor.Unlike stereo vision depth estimation methods,monocular depth estimation is ease of deployment,simpler and more economical data acquisition.It is widely used in fields such as autonomous driving,3D reconstruction,and robot naviga-tion.Due to the low sampling frame rate and low dynamic range of conventional frame-based cameras,there will be motion blur in high-speed scenes or texture loss in low-light scenarios,which significantly affects the accuracy of monocular depth estimation.Neu-romorphic vision sensors offer advantages like high temporal resolution,high dynamic range,and low data redundancy,providing a feasible solution for monocular depth es-timation in extreme scenarios.Current mainstream approaches employ a”reconstruct-then-estimate”cascade framework,converting asynchronous event streams into standard frames for depth estimation.However,this paradigm suffers from four critical limitations:1)dynamic features are attenuated by low-pass filtering during interpolation-based recon-struction;2)motion blur and noise introduced by intermediate frame generation modules lead to error accumulation;3)reconstructed image frame rates restrict high-speed target geometry capture capability;and 4)two-stage processing doubles inference latency.Con-sequently,constructing end-to-end depth estimation models directly from event streams while synergistically fusing heterogeneous visual features from both frame and event modalities has emerged as a pivotal challenge to overcome existing technical bottlenecks.Research on monocular depth estimation with neuromorphic vision is of great challenge and application value.Therefore,this dissertation focuses on the research of monocular depth estimation using neuromorphic cameras,and aims at addressing three key issues including“efficient leveraging spatio-temporal cues from event streams”,“develop-ing complementary fusion models for multi-modal asynchronous visual streams”,and“multi-task learning with asynchronous visual streams”.This dissertation explores and implements several innovative models and methods for monocular depth estimation using neuromorphic cameras.The main contributions of this dissertation are summarized as follows:Firstly,this dissertation proposes a monocular depth estimation method via recurrent Transformers,which realizes a novel monocular depth estimation framework with effi-cient leveraging of global spatial information and rich temporal cues from event streams.This method first designs a transformer-based encoder-decoder that utilizes multi-scale features to model global spatial information from events.Then,this method proposes a Gate Recurrent Vision Transformer(GRVi T),introducing a recursive mechanism into transformers,to leverage rich temporal cues from events.Finally,this method presents a Cross Attention-guided Skip Connection(CASC),performing cross attention to fuse multi-scale features,to improve global spatial modeling capabilities.The experimental results show that this method outperforms state-of-the-art methods by a margin on both synthetic and real-world datasets.Secondly,this dissertation proposes a monocular depth estimation method via cross frame-rate collaboration of heterogeneous visual streams,which realizes a high-rate monocular depth estimation framework to make cross frame-rate collaboration of comple-mentary DVS event streams and frames.This method first introduces a cross frame-rate multimodal fusion(CFMF)module for the joint representation of frames and events.By employing implicit spatial alignment and dynamic attention-based fusion,it addresses the misalignment between frames and events at different moments,robustly combining the strengths of both modalities in diverse challenging scenarios.Following the CFMF,a temporal consistent modeling(TCM)module adopting the recurrent structure is created to keep the temporal consistency of joint representations from CFMF.Experimental re-sults demonstrate that the depth estimation accuracy of this method outperforms existing five state-of-the-art methods and three baselines involving single modality on two public datasets(i.e.,DSEC and MVSEC)while achieving a high frame rate up to 100 Hz.Thirdly,this dissertation proposes a multi-modal joint monocular depth and object surface area estimation method,which develops a joint estimation framework for accu-rately estimating depth information and object surface area in the scene from two visual streams.Specifically,this method first builds a data acquisition platform to collect event streams,RGB images,and depth maps,and proposes an joint depth and object surface area estimation dataset.Then,This method introduces a joint estimation framework for monocular depth and object surface area,consisting of depth estimation and object sur-face area estimation branches.The object surface area estimation branch is designed with dedicated classification and area-bins modules to achieve accurate object surface area es-timation.Meanwhile,in addition to predicting scene depth information,the depth esti-mation branch is also used to assist in improving the robustness of the results of object surface area estimation.Experimental results demonstrate that this method can accurately estimate the objects visible surface area of different categories and sizes,achieving the best performance on the object surface area estimation dataset.Meanwhile,the results also validating the effectiveness of the depth estimation branch.In conclusion,this dissertation proposes several innovative methods for monocular depth estimation using neuromorphic cameras,which not only further improve the perfor-mance of monocular depth estimation,achieving robust and high-rate depth estimation,but also enable its application in assisting the accurate object surface area estimation.Moreover,this dissertation builds a solid technical foundation for the follow-up research work in the field of neuromorphic vision monocular depth estimation.更多还原