【作者】 姜璐；

【导师】 汪丰； 李超宏；

【作者基本信息】 东南大学 ， 生物医学工程， 2024， 硕士

【摘要】 眼底荧光血管造影(fundus fluorescein angiography,FFA)是临床上诊断视网膜疾病的重要检查方法,但由于其侵入式的检查方式和造影剂的体内循环导致该检查存在使用禁忌和过敏反应风险,使得部分患者无法顺利进行检查,且这种耗时耗力的方式不适用于大规模的眼底疾病早期筛查。得益于深度学习技术在计算机视觉领域的快速发展,使用图像生成算法从眼底结构图像生成FFA图像成为一个安全、高效的替代方案。本文针对FFA图像生成任务,建立由近红外(near-infrared red,NIR)图像、无赤光(red-free,RF)图像和FFA图像组成的多模态数据集,结合图像生成方法和多模态图像融合方法,提出了一对一和多对一的端到端FFA图像生成模型,实现了在国产共焦扫描激光眼底诊断设备上的本地化运行。本文的主要工作如下:(1)针对一对一生成FFA图像任务中难以准确生成血管细节和病理表现的问题,改进数据集预处理方法和生成模型结构。在图像预处理方面,提出了一种结合特征匹配和稠密光流法由粗到精实现多模态图像配准的方法。并提出一种图像增强方法来提高NIR图像的质量,从而进一步提高FFA图像生成效果。在生成模型方面,提出了一种基于生成对抗网络(generative adversarial networks,GAN)的端到端FFA图像生成模型(NIR2FFA-GAN),在模型中引入了自注意力机制提高全局信息利用能力,增强对血管等重要结构的关注,使用多尺度判别器提高图像的生成质量,同时增加显著结构损失来提高生成性能。通过对比评估,本文提出的NIR2FFA-GAN较其他先进的方法在生成眼底血管、荧光渗漏等细节上表现更好。(2)针对一对一图像生成方法获取的图像特征信息有限的问题,使用两个模态的眼底结构图像(NIR-RF)生成FFA图像。提出了一种适用于NIR-RF-FFA数据集的图像配准方法,使用基于互信息的稀疏光流法实现NIR和RF图像的精确配准,再根据(1)中的配准方法对NIR和FFA图像做配准,RF图像跟随NIR图像同步变换,实现了高精度的三模态图像配准。在生成模型方面,提出一种引入多模态图像融合模块的多对一图像生成对抗网络(Fusion2FFA-GAN),该模块基于Lite Transformer和可逆神经网络分别对全局特征和局部特征进行提取和融合,提高了细节信息的保留能力。通过定性定量评估证明了Fusion2FFA-GAN在生成的FFA图像的准确度上较NIR2FFA-GAN有所提升。(3)为了让算法在临床上发挥出更大的价值,将提出的FFA图像生成算法部署到眼底诊断设备上,实现本地化运行。在部署过程中,使用开放神经网络交换格式导出服务器端的模型,利用跨平台的计算机视觉库中的深度神经网络模块实现对模型的快速推理,并将该算法集成到终端的影像信息管理系统中。最后通过比较部署前后的模型性能证明模型部署基本不影响FFA图像生成质量,从而为算法的临床应用提供设备支持。更多还原

【Abstract】 Fluorescein fundus angiography(FFA)is an important clinical examination method for diagnosing retinal diseases.However,due to the invasive nature of examinations and the circulation of contrast agents in the body,there are risks of contraindications and allergic reactions associated with the FFA examination,which can prevent some patients from undergoing the examination smoothly.Moreover,it is not suitable for large-scale early screening of fundus diseases because of its time-consuming and labor-intensive nature.Thanks to the rapid development of deep learning technology in the field of computer vision,it has become a safe,efficient,and cost-effective alternative that using image generation algorithms to generate FFA images from fundus structural images.Therefore,a multi-modal dataset composed of near-infrared(NIR)images,red-free(RF)images,and FFA images is established for FFA image generation.Combining image generation methods and multi-modal image fusion methods,a one-to-one and many-to-one end-to-end FFA image generation model is proposed.In addition,localization operation is achieved on the terminal fundus medical equipment.The main contributions of work are summarized as follows:(1)To solve the problem that it is difficult to accurately generate vascular details and pathological manifestations in the task of one-to-one generation of FFA images,the image preprocessing methods and the generation model structure are improved.In terms of image preprocessing,a method combining feature matching and dense optical flow is proposed to achieve multi-modal image registration from coarse to fine.Additionally,an image enhancement technique is introduced to improve the quality of NIR images,thereby further enhancing the quality of the generated FFA images.Regarding the generation model,an end-to-end FFA image generation model based on generative adversarial networks(GAN),named NIR2FFA-GAN,is proposed.In this model,a self-attention mechanism is incorporated to enhance the utilization of global information and focus on important structures such as vessels.Multi-scale discriminators are employed to improve the quality of generated images,while significant structural loss is added to enhance generation performance.Comparative evaluations demonstrate that the proposed method outperforms other state-of-the-art methods in capturing details,such as retinal vasculature and fluorescein leakage.(2)In response to the limited feature information obtained by one-to-one image generation methods,a method is proposed to generate FFA images from two modalities of fundus structural images(NIR-RF).And an image registration method suitable for the NIR-RF-FFA dataset is introduced,which uses sparse optical flow based on mutual information to achieve precise registration of NIR and RF images.Subsequently,NIR and FFA images are registered according to the registration method in(1),and RF images are synchronously transformed with NIR images to achieve high-precision three-mode image registration.In terms of generation model,a many-to-one image generation adversarial network(Fusion2FFA-GAN)with multi-modal image fusion module is proposed.The module extracts and fuses global features and local features based on Lite Transformer and invertible neural network(INN)to improve the retention of details.And qualitative and quantitative evaluations demonstrate that Fusion2FFA-GAN improves the accuracy of generated FFA images compared to NIR2FFA-GAN.To make the algorithm play a greater value in clinical practice,the proposed FFA image generation algorithm is deployed on the terminal fundus diagnosis and treatment equipment to achieve localized operation.During the deployment process,the model is exported using the open neural network exchange(ONNX)format on the server side,and rapid inference of the model is achieved using the deep neural network(DNN)module in the open computer vision library(OpenCV).Then the algorithm is integrated into the terminal imaging information management system.Finally,by comparing the performance of the model before and after deployment,it is proved that model conversion and deployment has little effect on the quality of FFA image generation.更多还原