【摘要】 针对弱光环境下红外与可见光图像融合存在的纹理细节丢失、视觉效果和实时性差等问题，提出了一种基于跨模态特征交互和多尺度重建(Cross-modal Feature Interaction and Multi-scale Reconstruction,CFIMR)的红外与可见光图像融合算法CFIMRFusion。该算法构建了包括卷积注意力增强模块、编码器网络、跨模态特征交互融合模块和基于多尺度重建的解码器网络的四阶段融合框架。首先，设计卷积注意力增强模块提升弱可见光图像的对比度和纹理可见性，并利用编码器网络从红外图像和增强后的可见光图像中提取深层多尺度特征。然后，提出基于通道-空间注意力的跨模态特征交互融合模块，对红外显著特征和可见光细节特征进行互补融合。最后，为解决使用普通解码器重建图像时出现特征消失等问题，将融合得到的多尺度特征以跳跃连接的方式输入到解码器各级，重建高保真的融合图像。实验结果表明，CFIMRFusion融合图像的细节特征和整体视觉效果优于对比算法；且与最优对比算法相比，融合图像在TNO数据集中平均梯度、边缘强度分别提升了15.8%、18.2%，在LLVIP数据集中互信息、标准差分别提升了11.5%、9.5%，在MSRS数据集中边缘强度提升了10.1%；三个数据集上的融合速度分别为最快对比算法的24.1%、23.86%和25.2%。更多还原

【Abstract】 Objective The goal of infrared and visible light image fusion is to amalgamate image data from disparate sensors into a unified representation that preserves the complementary information and salient features inherent to both modalities. The autoencoder framework offers significant advantages in imagine fusion, whose encoder can efficiently extract image features, and decoder can precisely reconstruct the image. Moreover, by introducing mechanisms such as attention, the issues of detail loss and information retention can be effectively addressed.However, existing fusion methods still have many shortcomings: 1) Conventional autoencoder frameworks struggle to effectively learn deep features, leading to gradient vanishing issues during the image reconstruction process; 2) In low-light conditions, existing algorithms are frequently impeded by the suboptimal quality of visible light images, which significantly degrades the overall fusion quality; 3) Existing deep learning-based algorithms are often computationally intensive, thus being unsuitable for real-time applications. Therefore, a novel infrared and visible light image fusion algorithm based on Cross-modal Feature Interaction and Multi-scale R econstruction, CFIMRFusion, has been proposed.Methods The proposed algorithm comprises four key components: a convolutional attention enhancement module, an encoder network, a cross-modal feature interaction fusion module, and a decoder network based on multi-scale reconstruction(Fig.1). The convolutional attention enhancement module extracts features through convolutional operations, and enhances the contrast and texture visibility of degraded visible light images, thereby enhancing the detailed features of the images. The encoder module employs convolutions to extract deep features from both infrared images and the enhanced visible light images. To fully leverage the multi-modal features of infrared and visible light images, a cross-modal feature interaction fusion module has been developed, which performs complementary fusion on infrared and visible light image via a channel-spatial attention mechanism.Additionally, considering that the encoder structure with direct connections is prone to feature vanishing during training, a decoder network based on multi-scale reconstruction has been developed, where fused features e xtracted by encoders at different levels are skip-connected to the decoder network.Results and Discussions In the objective analysis of the proposed algorithm compared with GANMcC,SwinFuse, U2Fusion, LapH, MUFusion, CMRFusion, and TUFusion, the proposed algorithm obtained six optimal values and one suboptimal value on the TNO dataset(Tab.2) and LLVIP dataset(Tab.3), three optimal values and four sub-optimal values are obtained on the MSRS data set(Tab.4). The subjective evaluations on TNO, LLVIP, and MSRS datasets are shown in Fig.3-Fig.5 respectively. The fused images exhibit superior detail features and overall visual effects compared with the benchmark algorithms. Moreover, the fusion speed is 24.1%,23.86% and 25.2% of the fastest comparison algorithm respectively.Conclusions A novel infrared and visible light image fusion algorithm based on cross-modal feature interaction and multi-scale reconstruction, CFIMRFusion, has been developed, to enhance information acquisition in lowlight conditions and produce fused images with clear details and improved visual effects. Initially, the degraded visible light image is fed into a convolutional attention enhancement module to obtain enhanced features.Subsequently, the enhanced visible light image and the infrared image are fed an autoencoder network to extract multi-scale features, which are then complementarily fused via a cross-modal feature interaction fusion module.Finally, a decoder network based on multi-scale reconstruction is employed to generate the fused image with rich details and clear structures. Experimental results indicate that, compared to the optimal comparison algorithm, the fused images of CFIMRFusion achieved 15.8% and 18.2% increase in average gradient and edge intensity,respectively, on the TNO dataset; 11.5% and 9.5% increase in mutual information and standard deviation on the LLVIP dataset; and 10.1% increase in edge intensity in the MSRS dataset. Moreover, ablation studies further demonstrate the effectiveness of each module in the algorithm, where the complete model outperforms partial module combinations in terms of AG, EI, SD, SF, and VIF metrics. Regarding computational efficiency, the proposed CFIMRFusion algorithm achieves the shortest average runtime and is thus capable of satisfying the realtime requirements of applications in low-light scenarios with high temporal constraints.更多还原