【摘要】 目的 基于深度学习的脑电图-功能性近红外光谱技术(electroencephalogram-functional near-infrared spectroscopy, EEG-fNIRS)多模态脑机接口在康复工程中具有广泛的应用前景，但存在数据量不足的问题。为此，本文提出一种基于改进条件生成式对抗网络(conditional generative adversarial network, CGAN)的EEG-fNIRS多模态信号数据增广方法，以解决EEG-fNIRS多模态脑机接口与深度学习结合时面临的数据量匮乏的问题。方法 首先，对EEG和fNIRS数据进行滤波、归一化和下采样等预处理。然后，针对EEG的非平稳特点，在CGAN生成器和判别器中增加自注意力机制，获得EEG数据增广模型CGAN_E,加强捕捉和学习时变关键信息的能力。同时，针对fNIRS采样率低、信息量不充分问题，在CGAN生成器和判别器中增加上采样卷积层，获得fNIRS数据增广模型CGAN_f,加强模型的信息挖掘能力，并将CGAN_E和CGAN_f的条件信息设置为类标签；进而，利用CGAN_E和CGAN_f对每导EEG和fNIRS分别进行增广，并将多导EEG扩增数据和多导fNIRS[包括氧合血红蛋白浓度(oxyhemoglobin concentration, HbO)和脱氧血红蛋白浓度(deoxyhemoglobin concentration, HbR)两种]扩增数据串接融合，获得EEG-fNIRS多模态增广数据。最后，对公开的EEG-fNIRS多模态信号数据集TU-Berlin-A前6名受试者数据进行增广实验，并设计一维卷积神经网络分类器评估增广数据的质量。结果 基于EEG-fNIRS多模态信号公开数据集TU-Berlin-A前6名受试者的左右手运动想象数据进行实验研究表明，当数据扩增5倍时，本文方法取得94.81%的平均分类准确率。结论 CGAN_E和CGAN_f能够生成接近真实数据分布的EEG和fNIRS信号，验证了对CGAN改进和本文所提EEG-fNIRS多模态数据增广方法的有效性。更多还原

【Abstract】 Objective Electroencephalogram-functional near-infrared spectroscopy（EEG-fNIRS） multimodal brain-computer interface based deep learning has a wide application prospect in rehabilitation engineering, but it faces the problem of insufficient data. In order to solve the problem of small amount of data when EEG-fNIRS multi-module brain-computer interface is combined with deep learning, this paper proposes an EEG-fNIRS multi-module signal data augmentation method based on modified conditional generative adventive network（CGAN）. Methods Firstly, EEG and fNIRS data were preprocessed by filtering, normalization and downsampling. Then, according to the non-stationary characteristics of EEG,self-attention mechanism was added in CGAN generator and discriminator to obtain the EEG data augmented model CGAN_E,which strengthened the ability to capture and learn time-varying critical information. At the same time, an up-sampling convolution layer was added to the CGAN generator and discriminator to obtain the fNIRS data augmented model CGAN_f to compensate for the low sampling rate and insufficient information of fNIRS. The conditional information of CGAN_E and CGAN_f was set as class label. Furthermore, CGAN_E and CGAN_f were used to amplify each channel EEG and fNIRS,respectively. Multi-channel EEG augmentaded data and multi-channel fNIRS [including oxygenated hemoglobin concentration（HbO） and deoxygenated hemoglobin concentration（HbR）] augmentaded data were sequentially fused to obtain EEG-fNIRS multimodal augmented data. Finally, data augmentation experiments were performed on the first six subjects of the public EEG-fNIRS multi-modal signal dataset TU-Berlin-A,and a one-dimensional convolutional neural network classifier was designed to evaluate the quality of augmented data. Results Experimental studies based on left and right hand motion imagery data from the first six subjects in the EEG-FNIRS multimodal signal open dataset TU-Berlin-A showed that when the data were enlarged by a factor of 5,the average classification accuracy of our method was 94.81%. Conclusions CGAN_E and CGAN_f can generate EEG and fNIRS signals close to the real data distribution, which verifies the effectiveness of the improved CGAN and the EEG-fNIRS multimodal data augmentation method proposed in this paper.更多还原