【摘要】 为了解决视觉特征和文本特征融合后存在部分语义缺失从而导致视觉信息对文本信息的补充有较大偏差的问题，提出了一种基于视觉与文本语义增强的多模态命名实体识别方法。融合BERT文本特征提取和CLIP（contrastive language–image pre-training）视觉特征提取方法，设计了基于协同交叉注意力机制的特征交互单元，以增强视觉信息和文本信息之间的语义关系。CLIP通过对比学习框架进行预训练，优化模型以正确匹配视觉和对应的文本描述，最大化正样本（匹配的视觉-文本对）的相似性，同时最小化负样本（不匹配的视觉-文本对）的相似性。采用通用领域数据集TWITTER-2015和TWITTER-2017作为实验数据集。实验结果表明，本模型相比传统方法在多模态命名实体识别任务中的准确率、召回率、F₁值均有显著提升。更多还原

【Abstract】 In view of a solution of the partial semantic loss in the fusion of visual and textual features,which leads to a significant deviation in the supplementation of visual information to textual information,a multimodal named entity recognition method has thus been proposed based on visual and textual semantic enhancement.A feature interaction unit based on collaborative cross attention mechanism is designed for an enhancement of the semantic relationship between visual information and textual information by integrating BERT text feature extraction and CLIP （contrastive language image pre-training） visual feature extraction methods.CLIP pre-trains through a contrastive learning framework to optimize the model for a correct matching of visual and corresponding text descriptions,thus maximizing the similarity of positive samples （matched visual text pairs） while minimizing the similarity of negative samples （mismatched visual text pairs）.The general domain datasets TWITTER-2015 and TWITTER-2017 are adopted as experimental datasets in this article.Experimental results show that compared with traditional methods,this model is characterized with a significantly improved accuracy,recall,and F₁ score in multi-modal named entity recognition tasks.更多还原