【摘要】 基于深度学习模型的多模态学习方法已在静态、可控等简单场景下取得较优的语义理解性能，但在动态、开放等复杂场景下的泛化性仍然较低.近期已有不少研究工作尝试将类人知识引入多模态语义理解方法中，并取得不错效果.为了更深入了解当前知识驱动的多模态语义理解研究进展，文中在对相关方法进行系统调研与分析的基础上，归纳总结关系型和对齐型这两类主要的多模态知识表示框架.然后选择多个代表性应用进行具体介绍，包括图文匹配、目标检测、语义分割、视觉-语言导航等.此外，文中总结当前相关方法的优缺点并展望未来可能的发展趋势.更多还原

【Abstract】 Multimodal learning methods based on deep learning model achieve excellent semantic understanding performance in static, controllable and simple scenarios. However, their generalization ability in dynamic, open and other complex scenarios is still unsatisfactory. Human-like knowledge is introduced into multimodal semantic understanding methods in recent research, yielding impressive results. To gain deeper understanding of the current research progress in knowledge-driven multimodal semantic understanding, two main types of multimodal knowledge representation frameworks are summarized based on systematic investigation and analysis of relevant methods in this paper. The two main types of multimodal knowledge representation frameworks are relational and aligned, respectively. Several representative applications are discussed, including image-text matching, object detection, semantic segmentation, and vision-and-language navigation. In addition, the advantages and disadvan-tages of the current methods and the possible development trend in the future are concluded.更多还原