【摘要】 现有的多模态推荐方法通常分别提取图像、文本等模态特征并在训练阶段进行浅层融合,难以充分挖掘跨模态语义.此外,主流方法多采用随机初始化的用户表示,容易导致用户表示区分度不足.为此,文中提出强化用户语义表示的多模态推荐方法(Multimodal Recommendation with User Semantic Embedding Refinement, USERec),分别从物品角度和用户角度缓解现有问题.在物品侧,利用多模态大语言模型,以物品的文本模态指导视觉模态特征提取,实现深度语义融合,获得更适用于推荐任务的物品表示.在用户侧,为用户表示引入位置编码,增强用户索引空间的频谱多样性,并结合度敏感剪枝构建个性化局部图,再通过随机采样注意力机制补充用户的全局感知,增强用户表示的区分度.在4个真实数据集上的实验表明USERec的有效性.更多还原

【Abstract】 Existing multimodal recommendation methods typically extract features of the different modalities separately, such as images and texts, and only shallow fusion is performed during training.Therefore, it is difficult to fully explore cross-modal semantics. Moreover, mainstream methods mostly adopt randomly initialized user representations, resulting in insufficient discriminability among users. To address these issues, a multimodal recommendation method with user semantic embedding refinement(USERec) is proposed in this paper. The problems are alleviated from the perspectives of both the item and the user. On the item side, a multimodal large language model is utilized to achieve deep semantic fusion by guiding visual feature extraction with textual information. Thus, more suitable item representations for recommendation tasks can be obtained. On the user side, positional encoding is introduced into user representations to enhance the spectral diversity of the user index space. Personalized local graphs are then constructed through degree-sensitive pruning, and the global awareness of users is augmented via a randomly sampled attention mechanism, thereby improving the discriminability of user representations. Experiments on four real-world datasets verify the effectiveness of USERec.更多还原