【摘要】 目的 域适应技术旨在利用有标签的源域信息提升无标签目标域上的任务性能。近期，对比语言—图像预训练模型CLIP(contrastive language-image pre-training)展现出了强大的泛化能力，一些研究将其引入到域适应中，以提升模型在目标域上的泛化能力。然而，目前基于CLIP的域适应方法通常只调整文本模态的特征，保持视觉模态的特征不变，从而导致目标域的性能提升受限。为此，提出了双模态域无关提示引导的图像分类域适应方法DDAPs(dualmodality domain-agnostic prompts)。方法 DDAPs引入了双模态提示学习，即通过文本和视觉提示学习微调文本特征和图像特征，协同处理域差异的问题。一方面，DDAPs致力于学习更具判别性的文本和图像特征，使模型在当前下游分类任务上的性能更好；另一方面，DDAPs通过消除源域和目标域之间的域差异，学习域不变的文本和图像特征，以提升模型在目标域上的性能。以上两个目标可通过添加域无关文本提示模块和域无关视觉提示模块，使用分类损失和对齐损失微调CLIP来实现。对于分类损失，DDAPs利用源域的标签和目标域的伪标签对样本进行分类；而对于对齐损失，DDAPs则通过最大均值差异损失（maximum mean discrepancy,MMD）来对齐源域和目标域的图像特征分布，从而消除图像特征的域差异。结果 本方法既适用于单源域适应，也适用于多源域适应。对于单源域适应，本方法在Office-Home、VisDa-2017及Office-31这3个数据集上进行了实验，分别取得了87.1%、89.6%和91.6%的平均分类准确率，达到了当前最好的性能；对于多源域适应，本方法在Office-Home上进行了实验，取得了88.6%的平均分类准确率。同时，在Office-Home上进行了消融实验，验证了域无关文本提示模块和域无关视觉提示模块的有效性。结论 DDAPs通过域无关的文本和视觉提示模块微调CLIP预训练模型，使模型学习源域与目标域之间域不变且判别性的特征，有效提升了模型在目标域上的性能表现。更多还原

【Abstract】 Objective Domain adaptation aims to utilize information from a labeled source domain to assist tasks in the unlabeled target domain. Recently, contrastive language-image pre-training(CLIP) has demonstrated impressive generalization capabilities in classification downstream tasks. Some methods have incorporated CLIP into domain adaptation, enhancing the model’s generalization ability in the target domain. However, current domain adaptation methods based on CLIP typically adjust only the features of the textual modality, leaving the visual modality features unchanged. These existing methods overlook the importance of enhancing the discriminative capability of image features during classification and neglect the synergistic role of the visual modality in eliminating domain discrepancy. This issue is addressed by introducing a domain adaptation method for the image classification task guided by dual-modality domain-agnostic prompts(DDAPs).Method DDAPs introduces dual-modality prompt learning, simultaneously fine-tunes textual and visual features, and collaboratively addresses domain discrepancies. The key modules of DDAPs are the domain-agnostic textual prompt module and the domain-agnostic visual prompt module. The former employs textual prompt learning techniques to fine-tune the text encoder, fostering domain-agnostic and discriminative text features across domains. DDAPs adopts task-level text prompt learning, sharing the textual prompt module across various domains and categories. Similarly, the domain-agnostic visual prompt module uses visual prompt learning techniques to enhance the image encoder, cultivating domain-agnostic and discriminative image features. Task-level visual prompt learning is employed, ensuring that the visual prompt module is shared across diverse domains and samples. The added DDAPs were learned via classification loss and alignment loss to fine-tune the model. On the one hand, as the original pre-training task for CLIP involves matching paired images and text, it needs to learn more discriminative text and image features specific to the current downstream classification task. Therefore, DDAPs uses classification loss to train the added dual-modality domain-invariant prompt modules, enhancing the discriminative power of the features. For the source domain, the classification loss can directly use the existing labels, whereas for the target domain, the classification loss can use the collected pseudo labels. On the other hand, given the considerable visual differences between the images of the two domains, the extracted image representations contain domainspecific features. The target domain is encouraged to fully utilize the beneficial information from the source domain. Therefore, the maximum mean discrepancy loss of DDAPs is used to align the image feature distributions of the source and target domains, and domain-invariant and image features are learned. In image feature distribution alignment, the fusion results of image features and classification probabilities are aligned to enhance the discriminative capability of the aligned features and reduce incorrect category matching between the source and target domains.Result The experiments encompass three datasets: Office-Home, VisDa-2017, and Office-31. During training, all weights of the CLIP pre-trained model remain fixed, with only the weights of the newly added domain-invariant textual and visual prompt learning modules being updated. Experiments on single-source domain adaptation were conducted across these three datasets to assess DDAPs against existing methods. Average classification accuracies of 87. 1%, 89. 6%, and 91. 6% were obtained, indicating current state-of-the-art performance. Additionally, the versatility of DDAPs extended to multi-source domain adaptation, where it achieved an average classification accuracy of 88. 6% on the Office-Home dataset. Ablation studies on the OfficeHome dataset further confirm the importance of the domain-agnostic text prompt module and the domain-agnostic visual prompt module. Notably, the comprehensive version of DDAPs exceled, surpassing the performance of individually added single-modality prompt modules, and showed a 5% improvement over the CLIP pre-trained model. This situation underscores the effectiveness of employing dual-modality domain-agnostic prompts to collectively mitigate domain discrepancy.Moreover, experiments explored the sensitivity of hyperparameters. In the proposed DDAPs method, the primary hyperparameters include the weight of the alignment loss and the lengths of the prompt vectors. The findings reveal that when the weight of the alignment loss approaches its optimal value, the performance of the target domain remains stable. Similarly, variations in the lengths of prompt vectors do not significantly affect DDAPs performance. For a more intuitive grasp of DDAP, this study also employs t-distributed stochastic neighbor embedding to visualize the image features of different models, and the visualization demonstrates the superiority of DDAPs in addressing domain adaptation problems.Conclusion This study introduces a domain adaptation method called DDAPs for image classification tasks. DDAPs uses domaininvariant textual and visual prompts to eliminate domain discrepancies between the source and target domains collaboratively and learns domain-invariant and discriminative images and text features to enhance model performance in the target domain. DDAPs can be applied to both single-source domain adaptation and multi-source domain adaptation. The proposed DDAPs has been experimentally validated across multiple datasets, achieving state-of-the-art results and demonstrating the significance of collaborative handling of domain discrepancy from a dual-modality perspective.更多还原