mlsanityData to 14 Oct 2026

Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation

arXiv 2403.06946 ↗First posted 11 Mar 2024cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models (VLMs) like CLIP have demonstrated good zero-shot learning performance in the unsupervised domain adaptation task. Yet, most transfer approaches for VLMs focus on either the language or visual branches, overlooking the nuanced interplay between both modalities. In this work, we introduce a Unified Modality Separation (UniMoS) framework for unsupervised domain adaptation. Leveraging insights from modality gap studies, we craft a nimble modality separation network that distinctly disentangles CLIP's features into language-associated and vision-associated components. Our proposed Modality-Ensemble Training (MET) method fosters the exchange of modality-agnostic information while maintaining modality-specific nuances. We align features across domains using a modality discriminator. Comprehensive evaluations on three benchmarks reveal our approach sets a new state-of-the-art with minimal computational costs. Code: https://github.com/TL-UESTC/UniMoS

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 63 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

54 of 63 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
38 more
  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  3. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  4. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  5. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  9. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 683
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  11. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  13. 2017VisDA: The Visual Domain Adaptation Challengecited by 435
  14. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  15. 2019Contrastive Adaptation Network for Unsupervised Domain Adaptationcited by 407
  16. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 341
  17. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 335
  18. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  19. 2016Domain Separation Networkscited by 320
  20. 2023Teaching CLIP to Count to Tencited by 198
  21. 2021Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptationcited by 177
  22. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  23. 2022Domain Adaptation via Prompt Learningcited by 128
  24. 2020FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptationcited by 124
  25. 2021Cycle Self-Training for Domain Adaptationcited by 120
  26. 2022Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptationcited by 98
  27. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  28. 2022A Closer Look at Smoothness in Domain Adversarial Trainingcited by 85
  29. 2021TVT: Transferable Vision Transformer for Unsupervised Domain Adaptationcited by 81
  30. 2020Maximum Density Divergence for Domain Adaptationcited by 76
  31. 2020Domain Adaptation with Auxiliary Target Domain-Oriented Classifiercited by 76
  32. 2021Unsupervised Multi-source Domain Adaptation Without Access to Source Datacited by 76
  33. 2023AD-CLIP: Adapting Domains in Prompt Space Using CLIPcited by 62
  34. 2022Safe Self-Refinement for Transformer-based Domain Adaptationcited by 61
  35. 2023Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspectivecited by 54
  36. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  37. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 46
  38. 2021Pareto Domain Adaptationcited by 20

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  2. 2025Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognitioncited by 17
  3. 2025CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Surveycited by 15
  4. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  5. 2024CLIP the Divergence: Language-guided Unsupervised Domain Adaptationcited by 5
  6. 2025Unified modality separation: A vision-language framework for unsupervised domain adaptationcited by 4
  7. 2025ViLAaD: Enhancing "Attracting and Dispersing'' Source-Free Domain Adaptation with Vision-and-Language Modelcited by 2
  8. 2025Mitigating Negative Transfer via Reducing Environmental Disagreementcited by 1
  9. 2026CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptationcited by 1
  10. 2024Semantically Guided Action Anticipationcited by 0
  11. 2025Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Modelscited by 0
  12. 2026Learning ORDER-Aware Multimodal Representations for Composite Materials Designcited by 0
1 more
  1. 2026Towards Identifiable Representations under Misspecified Structurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
  2. 2024UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
  3. 2026One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models
  4. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models
  5. 2024Unsupervised Domain Adaption Harnessing Vision-Language Pre-training
  6. 2026LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
  7. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
  8. 2026Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration