mlsanityData to 7 Oct 2026

VLLaVO: Mitigating Visual Gap through LLMs

arXiv 2401.03253 ↗First posted 6 Jan 2024cs.CV cs.CL cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain shifts. To tackle this issue, cross-domain learning aims at extracting domain-invariant knowledge to reduce the domain shift between training and testing data. However, in visual cross-domain learning, traditional methods concentrate solely on the image modality, disregarding the potential benefits of incorporating the text modality. In this work, we propose VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners. VLLaVO uses vision-language models to convert images into detailed textual descriptions. A large language model is then finetuned on textual descriptions of the source/target domain generated by a designed instruction template. Extensive experimental results under domain generalization and unsupervised domain adaptation settings demonstrate the effectiveness of the proposed method.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 108 references that are in the map. First posted on arXiv 6 Jan 2024.

Builds on

83 of 108 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Learning Concise and Descriptive Attributes for Visual Recognition7 Aug
  2. 2023Exploring Vision-Language Models for Imbalanced Learning4 Apr
  3. 2022Visual Classification via Description from Large Language Models13 Oct
  4. 2022Prompt Distribution Learning6 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  13. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  14. 2013Describing Textures in the Wildcited by 2,110
  15. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  16. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  17. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  18. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  19. 2018Hierarchical Neural Story Generationcited by 1,182
  20. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  21. 2020In Search of Lost Domain Generalizationcited by 1,042
  22. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  23. 2020Designing Network Design Spacescited by 954
  24. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  25. 2021Robust fine-tuning of zero-shot modelscited by 785
  26. 2021Domain Generalization: A Surveycited by 760
  27. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  28. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  29. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  30. 2021Domain Generalization with MixStylecited by 563
  31. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  32. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  33. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  34. 2017VisDA: The Visual Domain Adaptation Challengecited by 432
  35. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  36. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  37. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  38. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025ViLAaD: Enhancing "Attracting and Dispersing'' Source-Free Domain Adaptation with Vision-and-Language Modelcited by 2
  2. 2025TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unsupervised Domain Adaption Harnessing Vision-Language Pre-training
  2. 2024Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation
  3. 2026Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  5. 2025Latent Domain Prompt Learning for Vision-Language Models
  6. 2025FLARE: Fully Integration of Vision-Language Representations for Deep Cross-Modal Understanding
  7. 2025Unified modality separation: A vision-language framework for unsupervised domain adaptation
  8. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Models