mlsanityData to 7 Oct 2026

APoLLo: Unified Adapter and Prompt Learning for Vision Language Models

arXiv 2312.01564 ↗First posted 4 Dec 2023cs.LG cs.AI cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 69 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

50 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  2. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  3. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct

Other references · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
34 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  12. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  13. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  14. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  15. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  16. 2017Learning multiple visual domains with residual adapterscited by 553
  17. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  18. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  19. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  20. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  21. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  22. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  23. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  24. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  25. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  26. 2017Incremental Learning Through Deep Adaptationcited by 108
  27. 2023Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusioncited by 81
  28. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  29. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performancecited by 51
  30. 2020CO2: Consistent Contrast for Unsupervised Visual Representation Learningcited by 38
  31. 2022Position-guided Text Prompt for Vision-Language Pre-trainingcited by 27
  32. 2023Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptationcited by 18
  33. 2023TTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Modelscited by 14
  34. 2023GDA: Generative Data Augmentation Techniques for Relation Extraction Taskscited by 10

Built on by

Cited by year

Citing papers per year: 2024: 10; 2025: 9.

19 papers.

Sort:

By area:

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  3. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  4. 2024CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learningcited by 17
  5. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  6. 2024Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspectivecited by 9
  7. 2024FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performancecited by 7
  8. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  9. 2024SafaRi:Adaptive Sequence Transformer for Weakly Supervised Referring Expression Segmentationcited by 6
  10. 2025Visual Textualization for Image Prompted Object Detectioncited by 6
  11. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  12. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
7 more
  1. 2025Agentic Design Review Systemcited by 2
  2. 2024SDPT: Synchronous Dual Prompt Tuning for Fusion-based Visual-Language Pre-trained Modelscited by 1
  3. 2024Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensemblingcited by 1
  4. 2024Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuningcited by 1
  5. 2025Can Text-to-Video Generation help Video-Language Alignment?cited by 1
  6. 2025Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentationcited by 1
  7. 2024How to Understand Named Entities: Using Common Sense for News Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LaViP:Language-Grounded Visual Prompts
  2. 2023Prompt Tuning based Adapter for Vision-Language Model Adaption
  3. 2023PRE: Vision-Language Prompt Learning with Reparameterization Encoder
  4. 2024Revisiting Prompt Pretraining of Vision-Language Models
  5. 2023Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models
  6. 2025Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
  7. 2024Multi-Modal Adapter for Vision-Language Models
  8. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models