mlsanityData to 7 Oct 2026

Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering

arXiv 2506.06905 ↗First posted 7 Jun 2025cs.AI cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new visual question answering (VQA) tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, does not always improve monotonically when increasing the number of examples. We hypothesize that this happens because the LMM is overwhelmed by extraneous information in the image embeddings that is irrelevant to the downstream task. To address this, we propose a meta-learning approach that induces few-shot capabilities in LMMs through a fixed set of soft prompts distilled from task-relevant visual features, which are adapted at test time using a small number of examples. We facilitate this distillation through an attention-mapper module that can be easily integrated with any LMM architecture and is jointly learned with soft prompts. Evaluation on the VL-ICL Bench shows that our method successfully achieves task adaptation in low-data regimes with just a few gradient steps, outperforming ICL by 21.2%. Comparisons with parameter-efficient finetuning methods demonstrate that meta-learning further enhances this adaptation by 7.7% for various VQA tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 52 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

41 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
26 more
  1. 2016Matching Networks for One Shot Learningcited by 2,246
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2021MetaICL: Learning to Learn In Contextcited by 450
  7. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  8. 2018How to train your MAMLcited by 306
  9. 2023In-Context Learning Creates Task Vectorscited by 302
  10. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  11. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  12. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  13. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  14. 2021Meta-learning via Language Model In-context Tuningcited by 163
  15. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  16. 2022Fusing finetuned models for better pretrainingcited by 110
  17. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  18. 2023Meta-in-context learning in large language modelscited by 59
  19. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  20. 2025Test-Time Learning for Large Language Modelscited by 46
  21. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  22. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Modelscited by 21
  23. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
  24. 2022MetaPrompting: Learning to Learn Better Promptscited by 19
  25. 2022Towards Unified Prompt Tuning for Few-shot Text Classificationcited by 18
  26. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts
  2. 2023Multimodal Prompt Retrieval for Generative Visual Question Answering
  3. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
  4. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
  5. 2024LIVE: Learnable In-Context Vector for Visual Question Answering
  6. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
  7. 2024VidCtx: Context-aware Video Question Answering with Image Models
  8. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems