mlsanityData to 7 Oct 2026

How to Configure Good In-Context Sequence for Visual Question Answering

arXiv 2312.01571 ↗First posted 4 Dec 2023cs.CV cs.AIcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Inspired by the success of Large Language Models in dealing with new tasks via In-Context Learning (ICL) in NLP, researchers have also developed Large Vision-Language Models (LVLMs) with ICL capabilities. However, when implementing ICL using these LVLMs, researchers usually resort to the simplest way like random sampling to configure the in-context sequence, thus leading to sub-optimal results. To enhance the ICL performance, in this study, we use Visual Question Answering (VQA) as case study to explore diverse in-context configurations to find the powerful ones. Additionally, through observing the changes of the LVLM outputs by altering the in-context sequence, we gain insights into the inner properties of LVLMs, improving our understanding of them. Specifically, to explore in-context configurations, we design diverse retrieval methods and employ different strategies to manipulate the retrieved demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2, VizWiz, and OK-VQA, we uncover three important inner properties of the applied LVLM and demonstrate which strategies can consistently improve the ICL VQA performance. Our code is provided in: https://github.com/GaryJiajia/OFv2_ICL_VQA.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 54 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

44 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
29 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  13. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  14. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  17. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  18. 2022Active Example Selection for In-Context Learningcited by 195
  19. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  20. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  21. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  22. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  23. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  24. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  25. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  26. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  27. 2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
  28. 2021Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioningcited by 23
  29. 2022Declaration-based Prompt Tuning for Visual Question Answeringcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 17; 2026: 6.

31 papers.

Sort:

By area:

  1. 2024VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learningcited by 34
  2. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  3. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  4. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  5. 2024Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learningcited by 16
  6. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  7. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  8. 2024DG-PIC: Domain Generalized Point-In-Context Learning for Point Cloud Understandingcited by 13
  9. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  10. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  11. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  12. 2024NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queriescited by 10
19 more
  1. 2025External Knowledge Injection for CLIP-Based Class-Incremental Learningcited by 10
  2. 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8
  3. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  4. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Modelscited by 6
  5. 2024LIVE: Learnable In-Context Vector for Visual Question Answeringcited by 5
  6. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Modelscited by 5
  7. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
  8. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 5
  9. 2025An Empirical Study on Configuring In-Context Learning Demonstrations for Unleashing MLLMs' Sentimental Perception Capabilitycited by 4
  10. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  11. 2025Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learningcited by 3
  12. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
  13. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  14. 2025Online In-Context Distillation for Low-Resource Vision Language Modelscited by 2
  15. 2026HIFICL: High-Fidelity In-Context Learning for Multimodal Taskscited by 1
  16. 2026Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathologycited by 0
  17. 2026Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learningcited by 0
  18. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  19. 2026MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SADL: An Effective In-Context Learning Method for Compositional Visual QA
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  3. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  4. 2024Visual In-Context Learning for Large Vision-Language Models
  5. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types
  6. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  7. 2025Evaluating Variance in Visual Question Answering Benchmarks
  8. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts