mlsanityData to 14 Oct 2026

LIVE: Learnable In-Context Vector for Visual Question Answering

arXiv 2406.13185 ↗First posted 19 Jun 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, applying ICL usually faces two major challenges: 1) using more ICDs will largely increase the inference time and 2) the performance is sensitive to the selection of ICDs. These challenges are further exacerbated in LMMs due to the integration of multiple data types and the combinational complexity of multimodal ICDs. Recently, to address these challenges, some NLP studies introduce non-learnable In-Context Vectors (ICVs) which extract useful task information from ICDs into a single vector and then insert it into the LLM to help solve the corresponding task. However, although useful in simple NLP tasks, these non-learnable methods fail to handle complex multimodal tasks like Visual Question Answering (VQA). In this study, we propose Learnable In-Context VEctor (LIVE) to distill essential task information from demonstrations, improving ICL performance in LMMs. Experiments show that LIVE can significantly reduce computational costs while enhancing accuracy in VQA tasks compared to traditional ICL and other non-learnable ICV methods. The code is available at urlhttps://github.com/ForJadeForest/LIVE-Learnable-In-Context-Vector.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

43 of 57 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
31 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  11. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  12. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  13. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  14. 2022Active Example Selection for In-Context Learningcited by 196
  15. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  16. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  17. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  18. 2022Analyzing Transformers in Embedding Spacecited by 132
  19. 2023Finding Support Examples for In-Context Learningcited by 104
  20. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  21. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  22. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  23. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  24. 2023In-context Example Selection with Influencescited by 61
  25. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 33
  26. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 30
  27. 2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
  28. 2017Distinguishing Antonyms and Synonyms in a Pattern-based Neural Networkcited by 19
  29. 2024Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustmentcited by 17
  30. 2024CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Mannercited by 11
  31. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  2. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  3. 2024Rhyme-aware Chinese lyric generator based on GPTcited by 10
  4. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  5. 2026Not All Tokens Matter Equally: Dynamic In-context Vector Distillation with Decisive-Token Supervision for Long-form Medical Report Generationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  2. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  3. 2025Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
  4. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis
  5. 2025Visual RAG: Expanding MLLM visual knowledge without fine-tuning
  6. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  7. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
  8. 2023Link-Context Learning for Multimodal LLMs