mlsanityData to 7 Oct 2026

Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering

arXiv 2409.07331 ↗First posted 11 Sep 2024cs.CV cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal large language models (MLLMs) have demonstrated great performance on visual question answering (VQA). When it comes to knowledge-based Visual Question Answering (KB-VQA), MLLMs may lack the specialized domain knowledge needed to answer questions, necessitating the retrieval of necessary information from external knowledge sources. Previous works like Retrival-Augmented VQA-v2 (RAVQA-v2) focus on utilizing as much input information, such as image-based textual descriptions and retrieved knowledge, as possible to improve performance, but they all overlook the issue that with the number of input tokens increasing, inference efficiency significantly decreases, which contradicts the demands of practical applications. To address this issue, we propose Retrieval-Augmented MLLMs with Compressed Contexts (RACC). RACC learns to compress and aggregate retrieved knowledge for a given image-question pair, generating a compact modulation in the form of Key-Value (KV) cache to adapt the downstream frozen MLLM, thereby achieving effective and efficient inference. RACC achieves a state-of-the-art (SOTA) performance of 63.92% on OK-VQA. Moreover, it significantly reduces inference latency by 22.0%-59.7% compared to the prominent RAVQA-v2. Abundant experiments show RACC's broad applicability. It is compatible with various off-the-shelf MLLMs and can also handle different knowledge sources including textual and multimodal documents.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 32 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

23 of 32 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  12. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
11 more
  1. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  2. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  3. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  4. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  5. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  6. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  7. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  8. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  9. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 36
  10. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  11. 2024Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Modelscited by 8

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  2. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  3. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering
  2. 2026Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning
  3. 2024Efficient Large Multi-modal Models via Visual Context Compression
  4. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  5. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering
  6. 2026MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering
  7. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  8. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA