mlsanityData to 7 Oct 2026

Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities

arXiv 2410.01690 ↗First posted 2 Oct 2024cs.AIcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The various limitations of Generative AI, such as hallucinations and model failures, have made it crucial to understand the role of different modalities in Visual Language Model (VLM) predictions. Our work investigates how the integration of information from image and text modalities influences the performance and behavior of VLMs in visual question answering (VQA) and reasoning tasks. We measure this effect through answer accuracy, reasoning quality, model uncertainty, and modality relevance. We study the interplay between text and image modalities in different configurations where visual content is essential for solving the VQA task. Our contributions include (1) the Semantic Interventions (SI)-VQA dataset, (2) a benchmark study of various VLM architectures under different modality configurations, and (3) the Interactive Semantic Interventions (ISI) tool. The SI-VQA dataset serves as the foundation for the benchmark, while the ISI tool provides an interface to test and apply semantic interventions in image and text inputs, enabling more fine-grained analysis. Our results show that complementary information between modalities improves answer and reasoning quality, while contradictory information harms model performance and confidence. Image text annotations have minimal impact on accuracy and uncertainty, slightly increasing image relevance. Attention analysis confirms the dominant role of image inputs over text in VQA tasks. In this study, we evaluate state-of-the-art VLMs that allow us to extract attention coefficients for each modality. A key finding is PaliGemma's harmful overconfidence, which poses a higher risk of silent failures compared to the LLaVA models. This work sets the foundation for rigorous analysis of modality integration, supported by datasets specifically designed for this purpose.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 47 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

15 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  8. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  9. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 51
  10. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  11. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directionscited by 25
  12. 2024PIN: Positional Insert Unlocks Object Localisation Abilities in VLMscited by 14
2 more
  1. 2021What Vision-Language Models `See' when they See Scenescited by 8
  2. 2024AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Taskscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  3. 2025Do Multimodal Large Language Models Understand Welding?cited by 0
  4. 2026Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepanciescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  2. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  3. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  4. 2024Towards Flexible Evaluation for Generative Visual Question Answering
  5. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  6. 2025Evaluating Variance in Visual Question Answering Benchmarks
  7. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  8. 2026Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting