mlsanityData to 14 Oct 2026

When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

arXiv 2608.19208 ↗First posted 12 Jun 2026cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) are frequently exposed to auxiliary textual context, the impact of which on visually grounded tasks remains underexplored. In this paper, we investigate the influence of task-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework. By maintaining an invariant prompt structure while varying auxiliary inputs, we observe that irrelevant text consistently biases model predictions across diverse benchmarks. To move beyond performance metrics, we characterize this sensitivity through a decision margin defined by the log-probability difference between binary candidates. Our analysis reveals a robust geometric regularity: contextconditioned margins follow a consistent affine transformation of their context-free counterparts. This finding demonstrates that irrelevant context does not manifest as unstructured stochastic noise but as a estimable distortion of model preference. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias. These findings provide a margin-level diagnostic view of irrelevant-context effects in MLLMs and offer a basis for future studies on noisy-context robustness

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 42 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

32 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
19 more
  1. 2016Visual Dialogcited by 377
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  3. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  4. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  5. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  6. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  7. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 185
  8. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  9. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  10. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  11. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMscited by 80
  12. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  13. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  14. 2020Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encodercited by 34
  15. 2025The Distracting Effect: Understanding Irrelevant Passages in RAGcited by 33
  16. 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
  17. 2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
  18. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generationcited by 11
  19. 2024Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  2. 2026V-FAT: Benchmarking Visual Fidelity Against Text-bias
  3. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
  4. 2025Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models
  5. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
  6. 2026The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
  7. 2026The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs
  8. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models