mlsanityData to 7 Oct 2026

GUIDE: Guiding Internal Evidence with Language Instructions

arXiv 2608.30712 ↗First posted 31 Aug 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models follow instructions about what to generate, but not necessarily about what evidence to rely on. Hence, models may continue to depend on shortcut-associated cues even when instructions suggest otherwise. We introduce GUIDE, a framework for controlling internal evidence usage through language instructions. GUIDE combines grouped parameter-efficient adaptation with instruction-conditioned gating to modulate multimodal evidence pathways during reasoning and generation. We further introduce a pathway-level evaluation framework that characterizes instruction-conditioned evidence modulation through reliance sensitivity, controlled perturbation analysis, pathway modulation, and autoregressive decoding dynamics. Across multimodal reasoning, classification, and generation, GUIDE induces structured and instruction-aligned redistribution of evidence reliance while largely preserving task behavior. Experiments on GQA, TextVQA, MM-IMDb, CREMA-D, RAVDESS, and Flickr30K show that GUIDE improves robustness under targeted evidence perturbations and enables controllable modulation across diverse multimodal settings. This suggests that multimodal instruction following can extend beyond output control toward regulating how different evidence sources contribute to model predictions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 30 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

18 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 15

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2023Steering Language Models With Activation Engineeringcited by 824
  7. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  8. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  9. 2017Gated Multimodal Units for Information Fusioncited by 244
  10. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  11. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  12. 2025Controlling Large Language Models Through Concept Activation Vectorscited by 27
3 more
  1. 2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26
  2. 2024Focus On This, Not That! Steering LLMs with Adaptive Feature Specificationcited by 5
  3. 2025MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corporacited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
  2. 2026Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
  3. 2026Learning Multimodal Embeddings with Evidence-Aligned Readout
  4. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
  5. 2025Emergent Bayesian Behaviour and Optimal Cue Combination in LLMs
  6. 2025Learning Deliberately, Acting Intuitively: Unlocking Test-Time Reasoning in Multimodal LLMs
  7. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models
  8. 2026Learning Domain Knowledge in Multimodal Large Language Models through Reinforcement Fine-Tuning