When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Multimodal language models are typically evaluated through external behavior: selecting the correct image--text match, rejecting unsupported captions, or answering visual queries correctly. However, correct behavior alone does not show that the model's internal decision state remains stable under controlled semantic stress. We study this gap through S3E (Structured Semantic Stress Evaluation), a framework for analyzing behavior-internal decoupling in multimodal language models. S3E uses a positive-anchored A/B forced-choice setup in which an image-supported caption is contrasted against semantic stress candidates under both original and swapped option orders, while hidden states are extracted at the pre-answer decision state. We focus on strict-correct trials, where the model consistently selects the correct caption across both orders. Rather than treating arbitrary hidden-state variation as evidence of instability, we measure whether semantic-conflict candidates induce excess decision-state displacement relative to meaning-preserving controls. Across Qwen3VL, Gemma3, and InternVL3, semantic stress consistently produces positive selected-layer excess displacement over lexical controls despite correct forced-choice behavior, while comparisons against random negatives are model-dependent. We interpret this as a scoped decision-state stress-sensitivity signal rather than evidence of downstream failure or hallucination. Our results suggest that forced-choice correctness alone is not a sufficient certificate of invariant internal decision geometry.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 22 references that are in the map. First posted on arXiv 7 Jun 2026.
Builds on
12 of 22 references.
Closest ancestors · 2
Same direction, newest first
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
- 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
Other references · 10
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2021Causal Abstractions of Neural Networkscited by 401
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
- 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Modelscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
- 2026HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide
- 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
- 2026Uncertainty-Aware Decision Making in Multimodal Large Language Models
- 2026When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models
- 2026CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs
- 2026Attending to Multimodal Generation One Token at a Time
- 2026From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making