CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in multimodal understanding and generation. However, when textual inputs conflict with visual evidence, they still suffer from hallucinations and produce responses inconsistent with visual content. Existing approaches mainly rely on decoding strategies, additional training, verification methods, or prompting techniques, but often lack fine-grained conflict localization and conflict-aware generation. In this work, we propose CAER, a backbone-agnostic framework for visual-language conflict detection and conflict-aware generation. CAER introduces a span-grounded evidence router that transforms claim representations into soft textual queries and retrieves corresponding evidence from frozen visual tokens, enabling fine-grained conflict estimation. Furthermore, we design a dual-prefix expert routing mechanism that learns separate experts for visually supported and contradicted inputs, enabling conflict-aware generation through explicit expert selection. Experiments on the public MMMC benchmark and our newly curated AgriConflict dataset demonstrate that CAER effectively detects visual-language conflicts and improves the reliability of open-source MLLMs without updating their backbone parameters.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 22 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
12 of 22 references.
Closest ancestors · 1
Same direction, newest first
Other references · 11
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
- 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
- 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
- 2025Robust Multimodal Large Language Models Against Modality Conflictcited by 18
- 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
- 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
- 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
- 2026Improving Vision-language Models with Perception-centric Process Reward Modelscited by 4
- 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
- 2025CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution
- 2026Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
- 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning
- 2026CORE: Conflict-Oriented Reasoning for General Multimodal Manipulation Detection
- 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
- 2026LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
- 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict