mlsanityData to 14 Oct 2026

Evaluating and Steering Modality Preferences in Multimodal Large Language Model

arXiv 2505.20977 ↗First posted 27 May 2025cs.CLcited by 33

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit modality preference, a tendency to favor one modality over another when processing multi-modal contexts. To study this question, we introduce textbfMCtextsuperscript2 benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making. Extensive experiments reveal that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences can serve as a useful indicator of downstream task performance of MLLMs. Further analysis shows that modality preference can be controlled by instruction guidance and captured within the latent representations of MLLMs. Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning. This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple multi-modal understanding and reasoning tasks.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 76 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

67 of 76 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs10 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models22 Aug
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023A Survey on Multimodal Large Language Models23 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
38 more
  1. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  2. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  4. 2022Visual Spatial Reasoningcited by 293
  5. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  6. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  7. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  8. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  9. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  10. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  11. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  12. 2018TallyQA: Answering Complex Counting Questionscited by 153
  13. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  14. 2022Balanced MSE for Imbalanced Visual Regressioncited by 89
  15. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67
  16. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  17. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 45
  18. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  19. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 44
  20. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 43
  21. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  22. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38
  23. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  24. 2021Multimodal Dialogue Response Generationcited by 32
  25. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  26. 2020On Modality Bias in the TVQA Datasetcited by 24
  27. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 24
  28. 2019Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshopcited by 22
  29. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 22
  30. 2021Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Modelscited by 21
  31. 2022On Modality Bias Recognition and Reductioncited by 20
  32. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoningcited by 19
  33. 2024Token Erasure as a Footprint of Implicit Vocabulary Items in LLMscited by 18
  34. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  35. 2024Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answeringcited by 13
  36. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  37. 2025Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoningcited by 10
  38. 2024Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledgecited by 8

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 21.

33 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  3. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  4. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  5. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  6. 2025When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMscited by 21
  7. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  8. 2026Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structurecited by 11
  9. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  10. 2026OPD-V: Visual On-Policy Self-Distillation with Modality Balancecited by 8
  11. 2025LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generationcited by 7
  12. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
21 more
  1. 2026Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognitioncited by 6
  2. 2026Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Seriescited by 5
  3. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 5
  4. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoningcited by 4
  5. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 4
  6. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  7. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  8. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 3
  9. 2026Decoupling Skeleton and Flesh: Efficient Multimodal Table Reasoning with Disentangled Alignment and Structure-aware Guidancecited by 3
  10. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  11. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflictcited by 2
  12. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1
  13. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1
  14. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Modelscited by 1
  15. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
  16. 2026Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidthcited by 0
  17. 2026User-Aware Active Knowledge Acquisition for Emotional Support Dialoguecited by 0
  18. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  19. 2026Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steeringcited by 0
  20. 2026MoGround: Measuring and Mitigating Modality Distraction in Vision-Language Modelscited by 0
  21. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
  2. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  3. 2025Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling
  4. 2025Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment
  5. 2026Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
  6. 2025MLLMs are Deeply Affected by Modality Bias
  7. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignment
  8. 2026Same Words, Different Judgments: How Preferences Vary Across Modalities