mlsanityData to 7 Oct 2026

Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

arXiv 2512.08923 ↗First posted 9 Dec 2025cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 43 references that are in the map. First posted on arXiv 9 Dec 2025.

Builds on

38 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Vision-Language Models Struggle to Align Entities across Modalities5 Mar
  2. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models22 Feb
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  4. 2024Cross-Modal Consistency in Multimodal Large Language Models14 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023A Survey on Multimodal Large Language Models23 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
18 more
  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2025Gemma 3 Technical Reportcited by 1,686
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2024Phi-4 Technical Reportcited by 675
  9. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  11. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  12. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  13. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  14. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  15. 2024Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIPcited by 23
  16. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  17. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 13
  18. 2024Privacy-Aware Visual Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 8
  2. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMscited by 6
  3. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  4. 2026Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generationcited by 0
  5. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMscited by 0
  6. 2026Through the LENS: Local Geometric Decomposition of Vision-Language Model Representationscited by 0
  7. 2026Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  2. 2024Law of Vision Representation in MLLMs
  3. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  4. 2025MLLMs are Deeply Affected by Modality Bias
  5. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Models
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2026LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
  8. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings