mlsanityData to 7 Oct 2026

Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

arXiv 2603.18472 ↗First posted 19 Mar 2026cs.AI cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains unclear. We present a multi-domain benchmark spanning language, culture, mathematics, physics and chemistry, organized into three cognitive levels: perception and recognition, combination and reasoning, and association and critical thinking. Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures. These results show that symbolic understanding remains a major bottleneck for multimodal intelligence and motivate training and evaluation schemes that prioritize grounded perception in discrete semantic spaces.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 133 of its 168 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

133 of 168 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning10 Sep
  2. 2025ArtMentor: AI-Assisted Evaluation of Artworks to Explore Multimodal Large Language Models Capabilities19 Feb
  3. 2025Can Vision-Language Models Evaluate Handwritten Math?13 Jan
  4. 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?2 Dec
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey26 Nov
  6. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models29 Oct
  7. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  8. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  9. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems21 Feb
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 122

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  13. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  15. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  16. 2023A Survey on Multimodal Large Language Modelscited by 898
  17. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  18. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  19. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  20. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  21. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  22. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  23. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  24. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  25. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  26. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  27. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  28. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  29. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  30. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  31. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  32. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  33. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  34. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  35. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  36. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  37. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  38. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180

50 most cited of 122 All 122 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
  2. 2025Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
  3. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
  4. 2025The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
  5. 2024REBUS: A Robust Evaluation Benchmark of Understanding Symbols
  6. 2026Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
  7. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
  8. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blind