Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains unclear. We present a multi-domain benchmark spanning language, culture, mathematics, physics and chemistry, organized into three cognitive levels: perception and recognition, combination and reasoning, and association and critical thinking. Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures. These results show that symbolic understanding remains a major bottleneck for multimodal intelligence and motivate training and evaluation schemes that prioritize grounded perception in discrete semantic spaces.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 133 of its 168 references that are in the map. First posted on arXiv 19 Mar 2026.
Builds on
133 of 168 references.
Closest ancestors · 11
Same direction, newest first
- 2025Benchmarking Vision-Language Models on Chinese Ancient Documents: From OCR to Knowledge Reasoning10 Sep
- 2025ArtMentor: AI-Assisted Evaluation of Artworks to Explore Multimodal Large Language Models Capabilities19 Feb
- 2025Can Vision-Language Models Evaluate Handwritten Math?13 Jan
- 2024OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?2 Dec
- 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey26 Nov
- 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models29 Oct
- 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems21 Feb
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
Other references · 122
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Visual Instruction Tuningcited by 7,792
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024GPT-4o System Cardcited by 4,467
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2023A Survey on Multimodal Large Language Modelscited by 898
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
- 2022A Systematic Evaluation of Large Language Models of Codecited by 360
- 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
- 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
- 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
- 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
- 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
- 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
- 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
- 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
- 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
50 most cited of 122 All 122 (JSON)
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Latent Visual Cache for Video Reasoningcited by 3
- 2026TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoningcited by 2
- 2026Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoningcited by 1
- 2026From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilitiescited by 0
- 2026INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoningcited by 0
- 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillationcited by 0
- 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation15 Sep 2026
- 2026INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning27 Aug 2026
- 2026From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities27 Aug 2026
- 2026Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning17 Aug 2026
- 2026TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning16 Jul 2026
- 2026Latent Visual Cache for Video Reasoning1 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
- 2025Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
- 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
- 2025The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
- 2024REBUS: A Robust Evaluation Benchmark of Understanding Symbols
- 2026Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
- 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
- 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blind