mlsanityData to 14 Oct 2026

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

arXiv 2502.09696 ↗First posted 13 Feb 2025cs.CVcited by 58

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 85 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

43 of 85 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Pixtral 12B9 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  4. 2024A Survey on Benchmarks of Multimodal Large Language Models16 Aug
  5. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  6. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation14 May
  7. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  8. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  9. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models18 Apr
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
1 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
18 more
  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  2. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  3. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  4. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  5. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  6. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  7. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  8. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  9. 2024ARC Prize 2024: Technical Reportcited by 104
  10. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  11. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 66
  12. 2021Scaling Scaling Laws with Board Gamescited by 63
  13. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  14. 2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
  15. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 30
  16. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  17. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 20
  18. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 28.

58 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  3. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  5. 2026Qwen3.5-Omni Technical Reportcited by 174
  6. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  7. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
  8. 2026Kimi K3: Open Frontier Intelligencecited by 138
  9. 2026Self-Distilled RLVRcited by 131
  10. 2025Visual Planning: Let's Think Only with Imagescited by 85
  11. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  12. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
38 more
  1. 2025Kwai Keye-VL Technical Reportcited by 47
  2. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  3. 2026STEP3-VL-10B Technical Reportcited by 32
  4. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  5. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 24
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  7. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  8. 2025NVIDIA Nemotron Nano V2 VLcited by 17
  9. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  10. 2025OmniEarth-Bench: Towards Holistic Evaluation of Earth's Six Spheres and Cross-Spheres Interactions with Multimodal Observational Earth Datacited by 14
  11. 2025VideoGameQA-Bench: Evaluating Vision-Language Models for Video Game Quality Assurancecited by 11
  12. 2025Information Density Principle for MLLM Benchmarkscited by 9
  13. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  14. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  15. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 9
  16. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
  17. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  18. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturationcited by 7
  19. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 7
  20. 2026HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoningcited by 5
  21. 2026Co-Evolving Policy Distillationcited by 5
  22. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmarkcited by 4
  23. 2026MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attentioncited by 4
  24. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Modelscited by 4
  25. 2025Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiationcited by 3
  26. 2025MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacitycited by 3
  27. 2025Perceptual Taxonomy: Evaluating and Guiding Hierarchical Scene Reasoning in Vision-Language Modelscited by 3
  28. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  29. 2025MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Modelscited by 2
  30. 2026MetaphorStar: Image Metaphor Understanding and Reasoning with End-to-End Visual Reinforcement Learningcited by 2
  31. 2026Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Modelscited by 2
  32. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  33. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  34. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  35. 2026CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisitioncited by 1
  36. 2026GenPuzzle: Benchmarking Visual Reasoning in Image Generation Modelscited by 1
  37. 2026Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Visioncited by 1
  38. 2025Aligning MLLM Benchmark With Human Preferences via Structural Equation Modelingcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
  2. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  3. 2025OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks
  4. 2026DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models
  5. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotation
  6. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models
  7. 2026MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
  8. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks