mlsanityData to 14 Oct 2026

Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks

arXiv 2509.02175 ↗First posted 2 Sep 2025cs.CV cs.AI cs.CLcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We propose RocketScience, an open-source contrastive VLM benchmark that tests for spatial relation understanding. It is comprised of entirely new real-world image-text pairs covering mostly relative spatial understanding and the order of objects. The benchmark is designed to be very easy for humans and hard for the current generation of VLMs, and this is empirically verified. Our results show a striking lack of spatial relation understanding in open source and frontier commercial VLMs and a surprisingly high performance of reasoning models. Additionally, we perform a disentanglement analysis to separate the contributions of object localization and spatial reasoning in chain-of-thought-based models and find that the performance on the benchmark is bottlenecked by spatial reasoning and not object localization capabilities. We release the dataset with a CC-BY-4.0 license and make the evaluation code available at: https://github.com/nilshoehing/rocketscience

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 77 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

53 of 77 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning18 Mar
  2. 2025FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks25 Feb
  3. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models12 Feb
  4. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  5. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  6. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  7. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  8. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 45

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  12. 2025Humanity's Last Examcited by 491
33 more
  1. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  2. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  3. 2023Teaching CLIP to Count to Tencited by 198
  4. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  6. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  7. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  8. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 107
  9. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  10. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  11. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  12. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  13. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  14. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  15. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  16. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  17. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Drivingcited by 35
  18. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  19. 2024How Far Are We from Intelligent Visual Deductive Reasoning?cited by 29
  20. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 24
  21. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  22. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  23. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 20
  24. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  25. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 18
  26. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  27. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  28. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 13
  29. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  30. 2023What's left can't be right -- The remaining positional incompetence of contrastive vision-language modelscited by 6
  31. 2024BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrievalcited by 6
  32. 2024NL-Eye: Abductive NLI for Imagescited by 5
  33. 2025Metamorphic Testing for Pose Estimation Systemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2026Grounding Isn't Knowing: Do VLMs Need Object Localization for Spatial Reasoning?
  3. 2025Spatial Reasoning in Foundation Models: Benchmarking Object-Centric Spatial Understanding
  4. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  5. 2024GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs
  6. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
  7. 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces
  8. 2026SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs