Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Vision language models (VLMs) perform well on many tasks but often fail at spatial reasoning, which is essential for navigation and interaction with physical environments. Many spatial reasoning tasks depend on fundamental two-dimensional (2D) skills, yet our evaluation shows that state-of-the-art VLMs give implausible or incorrect answers to composite spatial problems, including simple pathfinding tasks that humans solve effortlessly. To address this, we enhance 2D spatial reasoning in VLMs by training them only on basic spatial capabilities. We first disentangle 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. We hypothesize that mastering these skills substantially improves performance on complex spatial tasks that require advanced reasoning and combinatorial problem solving, while also generalizing to real-world scenarios. To test this, we introduce Sparkle, a framework that generates synthetic data to provide targeted supervision across these three capabilities and yields an instruction dataset for each. Experiments show that VLMs fine-tuned with Sparkle improve not only on basic tasks but also on composite and out-of-distribution real-world spatial reasoning tasks. These results indicate that enhancing basic spatial skills through synthetic generalization effectively advances complex spatial reasoning and offers a systematic strategy for boosting the spatial understanding of VLMs. Source codes of Sparkle are available at https://github.com/YihongT/Sparkle.
Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 64 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
52 of 64 references.
Closest ancestors · 8
Same direction, newest first
- 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
- 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Models20 Jun
- 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasoners4 Jun
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
- 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark8 Jan
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
- 2022Visual Spatial Reasoning30 Apr
- 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts18 Apr
Other references · 44
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Visual Instruction Tuningcited by 7,792
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
32 more
- 2015VQA: Visual Question Answeringcited by 2,489
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2019Towards VQA Models That Can Readcited by 1,696
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
- 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
- 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
- 2022Scaling Language-Image Pre-training via Maskingcited by 317
- 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
- 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
- 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
- 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
- 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
- 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
- 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
- 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
- 2024ITINERA: Integrating Spatial Optimization with Large Language Models for Open-domain Urban Itinerary Planningcited by 33
- 2023A Configurable Library for Generating and Manipulating Maze Datasetscited by 31
- 2021Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?cited by 10
- 2020Confidence-aware Non-repetitive Multimodal Transformers for TextCapscited by 5
Built on by
Citing papers per year: 2024: 2; 2025: 22; 2026: 9.
33 papers.
Sort:
By area:
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
- 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Modelscited by 29
- 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
- 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
- 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometricscited by 21
- 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
- 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
- 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 10
- 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understandingcited by 8
- 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
21 more
- 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
- 2025Vision-to-Music Generation: A Surveycited by 4
- 2025Vision language models are unreliable at trivial spatial cognitioncited by 4
- 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Modelscited by 4
- 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
- 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
- 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
- 2025GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedbackcited by 2
- 2025GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedracited by 2
- 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
- 2025E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Drivingcited by 2
- 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimationcited by 2
- 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 2
- 2024Exploring Compositional Generalization of Multimodal LLMs for Medical Imagingcited by 1
- 2025VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoningcited by 1
- 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
- 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
- 2026VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizingcited by 0
- 2026Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structurescited by 0
- 2026Reinforcing Dual-Path Reasoning in Spatial Vision Language Modelscited by 0
- 2026AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neurosciencecited by 0
- 2026AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neuroscience30 Jun 2026
- 2026Reinforcing Dual-Path Reasoning in Spatial Vision Language Models16 Jun 2026
- 2026Grounded 3D-Aware Spatial Vision-Language Modeling28 May 2026
- 2026Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures18 Mar 2026
- 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations17 Mar 2026
- 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation5 Feb 2026
- 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation29 Jan 2026
- 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning19 Jan 2026
- 2026VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizing12 Jan 2026
- 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark31 Dec 2025
- 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs22 Dec 2025
- 2025E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving4 Dec 2025
21 more
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks29 Oct 2025
- 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation25 Oct 2025
- 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study24 Oct 2025
- 2025VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoning29 Sep 2025
- 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation10 Sep 2025
- 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models1 Jul 2025
- 2025GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedra9 Jun 2025
- 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence9 Jun 2025
- 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs4 Jun 2025
- 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models8 May 2025
- 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation24 Apr 2025
- 2025Vision language models are unreliable at trivial spatial cognition22 Apr 2025
- 2025Large Language and Reasoning Models are Shallow Disjunctive Reasoners30 Mar 2025
- 2025Vision-to-Music Generation: A Survey27 Mar 2025
- 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models25 Mar 2025
- 2025GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedback19 Mar 2025
- 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics17 Feb 2025
- 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs5 Feb 2025
- 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding14 Jan 2025
- 2024Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging28 Dec 2024
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec 2024
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
- 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Modelscited by 29
- 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMscited by 24
- 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
- 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 10
- 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
- 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
- 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
- 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
- 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
12 most cited of 21 All 21 (JSON)
- 2026AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neuroscience30 Jun 2026
- 2026Reinforcing Dual-Path Reasoning in Spatial Vision Language Models16 Jun 2026
- 2026Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures18 Mar 2026
- 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations17 Mar 2026
- 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation5 Feb 2026
- 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation29 Jan 2026
- 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning19 Jan 2026
- 2026VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizing12 Jan 2026
- 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark31 Dec 2025
- 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs22 Dec 2025
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks29 Oct 2025
- 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation25 Oct 2025
12 newest of 21 All 21 (JSON)
- 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
- 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometricscited by 21
- 2025Vision language models are unreliable at trivial spatial cognitioncited by 4
- 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Modelscited by 4
- 2025GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedbackcited by 2
- 2024Exploring Compositional Generalization of Multimodal LLMs for Medical Imagingcited by 1
- 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
- 2026Grounded 3D-Aware Spatial Vision-Language Modeling28 May 2026
- 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence9 Jun 2025
- 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models8 May 2025
- 2025Vision language models are unreliable at trivial spatial cognition22 Apr 2025
- 2025GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedback19 Mar 2025
- 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics17 Feb 2025
- 2024Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging28 Dec 2024
- 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
- 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understandingcited by 8
- 2025Vision-to-Music Generation: A Surveycited by 4
- 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
- 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study24 Oct 2025
- 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation10 Sep 2025
- 2025Vision-to-Music Generation: A Survey27 Mar 2025
- 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding14 Jan 2025
Similar, not cited
Close in wording, no citation either way
- 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
- 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
- 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
- 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models
- 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
- 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?
- 2025Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
- 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective