# AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPO

arXiv: https://arxiv.org/abs/2502.14669 · HTML: https://mlsanity.com/paper/2502.14669/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Imagine while Reasoning in Space: Multimodal Visualization-of-Thought](https://mlsanity.com/paper/2501.07542/) (2025)
- [A Configurable Library for Generating and Manipulating Maze Datasets](https://mlsanity.com/paper/2309.10498/) (2023)
- [Neurosymbolic AI for Reasoning over Knowledge Graphs: A Survey](https://mlsanity.com/paper/2302.07200/) (2023)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)

## Built on by (most cited first, 30 of 33)

- [Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL](https://mlsanity.com/paper/2503.23157/) (2025)
- [CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment](https://mlsanity.com/paper/2508.02298/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning](https://mlsanity.com/paper/2601.21037/) (2026)
- [RAIDX: A Retrieval-Augmented Generation and GRPO Reinforcement Learning Framework for Explainable Deepfake Detection](https://mlsanity.com/paper/2508.04524/) (2025)
- [MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis](https://mlsanity.com/paper/2506.18512/) (2025)
- [Reasoning LLMs are Wandering Solution Explorers](https://mlsanity.com/paper/2505.20296/) (2025)
- [What MLLMs Learn about When they Learn about Multimodal Reasoning](https://mlsanity.com/paper/2510.01719/) (2025)
- [A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models](https://mlsanity.com/paper/2508.12903/) (2025)
- [STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPO](https://mlsanity.com/paper/2604.00558/) (2026)
- [Less Redundancy: Boosting Practicality of Vision Language Model in Walking Assistants](https://mlsanity.com/paper/2508.16070/) (2025)
- [Orchestrator: Active Inference for Multi-Agent Systems in Long-Horizon Tasks](https://mlsanity.com/paper/2509.05651/) (2025)
- [Constructing coherent spatial memory in LLM agents through graph rectification](https://mlsanity.com/paper/2510.04195/) (2025)
- [Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding](https://mlsanity.com/paper/2511.19923/) (2025)
- [Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks](https://mlsanity.com/paper/2604.10690/) (2026)
- [Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought](https://mlsanity.com/paper/2604.17912/) (2026)
- [Probing Visual Planning in Image Editing Models](https://mlsanity.com/paper/2604.22868/) (2026)
- [Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Training](https://mlsanity.com/paper/2605.28008/) (2026)
- [Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation](https://mlsanity.com/paper/2606.06076/) (2026)
- [Does RLVR Extend Reasoning Boundaries? Investigating Capability Expansion in Vision-Language Models](https://mlsanity.com/paper/2511.00710/) (2025)
- [The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality](https://mlsanity.com/paper/2511.18084/) (2025)
- [From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning](https://mlsanity.com/paper/2512.24532/) (2025)
- [GraphWalk: Enabling Reasoning in Large Language Models through Tool-Based Graph Navigation](https://mlsanity.com/paper/2604.01610/) (2026)
- [Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym](https://mlsanity.com/paper/2604.09338/) (2026)
- [Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models](https://mlsanity.com/paper/2605.29303/) (2026)
- [AlphaSpace: Enabling Robotic Actions through Semantic Tokenization and Symbolic Reasoning](https://mlsanity.com/paper/2503.18769/) (2025)
- [Assessing Consciousness-Related Behaviors in Large Language Models Using the Maze Test](https://mlsanity.com/paper/2508.16705/) (2025)
- [Robust Object Detection for Autonomous Driving via Curriculum-Guided Group Relative Policy Optimization](https://mlsanity.com/paper/2509.22688/) (2025)
- [Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring](https://mlsanity.com/paper/2511.07803/) (2025)
- [Deconstructing Spatial Complexity: Hierarchical Decomposition for LLM Spatial Reasoning](https://mlsanity.com/paper/2605.28144/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capabilities in language processing, yet they often struggle with tasks requiring genuine visual spatial reasoning. In this paper, we introduce a novel two-stage training framework designed to equip standard LLMs with visual reasoning abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT) on a curated dataset of tokenized maze representations to teach the model to predict step-by-step movement commands. Next, we apply Group Relative Policy Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted reward function to refine the model's sequential decision-making and encourage emergent chain-of-thought behaviors. Experimental results on synthetically generated mazes show that while a baseline model fails to navigate the maze, the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more robust and self-corrective reasoning, highlighting the potential of our approach to bridge the gap between language models and visual spatial tasks. These findings offer promising implications for applications in robotics, autonomous navigation, and other domains that require integrated visual and sequential reasoning.