# GridRoute: A Benchmark for LLM-Based Route Planning with Cardinal Movement in Grid Environments

arXiv: https://arxiv.org/abs/2505.24306 · HTML: https://mlsanity.com/paper/2505.24306/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 8 of 8)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning](https://mlsanity.com/paper/2310.03249/) (2023)
- [LLM-A*: Large Language Model Enhanced Incremental Heuristic Search on Path Planning](https://mlsanity.com/paper/2407.02511/) (2024)
- [Language Model Meets Prototypes: Towards Interpretable Text Classification Models through Prototypical Networks](https://mlsanity.com/paper/2412.03761/) (2024)
- [SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers](https://mlsanity.com/paper/2502.20545/) (2025)

## Built on by (most cited first, 4 of 4)

- [NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities](https://mlsanity.com/paper/2509.16656/) (2025)
- [LLMs for Text-Based Exploration and Navigation Under Partial Observability](https://mlsanity.com/paper/2604.09604/) (2026)
- [SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems](https://mlsanity.com/paper/2604.16022/) (2026)
- [TransitLM: A Large-Scale Dataset and Benchmark for Map-Free Transit Route Generation](https://mlsanity.com/paper/2605.22355/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have demonstrated their potential in planning and reasoning tasks, offering a flexible alternative to classical pathfinding algorithms. However, most existing studies focus on LLMs' independent reasoning capabilities and overlook the potential synergy between LLMs and traditional algorithms. To fill this gap, we propose a comprehensive evaluation benchmark GridRoute to assess how LLMs can take advantage of traditional algorithms. We also propose a novel hybrid prompting technique called Algorithm of Thought (AoT), which introduces traditional algorithms' guidance into prompting. Our benchmark evaluates six LLMs ranging from 7B to 72B parameters across various map sizes, assessing their performance in correctness, optimality, and efficiency in grid environments with varying sizes. Our results show that AoT significantly boosts performance across all model sizes, particularly in larger or more complex environments, suggesting a promising approach to addressing path planning challenges. Our code is open-sourced at https://github.com/LinChance/GridRoute.