# From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning

arXiv: https://arxiv.org/abs/2505.14425 · HTML: https://mlsanity.com/paper/2505.14425/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 42 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [Synthetic Data Generation Using Large Language Models: Advances in Text and Code](https://mlsanity.com/paper/2503.14023/) (2025)
- [Creative Robot Tool Use with Large Language Models](https://mlsanity.com/paper/2310.13065/) (2023)
- [Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset](https://mlsanity.com/paper/2409.17126/) (2024)
- [REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?](https://mlsanity.com/paper/2505.10872/) (2025)
- [Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs](https://mlsanity.com/paper/2405.20179/) (2024)
- [Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming](https://mlsanity.com/paper/2409.11041/) (2024)
- [ReLI: Cross-Lingual Language-to-Action Grounding for Human-Robot Interaction](https://mlsanity.com/paper/2505.01862/) (2025)

## Built on by (most cited first, 1 of 1)

- [For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances](https://mlsanity.com/paper/2609.19504/) (2026)

## Abstract (arXiv)

Instruction-tuned large language models (LLMs) have shown strong performance on a variety of tasks; however, generalizing from synthetic to human-authored instructions in grounded environments remains a challenge for them. In this work, we study generalization challenges in spatial grounding tasks where models interpret and translate instructions for building object arrangements on a 2.5D grid. We fine-tune LLMs using only synthetic instructions and evaluate their performance on a benchmark dataset containing both synthetic and human-written instructions. Our results reveal that while models generalize well on simple tasks, their performance degrades significantly on more complex tasks. We present a detailed error analysis of the gaps in instruction generalization.