mlsanityData to 14 Oct 2026

From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning

arXiv 2505.14425 ↗First posted 20 May 2025cs.CLcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Instruction-tuned large language models (LLMs) have shown strong performance on a variety of tasks; however, generalizing from synthetic to human-authored instructions in grounded environments remains a challenge for them. In this work, we study generalization challenges in spatial grounding tasks where models interpret and translate instructions for building object arrangements on a 2.5D grid. We fine-tune LLMs using only synthetic instructions and evaluate their performance on a benchmark dataset containing both synthetic and human-written instructions. Our results reveal that while models generalize well on simple tasks, their performance degrades significantly on more complex tasks. We present a detailed error analysis of the gaps in instruction generalization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 42 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

9 of 42 references.

References · 9

  1. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  2. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  3. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  4. 2023Creative Robot Tool Use with Large Language Modelscited by 43
  5. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 15
  6. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  7. 2024Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMscited by 6
  8. 2024Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programmingcited by 2
  9. 2025ReLI: Cross-Lingual Language-to-Action Grounding for Human-Robot Interactioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
  2. 2026Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks
  3. 2024Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?
  4. 2026From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
  5. 2025LTD-Bench: Evaluating Large Language Models by Letting Them Draw
  6. 2025FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations
  7. 2026From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
  8. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning