# FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

arXiv: https://arxiv.org/abs/2507.07644 · HTML: https://mlsanity.com/paper/2507.07644/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 53 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023)
- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark](https://mlsanity.com/paper/2412.07825/) (2024)
- [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024)
- [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024)
- [InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset](https://mlsanity.com/paper/1809.00716/) (2018)
- [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition](https://mlsanity.com/paper/1908.02660/) (2019)
- [CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis](https://mlsanity.com/paper/1904.01920/) (2019)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)

## Built on by (most cited first, 13 of 13)

- [BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction](https://mlsanity.com/paper/2510.16559/) (2025)
- [Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions](https://mlsanity.com/paper/2601.03590/) (2026)
- [TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning](https://mlsanity.com/paper/2601.16520/) (2026)
- [Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation](https://mlsanity.com/paper/2510.03863/) (2025)
- [NoReGeo: Non-Reasoning Geometry Benchmark](https://mlsanity.com/paper/2601.10254/) (2026)
- [SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in Large Language Models](https://mlsanity.com/paper/2603.03002/) (2026)
- [How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning](https://mlsanity.com/paper/2603.24866/) (2026)
- [World Craft: Agentic Framework to Create Visualizable Worlds via Text](https://mlsanity.com/paper/2601.09150/) (2026)
- [Spatial Competence Benchmark](https://mlsanity.com/paper/2604.09594/) (2026)
- [Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning](https://mlsanity.com/paper/2605.28277/) (2026)
- [Architect-Ant: Editable Automatic Furnishing of Architectural Floor Plans](https://mlsanity.com/paper/2606.10953/) (2026)
- [OmniLayout: A Schematic-Coupled Multimodal Benchmark for Constraint-Aware Geometric Reasoning in PCB Layout](https://mlsanity.com/paper/2607.03261/) (2026)
- [HALO: A Physics-Aware LLM Agent Framework for Nanophotonic Design](https://mlsanity.com/paper/2608.28877/) (2026)

## Abstract (arXiv)

We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes, such as (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed in shallow queries, they often fail to respect physical constraints, preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today's LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings.