mlsanityData to 14 Oct 2026

DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning

arXiv 2511.02627 ↗First posted 4 Nov 2025cs.AIcited by 1

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning ability. The generation of DecompSR allows users to independently vary several aspects of compositionality, namely: productivity (reasoning depth), substitutivity (entity and linguistic variability), overgeneralisation (input order, distractors) and systematicity (novel linguistic elements). DecompSR is built procedurally in a manner which makes it is correct by construction, which is independently verified using a symbolic solver to guarantee the correctness of the dataset. DecompSR is comprehensively benchmarked across a host of Large Language Models (LLMs) where we show that LLMs struggle with productive and systematic generalisation in spatial reasoning tasks whereas they are more robust to linguistic variation. DecompSR provides a provably correct and rigorous benchmarking dataset with a novel ability to independently vary the degrees of several key aspects of compositionality, allowing for robust and fine-grained probing of the compositional reasoning abilities of LLMs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 62 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

38 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  2. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark8 Jan
  3. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts18 Apr

Other references · 35

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  10. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  11. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  12. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
23 more
  1. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  3. 2020Transformers as Soft Reasoners over Languagecited by 270
  4. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  6. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  7. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  8. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  9. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  10. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  11. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  12. 2023NeurASP: Embracing Neural Networks into Answer Set Programmingcited by 92
  13. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  14. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 68
  15. 2024On Limitations of the Transformer Architecturecited by 63
  16. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 63
  17. 2025Logical Reasoning in Large Language Models: A Surveycited by 55
  18. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  19. 2023Unsupervised Compositional Concepts Discovery with Text-to-Image Generative Modelscited by 36
  20. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  21. 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
  22. 2024Limits of Transformer Language Models on Learning to Compose Algorithmscited by 13
  23. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculicited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
  2. 2026From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
  3. 2024Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning
  4. 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Models
  5. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  6. 2024PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models
  7. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition
  8. 202511Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis