QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
We introduce an extensive qualitative spatial and temporal reasoning (QSTR) benchmark for evaluating large language models (LLMs). We pose questions concerning compositional reasoning (using composition tables, CT), converse relations, and conceptual neighbourhoods (CN) for QSTR calculi, Point Algebra (PA), Allen's Interval Algebra, Interval and Duration (INDU), Region Connection Calculus (RCC-5, RCC-8, and RCC-22), the nine intersection model, cardinal direction calculus, and STAR. The RCC-22 CN is published here for the first time. An extended benchmark systematically varies question presentation including prefix/infix, words/symbols/nonce terms and schematic descriptions for selected calculi. We report results for contemporary frontier models. All models tested perform better than guessing but none can consistently answer all questions correctly. Performance varies sharply by calculus, with PA being the most straightforward, and RCC-22 the most difficult. We release the benchmark, and our results under an open licence to facilitate further assessment of qualitative spatio/temporal reasoning in LLMs.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 77 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
14 of 77 references.
Closest ancestors · 4
Same direction, newest first
- 2025DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning4 Nov
- 2024Can Large Language Models Reason about the Region Connection Calculus?29 Nov
- 2023An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-827 Sep
- 2023Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs22 Apr
Other references · 10
- 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
- 2025Small Language Models are the Future of Agentic AIcited by 270
- 2025Measuring AI Ability to Complete Long Software Taskscited by 156
- 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
- 2024Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scorescited by 25
- 2023Human-in-the-Loop through Chain-of-Thoughtcited by 20
- 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
- 2016A Survey of Qualitative Spatial and Temporal Calculi -- Algebraic and Computational Propertiescited by 3
- 2024Understanding Understanding: A Pragmatic Framework Motivated by Large Language Modelscited by 1
- 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning
- 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
- 2026BaziQA-Benchmark: Evaluating Symbolic and Temporally Compositional Reasoning in Large Language Models
- 2025Large Language and Reasoning Models are Shallow Disjunctive Reasoners
- 2025Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
- 2026QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
- 2024A NotSo Simple Way to Beat Simple Bench
- 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark