# QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi

arXiv: https://arxiv.org/abs/2605.18380 · HTML: https://mlsanity.com/paper/2605.18380/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 14 of its 77 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores](https://mlsanity.com/paper/2410.03492/) (2024)
- [Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs](https://mlsanity.com/paper/2304.11164/) (2023)
- [Human-in-the-Loop through Chain-of-Thought](https://mlsanity.com/paper/2306.07932/) (2023)
- [Scaling Parameter-Constrained Language Models with Quality Data](https://mlsanity.com/paper/2410.03083/) (2024)
- [An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8](https://mlsanity.com/paper/2309.15577/) (2023)
- [Can Large Language Models Reason about the Region Connection Calculus?](https://mlsanity.com/paper/2411.19589/) (2024)
- [A Survey of Qualitative Spatial and Temporal Calculi -- Algebraic and Computational Properties](https://mlsanity.com/paper/1606.00133/) (2016)
- [Understanding Understanding: A Pragmatic Framework Motivated by Large Language Models](https://mlsanity.com/paper/2406.10937/) (2024)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited](https://mlsanity.com/paper/2507.12059/) (2025)
- [DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning](https://mlsanity.com/paper/2511.02627/) (2025)

## Built on by (most cited first, 1 of 1)

- [Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives](https://mlsanity.com/paper/2610.17402/) (2026)

## Abstract (arXiv)

We introduce an extensive qualitative spatial and temporal reasoning (QSTR) benchmark for evaluating large language models (LLMs). We pose questions concerning compositional reasoning (using composition tables, CT), converse relations, and conceptual neighbourhoods (CN) for QSTR calculi, Point Algebra (PA), Allen's Interval Algebra, Interval and Duration (INDU), Region Connection Calculus (RCC-5, RCC-8, and RCC-22), the nine intersection model, cardinal direction calculus, and STAR. The RCC-22 CN is published here for the first time. An extended benchmark systematically varies question presentation including prefix/infix, words/symbols/nonce terms and schematic descriptions for selected calculi. We report results for contemporary frontier models. All models tested perform better than guessing but none can consistently answer all questions correctly. Performance varies sharply by calculus, with PA being the most straightforward, and RCC-22 the most difficult. We release the benchmark, and our results under an open licence to facilitate further assessment of qualitative spatio/temporal reasoning in LLMs.