TRAC: A Textual Benchmark for Reasoning about Actions and Change
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Reasoning about actions and change (RAC) is essential to understand and interact with the ever-changing environment. Previous AI research has shown the importance of fundamental and indispensable knowledge of actions, i.e., preconditions and effects. However, traditional methods rely on logical formalization which hinders practical applications. With recent transformer-based language models (LMs), reasoning over text is desirable and seemingly feasible, leading to the question of whether LMs can effectively and efficiently learn to solve RAC problems. We propose four essential RAC tasks as a comprehensive textual benchmark and generate problems in a way that minimizes the influence of other linguistic requirements (e.g., grounding) to focus on RAC. The resulting benchmark, TRAC, encompassing problems of various complexities, facilitates a more granular evaluation of LMs, precisely targeting the structural generalization ability much needed for RAC. Experiments with three high-performing transformers indicates that additional efforts are needed to tackle challenges raised by TRAC.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 24 references that are in the map. First posted on arXiv 25 Nov 2022.
Builds on
12 of 24 references.
References · 12
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
- 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
- 2020Transformers as Soft Reasoners over Languagecited by 270
- 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
- 2018Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehensioncited by 62
- 2017Simulating Action Dynamics with Neural Process Networkscited by 53
- 2020PRover: Proof Generation for Interpretable Reasoning over Rulescited by 53
- 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
- 2019Regression Planning Networkscited by 32
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
- 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset
- 2025ProRAC: A Neuro-symbolic Method for Reasoning about Actions with LLM-based Progression
- 2025LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actions
- 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
- 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
- 2025Pre-Act: Multi-Step Planning and Reasoning Improves Acting in LLM Agents
- 2022ReAct: Synergizing Reasoning and Acting in Language Models