Exploring State Tracking Capabilities of Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on state tracking, a problem where models need to keep track of the state governing a number of entities. To isolate the state tracking component from other factors, we propose a benchmark based on three well-defined state tracking tasks and analyse the performance of LLMs in different scenarios. The results indicate that the recent generation of LLMs (specifically, GPT-4 and Llama3) are capable of tracking state, especially when integrated with mechanisms such as Chain of Thought. However, models from the former generation, while understanding the task and being able to solve it at the initial stages, often fail at this task after a certain number of steps.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Nov 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Scaling Laws for State Dynamics in Large Language Models
- 2025On the Limits of Innate Planning in Large Language Models
- 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
- 2024LLM2: Let Large Language Models Harness System 2 Reasoning
- 2025Tracking World States with Language Models: State-Based Evaluation Using Chess
- 2026A State-Transition Framework for Efficient LLM Reasoning
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2026Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering