Large Language Models' Reasoning Stalls: An Investigation into the Capabilities of Frontier Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Empirical methods to examine the capability of Large Language Models (LLMs) to use Automated Theorem Prover (ATP) reasoning strategies are studied. We evaluate the performance of State of the Art models from December 2023 and August 2024 on PRONTOQA steamroller reasoning problems. For that, we develop methods for assessing LLM response accuracy and correct answer correlation. Our results show that progress in improving LLM reasoning abilities has stalled over the nine month period. By tracking completion tokens, we show that almost all improvement in reasoning ability since GPT-4 was released can be attributed to either hidden system prompts or the training of models to automatically use generic Chain of Thought prompting strategies. Among the ATP reasoning strategies tried, we found that current frontier LLMs are best able to follow the bottom-up (also known as forward-chaining) strategy. A low positive correlation was found between an LLM response containing correct reasoning and arriving at the correct conclusion.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 0 of its 94 references that are in the map. First posted on arXiv 26 May 2025.
Builds on
0 of 94 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Steamroller Problems: An Evaluation of LLM Reasoning Capability with Automated Theorem Prover Strategies
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2025Large Language Models and Mathematical Reasoning Failures
- 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
- 2025Reasoning Capabilities and Invariability of Large Language Models
- 2024General Purpose Verification for Chain of Thought Prompting
- 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models