mlsanityData to 7 Oct 2026

When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1

arXiv 2410.01792 ↗First posted 2 Oct 2024cs.CL cs.AIcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In "Embers of Autoregression" (McCoy et al., 2023), we showed that several large language models (LLMs) have some important limitations that are attributable to their origins in next-word prediction. Here we investigate whether these issues persist with o1, a new system from OpenAI that differs from previous LLMs in that it is optimized for reasoning. We find that o1 substantially outperforms previous LLMs in many cases, with particularly large improvements on rare variants of common tasks (e.g., forming acronyms from the second letter of each word in a list, rather than the first letter). Despite these quantitative improvements, however, o1 still displays the same qualitative trends that we observed in previous systems. Specifically, o1 -- like previous LLMs -- is sensitive to the probability of examples and tasks, performing better and requiring fewer "thinking tokens" in high-probability settings than in low-probability ones. These results show that optimizing a language model for reasoning can mitigate but might not fully overcome the language model's probability sensitivity.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 4 of its 12 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

4 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 3

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoningcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 5.

22 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  3. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
  4. 2025What is a Number, That a Large Language Model May Know It?cited by 14
  5. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 13
  6. 2025A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experimentscited by 10
  7. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  8. 2025AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Modelscited by 7
  9. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  10. 2025LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integrationcited by 5
  11. 2025LLM Agents Are Hypersensitive to Nudgescited by 5
  12. 2025Evaluating Language Models' Evaluations of Gamescited by 4
10 more
  1. 2025LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscationcited by 3
  2. 2025ATTS: Asynchronous Test-Time Scaling via Conformal Predictioncited by 3
  3. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  4. 2025ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluationcited by 1
  5. 2026DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72cited by 1
  6. 2026Position: Behavioral Systems Require Behavioral Testscited by 1
  7. 2025Towards Contamination Resistant Benchmarkscited by 0
  8. 2026Lifted Relational Probabilistic Inference via Implicit Learningcited by 0
  9. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoningcited by 0
  10. 2026Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Playcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
  2. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  3. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
  4. 2025Large Language Models Imitate Logical Reasoning, but at what Cost?
  5. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer
  6. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Models
  7. 2024On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability
  8. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches