# A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1

arXiv: https://arxiv.org/abs/2502.10867 · HTML: https://mlsanity.com/paper/2502.10867/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [Token-level Direct Preference Optimization](https://mlsanity.com/paper/2404.11999/) (2024)
- [Limitations of Autoregressive Models and Their Alternatives](https://mlsanity.com/paper/2010.11939/) (2020)
- [Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision](https://mlsanity.com/paper/2402.02658/) (2024)
- [Critique Ability of Large Language Models](https://mlsanity.com/paper/2310.04815/) (2023)
- [Why Can Large Language Models Generate Correct Chain-of-Thoughts?](https://mlsanity.com/paper/2310.13571/) (2023)

## Built on by (most cited first, 25 of 25)

- [Mind with Eyes: from Language Reasoning to Multimodal Reasoning](https://mlsanity.com/paper/2503.18071/) (2025)
- [AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models](https://mlsanity.com/paper/2505.17312/) (2025)
- [A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows](https://mlsanity.com/paper/2512.08769/) (2025)
- [Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2505.18536/) (2025)
- [Towards Responsible and Explainable AI Agents with Consensus-Driven Reasoning](https://mlsanity.com/paper/2512.21699/) (2025)
- [Standardization of Psychiatric Diagnoses -- Role of Fine-tuned LLM Consortium and OpenAI-gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2510.25588/) (2025)
- [QUITE: A Query Rewrite System Beyond Rules with LLM Agents](https://mlsanity.com/paper/2506.07675/) (2025)
- [Memento 2: Learning by Stateful Reflective Memory](https://mlsanity.com/paper/2512.22716/) (2025)
- [A Practical Guide to Agentic AI Transition in Organizations](https://mlsanity.com/paper/2602.10122/) (2026)
- [MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search](https://mlsanity.com/paper/2604.17265/) (2026)
- [Self-Verifying Reflection Helps Transformers with CoT Reasoning](https://mlsanity.com/paper/2510.12157/) (2025)
- [Evaluating the Efficacy of LLM-Based Reasoning for Multiobjective HPC Job Scheduling](https://mlsanity.com/paper/2506.02025/) (2025)
- [DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models](https://mlsanity.com/paper/2507.09955/) (2025)
- [Standardization of Neuromuscular Reflex Analysis -- Role of Fine-Tuned Vision-Language Model Consortium and OpenAI gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2508.12473/) (2025)
- [Semi-structured LLM Reasoners Can Be Rigorously Audited](https://mlsanity.com/paper/2505.24217/) (2025)
- [Flowr -- Scaling Up Retail Supply Chain Operations Through Agentic AI in Large Scale Supermarket Chains](https://mlsanity.com/paper/2604.05987/) (2026)
- [Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens](https://mlsanity.com/paper/2512.00729/) (2025)
- [UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling](https://mlsanity.com/paper/2605.30898/) (2026)
- [Semantic Distance Measurement based on Multi-Kernel Gaussian Processes](https://mlsanity.com/paper/2512.12238/) (2025)
- [Phylogenetic Tree Inference with Tropical Axial Attention](https://mlsanity.com/paper/2605.13894/) (2026)
- [Train the Trainers -- An Agentic AI Framework for Peer-Based Mental Health Support in Battlefield Environments](https://mlsanity.com/paper/2605.16269/) (2026)
- [IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs](https://mlsanity.com/paper/2607.01431/) (2026)
- [BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs](https://mlsanity.com/paper/2608.14161/) (2026)
- [Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search](https://mlsanity.com/paper/2608.15669/) (2026)
- [BaseCamp --- An Agentic AI Framework for Automating DNA Sequencing Data Pipelines](https://mlsanity.com/paper/2609.28557/) (2026)

## Abstract (arXiv)

OpenAI o1 has shown that applying reinforcement learning to integrate reasoning steps directly during inference can significantly improve a model's reasoning capabilities. This result is exciting as the field transitions from the conventional autoregressive method of generating answers to a more deliberate approach that models the slow-thinking process through step-by-step reasoning training. Reinforcement learning plays a key role in both the model's training and decoding processes. In this article, we present a comprehensive formulation of reasoning problems and investigate the use of both model-based and model-free approaches to better support this slow-thinking framework.