# Reasoning Language Models: A Blueprint

arXiv: https://arxiv.org/abs/2501.11223 · HTML: https://mlsanity.com/paper/2501.11223/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 38 of its 191 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 38)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [Free Process Rewards without Process Labels](https://mlsanity.com/paper/2412.01981/) (2024)
- [Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B](https://mlsanity.com/paper/2406.07394/) (2024)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions](https://mlsanity.com/paper/2411.14405/) (2024)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [AgentGym: Evolving Large Language Model-based Agents across Diverse Environments](https://mlsanity.com/paper/2406.04151/) (2024)
- [Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning](https://mlsanity.com/paper/2412.09078/) (2024)
- [OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models](https://mlsanity.com/paper/2410.09671/) (2024)
- [Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap](https://mlsanity.com/paper/2402.19450/) (2024)
- [Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation](https://mlsanity.com/paper/2410.02725/) (2024)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [FIMO: A Challenge Formal Dataset for Automated Theorem Proving](https://mlsanity.com/paper/2309.04295/) (2023)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective](https://mlsanity.com/paper/2412.14135/) (2024)
- [Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTS](https://mlsanity.com/paper/2402.03289/) (2024)

## Built on by (most cited first, 30 of 42)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaos](https://mlsanity.com/paper/2502.15806/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluation](https://mlsanity.com/paper/2502.01477/) (2025)
- [Generative AI Act II: Test Time Scaling Drives Cognition Engineering](https://mlsanity.com/paper/2504.13828/) (2025)
- [CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning](https://mlsanity.com/paper/2505.13271/) (2025)
- [Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models](https://mlsanity.com/paper/2506.04832/) (2025)
- [When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models](https://mlsanity.com/paper/2504.02010/) (2025)
- [How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients](https://mlsanity.com/paper/2504.10766/) (2025)
- [VIDEOP2R: Video Understanding from Perception to Reasoning](https://mlsanity.com/paper/2511.11113/) (2025)
- [Schoenfeld's Anatomy of Mathematical Reasoning by Language Models](https://mlsanity.com/paper/2512.19995/) (2025)
- [Affordable AI Assistants with Knowledge Graph of Thoughts](https://mlsanity.com/paper/2504.02670/) (2025)
- [Can Large Language Models Adequately Perform Symbolic Reasoning Over Time Series?](https://mlsanity.com/paper/2508.03963/) (2025)
- [Children's Mental Models of AI Reasoning: Implications for AI Literacy Education](https://mlsanity.com/paper/2505.16031/) (2025)
- [A Study on Thinking Patterns of Large Reasoning Models in Code Generation](https://mlsanity.com/paper/2509.13758/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge](https://mlsanity.com/paper/2511.03070/) (2025)
- [Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2603.00296/) (2026)
- [Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions](https://mlsanity.com/paper/2605.00226/) (2026)
- [One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligence](https://mlsanity.com/paper/2506.10157/) (2025)
- [Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering](https://mlsanity.com/paper/2509.10249/) (2025)
- [Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction](https://mlsanity.com/paper/2606.28186/) (2026)
- [BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models](https://mlsanity.com/paper/2511.13095/) (2025)
- [GraphSeek: Next-Generation Graph Analytics with LLMs](https://mlsanity.com/paper/2602.11052/) (2026)
- [A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2](https://mlsanity.com/paper/2603.19253/) (2026)
- [Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models](https://mlsanity.com/paper/2506.13300/) (2025)

## Abstract (arXiv)

Reasoning language models (RLMs), also known as Large Reasoning Models (LRMs), such as OpenAI's o1 and o3, DeepSeek-R1, and Alibaba's QwQ, have redefined AI's problem-solving capabilities by extending LLMs with advanced reasoning mechanisms. Yet, their high costs, proprietary nature, and complex architectures - uniquely combining reinforcement learning (RL), search heuristics, and LLMs - present accessibility and scalability challenges. To address these, we propose a comprehensive blueprint that organizes RLM components into a modular framework, based on a survey and analysis of all RLM works. This blueprint incorporates diverse reasoning structures (chains, trees, graphs, and nested forms), reasoning strategies (e.g., Monte Carlo Tree Search, Beam Search), RL concepts (policy, value models and others), supervision schemes (Outcome-Based and Process-Based Supervision), and other related concepts (e.g., Test-Time Compute, Retrieval-Augmented Generation, agent tools). We also provide detailed mathematical formulations and algorithmic specifications to simplify RLM implementation. By showing how schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts fit as special cases, we demonstrate the blueprint's versatility and unifying potential. To illustrate its utility, we introduce x1, a modular implementation for rapid RLM prototyping and experimentation. Using x1 and a literature review, we provide key insights, such as multi-phase training for policy and value models, and the importance of familiar training distributions. Finally, we discuss scalable RLM cloud deployments and we outline how RLMs can integrate with a broader LLM ecosystem. Our work demystifies RLM construction, democratizes advanced reasoning capabilities, and fosters innovation, aiming to mitigate the gap between "rich AI" and "poor AI" by lowering barriers to RLM design and experimentation.