mlsanityData to 14 Oct 2026

Reasoning Language Models: A Blueprint

arXiv 2501.11223 ↗First posted 20 Jan 2025cs.AI cs.CLcited by 42

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Reasoning language models (RLMs), also known as Large Reasoning Models (LRMs), such as OpenAI's o1 and o3, DeepSeek-R1, and Alibaba's QwQ, have redefined AI's problem-solving capabilities by extending LLMs with advanced reasoning mechanisms. Yet, their high costs, proprietary nature, and complex architectures - uniquely combining reinforcement learning (RL), search heuristics, and LLMs - present accessibility and scalability challenges. To address these, we propose a comprehensive blueprint that organizes RLM components into a modular framework, based on a survey and analysis of all RLM works. This blueprint incorporates diverse reasoning structures (chains, trees, graphs, and nested forms), reasoning strategies (e.g., Monte Carlo Tree Search, Beam Search), RL concepts (policy, value models and others), supervision schemes (Outcome-Based and Process-Based Supervision), and other related concepts (e.g., Test-Time Compute, Retrieval-Augmented Generation, agent tools). We also provide detailed mathematical formulations and algorithmic specifications to simplify RLM implementation. By showing how schemes like LLaMA-Berry, QwQ, Journey Learning, and Graph of Thoughts fit as special cases, we demonstrate the blueprint's versatility and unifying potential. To illustrate its utility, we introduce x1, a modular implementation for rapid RLM prototyping and experimentation. Using x1 and a literature review, we provide key insights, such as multi-phase training for policy and value models, and the importance of familiar training distributions. Finally, we discuss scalable RLM cloud deployments and we outline how RLMs can integrate with a broader LLM ecosystem. Our work demystifies RLM construction, democratizes advanced reasoning capabilities, and fosters innovation, aiming to mitigate the gap between "rich AI" and "poor AI" by lowering barriers to RLM design and experimentation.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 191 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

38 of 191 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeek-V3 Technical Report27 Dec
  2. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models12 Oct
  3. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  8. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  9. 2022Self-critiquing models for assisting human evaluatorscited by 380
  10. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  11. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  12. 2024Free Process Rewards without Process Labelscited by 165
23 more
  1. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 158
  2. 2019On the Measure of Intelligencecited by 144
  3. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  4. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  5. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  6. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  7. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  8. 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoningcited by 110
  9. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  10. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  11. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  12. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  13. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 55
  14. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  15. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  16. 2024Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTScited by 43
  17. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 30
  18. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  19. 2024Multi-Head RAG: Solving Multi-Aspect Problems with LLMscited by 18
  20. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  21. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  22. 2024CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Taskscited by 10
  23. 2025Understanding Before Reasoning: Enhancing Chain-of-Thought with Iterative Summarization Pre-Promptingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 13.

42 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  5. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 43
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  7. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  8. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 17
  9. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  10. 2025CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learningcited by 15
  11. 2025Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Modelscited by 14
  12. 2025When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Modelscited by 13
30 more
  1. 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradientscited by 8
  2. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  3. 2025Schoenfeld's Anatomy of Mathematical Reasoning by Language Modelscited by 8
  4. 2025Affordable AI Assistants with Knowledge Graph of Thoughtscited by 6
  5. 2025Can Large Language Models Adequately Perform Symbolic Reasoning Over Time Series?cited by 6
  6. 2025Children's Mental Models of AI Reasoning: Implications for AI Literacy Educationcited by 5
  7. 2025A Study on Thinking Patterns of Large Reasoning Models in Code Generationcited by 5
  8. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  9. 2025Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledgecited by 3
  10. 2026Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoningcited by 3
  11. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 3
  12. 2025One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligencecited by 2
  13. 2025Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineeringcited by 2
  14. 2026Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Predictioncited by 2
  15. 2025BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Modelscited by 1
  16. 2026GraphSeek: Next-Generation Graph Analytics with LLMscited by 1
  17. 2026A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2cited by 1
  18. 2025Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Modelscited by 0
  19. 2025Initial Steps in Integrating Large Reasoning and Action Models for Service Compositioncited by 0
  20. 2025Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCIcited by 0
  21. 2025Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensicscited by 0
  22. 2025rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injectioncited by 0
  23. 2026The Critical Horizon: Inspection Design Principles for Multi-Stage Operations and Deep Reasoningcited by 0
  24. 2026Characterize Then Distill: Mechanistic Reasoning in Large Output Spacescited by 0
  25. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  26. 2026Maia 200: A Software Defined Dataflow System for Large-scale AI Accelerationcited by 0
  27. 2026AI Models Can Predict and Collaboratively Modulate Human Memory Searchcited by 0
  28. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0
  29. 2026Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layercited by 0
  30. 2026When2Think: Learning Difficulty-Aware Length Control for Efficient Hybrid Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  2. 2025Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMs
  3. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
  5. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  6. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches
  7. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
  8. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs