mlsanityData to 7 Oct 2026

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

arXiv 2502.02508 ↗First posted 4 Feb 2025cs.CL cs.AIcited by 53

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verifier, resulting in a two-player system. Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks. Thus, we pose a new research problem: Can we internalize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM? This work explores an orthogonal direction focusing on post-training LLMs for autoregressive searching (i.e., an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our approach results in Satori, a 7B LLM trained on open-source models and data. Extensive empirical evaluations demonstrate that Satori achieves state-of-the-art performance on mathematical reasoning benchmarks while exhibits strong generalization to out-of-domain tasks. Code, data, and models are fully open-sourced.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 100 references that are in the map. First posted on arXiv 4 Feb 2025.

Builds on

41 of 100 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024Free Process Rewards without Process Labels2 Dec
  3. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  4. 2024Progress or Regress? Self-Improvement Reversal in Post-training6 Jul
  5. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  6. 2023Let's Verify Step by Step31 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2024Phi-4 Technical Reportcited by 675
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  12. 2025Humanity's Last Examcited by 467
22 more
  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2022Self-critiquing models for assisting human evaluatorscited by 372
  3. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  4. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  8. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  9. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  10. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  11. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 142
  12. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  13. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  14. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  15. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 90
  16. 2018Kickstarting Deep Reinforcement Learningcited by 75
  17. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  18. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 54
  19. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  20. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 25
  21. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  22. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 20.

53 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  5. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  6. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  7. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  8. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  9. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  10. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  11. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 39
  12. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
38 more
  1. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  2. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  3. 2026HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systemscited by 20
  4. 2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineeringcited by 16
  5. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  6. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 13
  7. 2025AdapThink: Adaptive Thinking Preferences for Reasoning Language Modelcited by 11
  8. 2025Reinforcement Learning Teachers of Test Time Scalingcited by 10
  9. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  10. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
  11. 2025Do LLM Agents Know How to Ground, Recover, and Assess? A Benchmark for Epistemic Competence in Information-Seeking Agentscited by 7
  12. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  13. 2025Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learningcited by 6
  14. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RLcited by 5
  15. 2025Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Gamescited by 4
  16. 2025A Survey of LLM × DATAcited by 4
  17. 2025Learning to Ponder: Adaptive Reasoning in Latent Spacecited by 4
  18. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  19. 2026What If We Allocate Test-Time Compute Adaptively?cited by 4
  20. 2026GSRM: Generative Speech Reward Model for Speech RLHFcited by 4
  21. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  22. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  23. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  24. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  25. 2026SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learningcited by 2
  26. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  27. 2026ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learningcited by 2
  28. 2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
  29. 2025ScRPO: From Errors to Insightscited by 1
  30. 2026Asking the Right Questions: Improving Reasoning with Generated Stepping Stonescited by 1
  31. 2026Confidence-Aware Alignment Makes Reasoning LLMs More Reliablecited by 1
  32. 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoningcited by 1
  33. 2026Efficient Agentic Reasoning Through Self-Regulated Simulative Planningcited by 1
  34. 2026COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learningcited by 1
  35. 2025Reasoning without Regretcited by 0
  36. 2025Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoningcited by 0
  37. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  38. 2026SW-A2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Webcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learning
  2. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  3. 2025Rectifying LLM Thought from Lens of Optimization
  4. 2026Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
  5. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
  6. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs
  8. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning