mlsanityData to 14 Oct 2026

T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling

arXiv 2501.11651 ↗First posted 20 Jan 2025cs.LG cs.CLcited by 77

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 41 of its 45 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

41 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2024Qwen2.5 Technical Reportcited by 4,942
  12. 2023Let's Verify Step by Stepcited by 3,789
27 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  13. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  14. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  15. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  16. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  17. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  18. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  19. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  20. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  21. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  22. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  23. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  24. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  25. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
  26. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  27. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 59; 2026: 18.

77 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2025s1: Simple test-time scalingcited by 1,341
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  5. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  6. 2025LIMR: Less is More for RL Scalingcited by 139
  7. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  8. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  9. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  10. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 104
  11. 2025S*: Test Time Scaling for Code Generationcited by 98
  12. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
38 more
  1. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  2. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  3. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  4. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  5. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 55
  6. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  7. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  8. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  9. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  10. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  11. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  12. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 26
  13. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  14. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22
  15. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  16. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectivenesscited by 20
  17. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  18. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  19. 2025An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraintcited by 17
  20. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  21. 2025GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Trainingcited by 14
  22. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 14
  23. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14
  24. 2026CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agentscited by 14
  25. 2025Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Onescited by 13
  26. 2026Reasoning over Semantic IDs Enhances Generative Recommendationcited by 13
  27. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  28. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12
  29. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  30. 2025Temporal Consistency for LLM Reasoning Process Error Identificationcited by 10
  31. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  32. 2025Reinforcement Learning Teachers of Test Time Scalingcited by 10
  33. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 10
  34. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  35. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  36. 2025Test-time Scaling Techniques in Theoretical Physics -- A Comparison of Methods on the TPBench Datasetcited by 8
  37. 2026UniT: Unified Multimodal Chain-of-Thought Test-time Scalingcited by 8
  38. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
  3. 2025Training Language Models to Reason Efficiently
  4. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking
  5. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs
  6. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  8. 2025Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models