mlsanityData to 14 Oct 2026

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

arXiv 2510.18855 ↗First posted 21 Oct 2025cs.CL cs.AIcited by 55

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

We present Ring-1T, the first open-source, state-of-the-art thinking model with a trillion-scale parameter. It features 1 trillion total parameters and activates approximately 50 billion per token. Training such models at a trillion-parameter scale introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system. To address these, we pioneer three interconnected innovations: (1) IcePop stabilizes RL training via token-level discrepancy masking and clipping, resolving instability from training-inference mismatches; (2) C3PO++ improves resource utilization for long rollouts under a token budget by dynamically partitioning them, thereby obtaining high time efficiency; and (3) ASystem, a high-performance RL framework designed to overcome the systemic bottlenecks that impede trillion-parameter model training. Ring-1T delivers breakthrough results across critical benchmarks: 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-1. Notably, it attains a silver medal-level result on the IMO-2025, underscoring its exceptional reasoning capabilities. By releasing the complete 1T parameter MoE model to the community, we provide the research community with direct access to cutting-edge reasoning capabilities. This contribution marks a significant milestone in democratizing large-scale reasoning intelligence and establishes a new baseline for open-source model performance.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 53 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

31 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Truncated Proximal Policy Optimization18 Jun
  2. 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs17 Jun
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  6. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May

Other references · 25

  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  11. 2025Kimi K2: Open Agentic Intelligencecited by 567
  12. 2025Humanity's Last Examcited by 491
13 more
  1. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  2. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  3. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  4. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  5. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  6. 2025BIG-Bench Extra Hardcited by 125
  7. 2024ARC Prize 2024: Technical Reportcited by 104
  8. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 61
  9. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  10. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 28
  11. 2025AWorld: Orchestrating the Training Recipe for Agentic AIcited by 17
  12. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  13. 2025FlexLink: Boosting your NVLink Bandwidth by 27% without accuracy concerncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 51.

55 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  2. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  3. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  4. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  5. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  6. 2026UI-Venus-1.5 Technical Reportcited by 41
  7. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  8. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 37
  9. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  10. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 29
  11. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 25
  12. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
38 more
  1. 2026ERNIE 5.0 Technical Reportcited by 11
  2. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 10
  3. 2026Training AI Scientists to Replicate Researchcited by 9
  4. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 7
  5. 2026GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignmentcited by 6
  6. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  7. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  8. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMscited by 6
  9. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  10. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
  11. 2026Rethinking the Divergence Regularization in LLM RLcited by 5
  12. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 5
  13. 2026STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamicscited by 4
  14. 2026P^2O: Joint Policy and Prompt Optimizationcited by 3
  15. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Informationcited by 3
  16. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  17. 2025Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoEcited by 2
  18. 2025Sigma-MoE-Tiny Technical Reportcited by 2
  19. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Taskscited by 2
  20. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 2
  21. 2026UI-Venus-2 Technical Reportcited by 2
  22. 2026Target Policy Optimizationcited by 1
  23. 2026MinT: Managed Infrastructure for Training and Serving Millions of LLMscited by 1
  24. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewardscited by 1
  25. 2026Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRAcited by 1
  26. 2026An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offscited by 0
  27. 2026Beyond Distribution Sharpening: The Importance of Task Rewardscited by 0
  28. 2026Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scalingcited by 0
  29. 2026Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancycited by 0
  30. 2026Predictive Divergence Masks for LLM RLcited by 0
  31. 2026Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoningcited by 0
  32. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  33. 2026Motif 3: Technical Reportcited by 0
  34. 2026Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shapingcited by 0
  35. 2026SciLENS: RL-Driven Autonomous Agents for Scientific Localized Evidence Navigation and Synthesiscited by 0
  36. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  37. 2026Bellman Policy Optimizationcited by 0
  38. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  2. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  3. 2026CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation
  4. 2026Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming
  5. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  6. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  7. 2025Laminar: A Scalable Asynchronous RL Post-Training Framework
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale