Each Prompt Matters: Scaling Reinforcement Learning Without Wasting Rollouts on Hundred-Billion-Scale MoE
On the map: LLM reasoning → Reinforcement learning for reasoning
also in 1 more
Abstract (arXiv)
We present CompassMax-V3-Thinking, a hundred-billion-scale MoE reasoning model trained with a new RL framework built on one principle: each prompt must matter. Scaling RL to this size exposes critical inefficiencies-zero-variance prompts that waste rollouts, unstable importance sampling over long horizons, advantage inversion from standard reward models, and systemic bottlenecks in rollout processing. To overcome these challenges, we introduce several unified innovations: (1) Multi-Stage Zero-Variance Elimination, which filters out non-informative prompts and stabilizes group-based policy optimization (e.g. GRPO) by removing wasted rollouts; (2) ESPO, an entropy-adaptive optimization method that balances token-level and sequence-level importance sampling to maintain stable learning dynamics; (3) a Router Replay strategy that aligns training-time MoE router decisions with inference-time behavior to mitigate train-infer discrepancies, coupled with a reward model adjustment to prevent advantage inversion; (4) a high-throughput RL system with FP8-precision rollouts, overlapped reward computation, and length-aware scheduling to eliminate performance bottlenecks. Together, these contributions form a cohesive pipeline that makes RL on hundred-billion-scale MoE models stable and efficient. The resulting model delivers strong performance across both internal and public evaluations.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 23 references that are in the map. First posted on arXiv 8 Dec 2025.
Builds on
14 of 23 references.
Closest ancestors · 6
Same direction, newest first
- 2025ESPO: Entropy Importance Sampling Policy Optimization29 Nov
- 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
- 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
- 2025Geometric-Mean Policy Optimization28 Jul
- 2025RM-R1: Reward Modeling as Reasoning5 May
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
Other references · 8
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022Editing Models with Task Arithmeticcited by 1,076
- 2025Group Sequence Policy Optimizationcited by 788
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
- 2025ToolRL: Reward is All Tool Learning Needscited by 342
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
- 2025Defeating the Training-Inference Mismatch via FP16cited by 62
- 2025Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asiacited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
- 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
- 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
- 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
- 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
- 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
- 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
- 2026P^2O: Joint Policy and Prompt Optimization