mlsanityData to 7 Oct 2026

Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning

arXiv 2607.22186 ↗First posted 24 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data destabilizes optimization and can cause policy collapse. Existing methods gate tokens by ratio magnitude alone, applying one threshold at every position. We show that the ratio's natural scale is set by token entropy, so deviations from mid-trajectory weight updates stay within this scale and carry genuine exploration. We further identify an overlooked low-entropy regime that breaks this scaling, where a near-zero probability amplifies train--inference mismatch into noise far beyond what the local entropy admits. A magnitude threshold admits this noise and discards the exploration. We therefore propose the Entropy-Normalized Trust Region (ENTR). Across long-horizon agentic tasks and mathematical reasoning benchmarks, ENTR outperforms existing asynchronous methods. It improves avg@1 on BrowseComp-Plus by 6.9% over the strongest baseline, trains stably up to 30 policy versions of staleness, and matches synchronous GRPO at a 2.6× speedup.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

34 of 46 references.

References · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  6. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  7. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  8. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  9. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  10. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  11. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  12. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
22 more
  1. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 49
  2. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  3. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  4. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
  5. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
  6. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 23
  7. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learningcited by 13
  8. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 13
  9. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  10. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  11. 2026StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Trainingcited by 9
  12. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  13. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  14. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 7
  15. 2026Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Trainingcited by 5
  16. 2026Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learningcited by 5
  17. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 4
  18. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoningcited by 3
  19. 2026Clipping-Free Policy Optimization for Large Language Modelscited by 3
  20. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learningcited by 3
  21. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Informationcited by 2
  22. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  2. 2026Extreme Region Policy Distillation
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
  4. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  5. 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
  6. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
  7. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
  8. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning