mlsanityData to 7 Oct 2026

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

arXiv 2609.09783 ↗First posted 9 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Asynchronous reinforcement learning has become the standard way to scale training for large language models (LLM), but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE delivers a 9.8% relative improvement in mean@1 on BrowseComp-Plus over the strongest baseline, runs 2.46× faster per step than synchronous training, and remains stable 50 updates off-policy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

37 of 50 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  11. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  12. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
25 more
  1. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  2. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  3. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 49
  4. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  5. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
  6. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  7. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  8. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learningcited by 13
  9. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 13
  10. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  11. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  12. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  13. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  14. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 7
  15. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
  16. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
  17. 2026Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Trainingcited by 5
  18. 2026Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learningcited by 5
  19. 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVRcited by 4
  20. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 4
  21. 2026Clipping-Free Policy Optimization for Large Language Modelscited by 3
  22. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learningcited by 3
  23. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3
  24. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Informationcited by 2
  25. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  2. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
  4. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
  5. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  6. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  7. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  8. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples