mlsanityData to 7 Oct 2026

Trust Region Masking for Long-Horizon LLM Reinforcement Learning

arXiv 2512.23075 ↗First posted 28 Dec 2025cs.LG cs.AI cs.ITcited by 13

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods for Large Language Models optimize a policy πθ via a surrogate objective computed from samples of a rollout policy πroll. However, modern LLM-RL pipelines suffer from unavoidable implementation divergences -- backend discrepancies, Mixture-of-Experts routing discontinuities, and distributed training staleness -- causing off-policy mismatch (πroll neq πθ) and approximation errors between the surrogate and the true objective. We demonstrate that classical trust region bounds on this error scale as O(T2) with sequence length T, rendering them vacuous for long-horizon tasks. To address this, we derive a family of bounds -- both KL-based and TV-based -- including a Pinsker-Marginal bound (O(T3/2)), a Mixed bound (O(T)), and an Adaptive bound that strictly generalizes the Pinsker-Marginal bound via per-position importance-ratio decomposition. Taking the minimum over all bounds yields the tightest known guarantee across all divergence regimes. Crucially, all bounds depend on the maximum token-level divergence DKLtok,max (or DTVtok,max), a sequence-level quantity that cannot be controlled by token-independent methods like PPO clipping. We propose Trust Region Masking (TRM), which masks entire sequences violating the trust region, enabling the first non-vacuous monotonic improvement guarantees for long-horizon LLM-RL.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 28 Dec 2025.

Builds on

20 of 37 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
8 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  5. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  7. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  8. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RLcited by 5
  2. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 5
  3. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  4. 2026Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RLcited by 3
  5. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learningcited by 3
  6. 2026Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancycited by 0
  7. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 0
  8. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  9. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  10. 2026Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?cited by 0
  11. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agentscited by 0
  12. 2026Rethinking Training-Inference Mismatch in LLM Reinforcement Learning: Where It Arises and How to Correct Itcited by 0
1 more
  1. 2026CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking the Divergence Regularization in LLM RL
  2. 2026Extreme Region Policy Distillation
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  4. 2026Predictive Divergence Masks for LLM RL
  5. 2025Trust-Region Adaptive Policy Optimization
  6. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
  7. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
  8. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization