mlsanityData to 14 Oct 2026

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

arXiv 2610.02039 ↗First posted 1 Oct 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose Cancellation-Aware Response Masking (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to 3.13 percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by 2.88 points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 34 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

28 of 34 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Rethinking the Divergence Regularization in LLM RL8 Jun
  2. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards9 Feb
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning4 Feb
  4. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration2 Nov
  5. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  6. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  12. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
5 more
  1. 2024FullStack Bench: Evaluating LLMs as Full Stack Coderscited by 58
  2. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  3. 2024Simple Policy Optimizationcited by 32
  4. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learningcited by 16
  5. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mask-Aware Policy Gradients for Diffusion Language Models
  2. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  3. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  4. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
  5. 2026VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
  6. 2026Predictive Divergence Masks for LLM RL
  7. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
  8. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples