mlsanityData to 14 Oct 2026

Understanding Enrichment in Reinforcement Learning

arXiv 2610.02846 ↗First posted 2 Oct 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: RL training of language and agent models

Abstract (arXiv)

When rewards are sparse, reinforcement learning with verifiable rewards (RLVR) often uses hints or intermediate guidance to generate more successful rollouts. This enrichment biases policy-gradient updates unless corrected via importance weights, but existing methods omit correction or truncate importance weights in order to avoid the high variance of correction. It thus remains unclear what exactly is gained or lost in RLVR by correcting enriched rollouts. We show, mathematically, that omitted or truncated correction implicitly reweights the defined reward, and we decompose the resulting gradient error into scale, rotation, and variance to explain their distinct effects on learning. To make correction practical, we develop a novel sequential Monte Carlo (SMC) weight correction mechanism that, under stability and particle-order assumptions, tempers the exponential compounding of standard correction variance over the length of a sample to an additive accumulation. We then apply our analysis of enrichment to interpreting the results of a fine-tune of Qwen3-1.7B on a sparse band of OpenMathReasoning, establishing a concrete mechanism of how enrichment, both corrected and uncorrected, helps avoid collapse in sparse domains. Our main contribution is to understand, in general, how enrichment and correction can affect training, as opposed to claiming that either mode of operation is superior to unenriched RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 38 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

29 of 38 references.

References · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  10. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  11. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  12. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
17 more
  1. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  2. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 63
  3. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  4. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoningcited by 52
  5. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  6. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  7. 2025Learning to Reason at the Frontier of Learnabilitycited by 23
  8. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  9. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  10. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  11. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  12. 2024Policy Gradient with Active Importance Samplingcited by 10
  13. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspectivecited by 7
  14. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  15. 2025LLM-CoT Enhanced Graph Neural Recommendation with Harmonized Group Policy Optimizationcited by 6
  16. 2026Process Reward Informed Tree Rollout for Effective Multi-Turn RLcited by 2
  17. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVRcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  2. 2026When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
  3. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
  4. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
  5. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  6. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
  7. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
  8. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards