mlsanityData to 7 Oct 2026

DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay

arXiv 2603.16157 ↗First posted 17 Mar 2026cs.LG cs.AIcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

While Reinforcement Learning (RL) enhances Large Language Model reasoning, on-policy algorithms like GRPO are sample-inefficient as they discard past rollouts. Existing experience replay methods address this by reusing accurate samples for direct policy updates, but this often incurs high computational costs and causes mode collapse via overfitting. We argue that historical data should prioritize sustaining diversity rather than simply reinforcing accuracy. To this end, we propose Dynamic Jensen-Shannon Replay (DyJR), a simple yet effective regularization framework using a dynamic reference distribution from recent trajectories. DyJR introduces two innovations: (1) A Time-Sensitive Dynamic Buffer that uses FIFO and adaptive sizing to retain only temporally proximal samples, synchronizing with model evolution; and (2) Jensen-Shannon Divergence Regularization, which replaces direct gradient updates with a distributional constraint to prevent diversity collapse. Experiments on mathematical reasoning and Text-to-SQL benchmarks demonstrate that DyJR significantly outperforms GRPO as well as baselines such as RLEP and Ex-GRPO, while maintaining training efficiency comparable to the original GRPO. Furthermore, from the perspective of Rank-k token probability evolution, we show that DyJR enhances diversity and mitigates over-reliance on Rank-1 tokens, elucidating how specific sub-modules of DyJR influence the training dynamics.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 37 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

32 of 37 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction13 Feb
  2. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  3. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration4 Oct
  4. 2025ExGRPO: Learning to Reason from Experience2 Oct
  5. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  6. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model9 Jul
  7. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  8. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  9. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  10. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  11. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  12. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
2 more
  1. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  8. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  9. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  10. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  11. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  12. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
6 more
  1. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  2. 2025MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosiscited by 32
  3. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  4. 2025Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policiescited by 22
  5. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  6. 2024How Do Humans Write Code? Large Models Do It the Same Way Toocited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  2. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  3. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  2. 2026Entropy-Preserving Reinforcement Learning
  3. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  4. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
  5. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  6. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  7. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
  8. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay