mlsanityData to 14 Oct 2026

Exploration-Preserving Policy Optimization

arXiv 2610.04011 ↗First posted 2 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

35 of 41 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
23 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  6. 2018Understanding the impact of entropy on policy optimizationcited by 176
  7. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  8. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  9. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  10. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  11. 2026Maximum Likelihood Reinforcement Learningcited by 36
  12. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  13. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
  14. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learningcited by 11
  15. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 11
  16. 2026Entropy-Preserving Reinforcement Learningcited by 11
  17. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 9
  18. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  19. 2024Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimationcited by 4
  20. 2026Rotation-Preserving Supervised Fine-Tuningcited by 3
  21. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 3
  22. 2026Group Entropy-Controlled Policy Optimizationcited by 3
  23. 2026When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPOcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
  2. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  3. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning
  4. 2026P^2O: Joint Policy and Prompt Optimization
  5. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
  6. 2026expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling
  7. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  8. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoning