mlsanityData to 14 Oct 2026

PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization

arXiv 2607.16206 ↗First posted 8 May 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper introduces PPO-HSC (Proximal Policy Optimization with High-order Sampling Coverage), an exploratory reinforcement learning framework designed to address the "Invisible Shackles" of mode collapse in Large Language Model (LLM) fine-tuning. While standard Reinforcement Learning from Verifiable Rewards (RLVR) effectively reinforces high-reward trajectories, it often leads models to over-optimize known solutions, sacrificing curiosity and the ability to explore broader solution manifolds. To overcome this, PPO-HSC incorporates a High-order Sampling Coverage (HSC) reward that incentivizes the discovery of "low-similarity yet high-validity" reasoning patterns. By maintaining a dynamic trajectory library of verified unique solutions, the framework provides a differentiable signal that rewards semantic novelty while ensuring structural rationality through a plausibility constraint. Empirical evaluations on mathematical reasoning (GSM8K, SVAMP) and code generation tasks demonstrate that PPO-HSC significantly enhances solution diversity and state-space coverage while maintaining or surpassing the accuracy and syntax integrity of state-of-the-art RL baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 6 of its 10 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

6 of 10 references.

References · 6

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  5. 2015Illuminating search spaces by mapping elitescited by 425
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
  2. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards
  3. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  4. 2025Truncated Proximal Policy Optimization
  5. 2026TPPO: Tracking-Value PPO For Language Model Reasoning
  6. 2026DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning
  7. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  8. 2026Poly-EPO: Training Exploratory Reasoning Models