mlsanityData to 14 Oct 2026

Marginal Coverage Credit Reduces Redundant Exploration in Parallel State-Entropy Optimization

arXiv 2608.27507 ↗First posted 27 Aug 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy Gradient for Parallel State Entropy maximization (PGPSE) expands state-space coverage by training independently parameterized policies in replicated copies of the same environment. However, its pooled team-entropy score measures only collective exploration and cannot identify policies that contribute non-redundant coverage. We introduce Marginal Coverage Credit for PGPSE (MCC-PGPSE), which combines leave-one-policy-out coverage with state-owner specialization to estimate policy-specific credit. MCC-PGPSE preserves PGPSE's pooled objective and redistributes non-negative auxiliary intrinsic rewards according to these credits without changing their total mass. This redistribution is designed to discourage redundant visitation and promote complementary coverage. We evaluated MCC-PGPSE in controlled environments, seven public discrete-state benchmarks, and representative Room and Maze settings from the original PGPSE protocol. Across all tested settings, MCC-PGPSE produced positive final window gains in normalized team state entropy and state support over the Entropy baseline. Controlled-task comparisons and the fixed-suite public aggregate were significant, whereas five-seed original-protocol comparisons were directionally consistent. Ablations and credit alignment controls indicate that most gains arise from leave-one-policy-out coverage rather than non-uniform weighting, mismatched credit, or neural novelty alone. These results support contribution-conditioned auxiliary reward allocation as an interpretable approach to improving complementary coverage among parallel policies in discrete state spaces.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 32 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

23 of 32 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  12. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
11 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2016Deep Exploration via Bootstrapped DQNcited by 527
  3. 2017Population Based Training of Neural Networkscited by 367
  4. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  5. 2019Self-Supervised Exploration via Disagreementcited by 265
  6. 2017Count-Based Exploration with Neural Density Modelscited by 257
  7. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  8. 2017Parameter Space Noise for Explorationcited by 161
  9. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  10. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 73
  11. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Global Optimality for Constrained Exploration via Penalty Regularization
  2. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
  3. 2026OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Search
  4. 2026Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off
  5. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agents
  6. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
  7. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
  8. 2026Targeted Exploration via Unified Entropy Control for Reinforcement Learning