mlsanityData to 14 Oct 2026

On The Presence of Double-Descent in Deep Reinforcement Learning

arXiv 2511.06895 ↗First posted 10 Nov 2025cs.LG cs.AI stat.MLcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The double descent (DD) paradox, where over-parameterized models see generalization improve past the interpolation point, remains largely unexplored in the non-stationary domain of Deep Reinforcement Learning (DRL). We present preliminary evidence that DD exists in model-free DRL, investigating it systematically across varying model capacity using the Actor-Critic framework. We rely on an information-theoretic metric, Policy Entropy, to measure policy uncertainty throughout training. Preliminary results show a clear epoch-wise DD curve; the policy's entrance into the second descent region correlates with a sustained, significant reduction in Policy Entropy. This entropic decay suggests that over-parameterization acts as an implicit regularizer, guiding the policy towards robust, flatter minima in the loss landscape. These findings establish DD as a factor in DRL and provide an information-based mechanism for designing agents that are more general, transferable, and robust.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 6 of its 8 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

6 of 8 references.

References · 6

  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  3. 2018Generalization and Regularization in DQNcited by 118
  4. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 65
  5. 2020A Brief Prehistory of Double Descentcited by 44
  6. 2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavior-Consistent Deep Reinforcement Learning
  2. 2025A Dual-Agent Adversarial Framework for Robust Generalization in Deep Reinforcement Learning
  3. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  4. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space
  5. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  6. 2025Mirror Descent Actor Critic via Bounded Advantage Learning
  7. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Games
  8. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm