mlsanityData to 14 Oct 2026

Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

arXiv 2010.14498 ↗First posted 27 Oct 2020cs.LG stat.MLcited by 134

Not on the map. Why?

Cited by 54 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We identify an implicit under-parameterization phenomenon in value-based deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network. We characterize this loss of expressivity via a drop in the rank of the learned value network features, and show that this typically corresponds to a performance drop. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings. We formally analyze this phenomenon and show that it results from a pathological interaction between bootstrapping and gradient-based optimization. We further show that mitigating implicit under-parameterization by controlling rank collapse can improve performance.

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 19; 2023: 15; 2024: 35; 2025: 31; 2026: 23.

134 papers.

Sort:

By area:

  1. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  2. 2024One Step Diffusion via Shortcut Modelscited by 391
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  5. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  6. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  7. 2023Understanding plasticity in neural networkscited by 174
  8. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  9. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  10. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  11. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  12. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
38 more
  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  2. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  3. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  4. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  5. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  6. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  7. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  8. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  9. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  10. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  11. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  12. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  13. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 57
  14. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  15. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  16. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  17. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  18. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  19. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  20. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  21. 2023Small batch deep reinforcement learningcited by 36
  22. 2024Learning Continually by Spectral Regularizationcited by 34
  23. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  24. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  25. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 30
  26. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  27. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 28
  28. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  29. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  30. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  31. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  32. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  33. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  34. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  35. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  36. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 19
  37. 2022When, where, and how to add new neurons to ANNscited by 18
  38. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 18

50 most cited of 134 All 134 (JSON)