mlsanityData to 14 Oct 2026

Exploration Behavior of Untrained Policies

arXiv 2506.22566 ↗First posted 27 Jun 2025cs.LG cs.AI stat.MLcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Exploration remains a fundamental challenge in reinforcement learning (RL), particularly in environments with sparse or adversarial reward structures. In this work, we study how the architecture of deep neural policies implicitly shapes exploration before training. We theoretically and empirically demonstrate strategies for generating ballistic or diffusive trajectories from untrained policies in a toy model. Using the theory of infinite-width networks and a continuous-time limit, we show that untrained policies return correlated actions and result in non-trivial state-visitation distributions. We discuss the distributions of the corresponding trajectories for a standard architecture, revealing insights into inductive biases for tackling exploration. Our results establish a theoretical and experimental framework for using policy initialization as a design tool to understand exploration behavior in early training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 21 references that are in the map. First posted on arXiv 27 Jun 2025.

Builds on

16 of 21 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2016Deep Exploration via Bootstrapped DQNcited by 527
  8. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  11. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  12. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
4 more
  1. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  2. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  3. 2021Periodic Activation Functions Induce Stationaritycited by 21
  4. 2021LipBaB: Computing exact Lipschitz constant of ReLU networkscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
  2. 2024Trajectory-Oriented Policy Optimization with Sparse Rewards
  3. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learning
  4. 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
  5. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
  6. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space
  7. 2026Rethinking Plasticity in Deep Reinforcement Learning
  8. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning