# Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2108.10533 · HTML: https://mlsanity.com/paper/2108.10533/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [RLlib: Abstractions for Distributed Reinforcement Learning](https://mlsanity.com/paper/1712.09381/) (2017)
- [Maximum Entropy-Regularized Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1905.08786/) (2019)
- [Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning](https://mlsanity.com/paper/1909.03198/) (2019)
- [Efficient Entropy for Policy Gradient with Multidimensional Action Space](https://mlsanity.com/paper/1806.00589/) (2018)
- [Neural Network Based Reinforcement Learning for Audio-Visual Gaze Control in Human-Robot Interaction](https://mlsanity.com/paper/1711.06834/) (2017)
- [Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning](https://mlsanity.com/paper/1911.00828/) (2019)

## Built on by (most cited first, 1 of 1)

- [RLocator: Reinforcement Learning for Bug Localization](https://mlsanity.com/paper/2305.05586/) (2023)

## Abstract (arXiv)

Encouraging exploration is a critical issue in deep reinforcement learning. We investigate the effect of initial entropy that significantly influences the exploration, especially at the earlier stage. Our main observations are as follows: 1) low initial entropy increases the probability of learning failure, and 2) this initial entropy is biased towards a low value that inhibits exploration. Inspired by the investigations, we devise entropy-aware model initialization, a simple yet powerful learning strategy for effective exploration. We show that the devised learning strategy significantly reduces learning failures and enhances performance, stability, and learning speed through experiments.