# Dynamic Bottleneck for Robust Self-Supervised Exploration

arXiv: https://arxiv.org/abs/2110.10735 · HTML: https://mlsanity.com/paper/2110.10735/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 61 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 30 of 45)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Provably Efficient Exploration in Policy Optimization](https://mlsanity.com/paper/1912.05830/) (2019)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)

## Built on by (most cited first, 21 of 21)

- [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing](https://mlsanity.com/paper/2206.02829/) (2022)
- [Behavior Contrastive Learning for Unsupervised Skill Discovery](https://mlsanity.com/paper/2305.04477/) (2023)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Selective Visual Representations Improve Convergence and Generalization for Embodied AI](https://mlsanity.com/paper/2311.04193/) (2023)
- [Representation Learning in Deep RL via Discrete Information Bottleneck](https://mlsanity.com/paper/2212.13835/) (2022)
- [LECO: Learnable Episodic Count for Task-Specific Intrinsic Reward](https://mlsanity.com/paper/2210.05409/) (2022)
- [Ignorance is Bliss: Robust Control via Information Gating](https://mlsanity.com/paper/2303.06121/) (2023)
- [Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective](https://mlsanity.com/paper/2305.18464/) (2023)
- [Simplifying Latent Dynamics with Softly State-Invariant World Models](https://mlsanity.com/paper/2401.17835/) (2024)
- [Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation](https://mlsanity.com/paper/2502.02853/) (2025)
- [Constrained Ensemble Exploration for Unsupervised Skill Discovery](https://mlsanity.com/paper/2405.16030/) (2024)
- [Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning](https://mlsanity.com/paper/2404.19346/) (2024)
- [Effective Exploration Based on the Structural Information Principles](https://mlsanity.com/paper/2410.06621/) (2024)
- [Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors](https://mlsanity.com/paper/2410.17551/) (2024)
- [Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration](https://mlsanity.com/paper/2509.20648/) (2025)
- [CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Control](https://mlsanity.com/paper/2211.15205/) (2022)
- [Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2405.07223/) (2024)
- [Rollout Total Correlation for Deep Reinforcement Learning](https://mlsanity.com/paper/2209.05333/) (2022)
- [Maximum Total Correlation Reinforcement Learning](https://mlsanity.com/paper/2505.16734/) (2025)
- [Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction](https://mlsanity.com/paper/2511.03836/) (2025)
- [Revisiting TD Target Aggregation under Uncertainty in Q-Learning](https://mlsanity.com/paper/2608.03069/) (2026)

## Abstract (arXiv)

Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics-irrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.