mlsanityData to 7 Oct 2026

Dynamic Bottleneck for Robust Self-Supervised Exploration

arXiv 2110.10735 ↗First posted 20 Oct 2021cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Exploration methods based on pseudo-count of transitions or curiosity of dynamics have achieved promising results in solving reinforcement learning with sparse rewards. However, such methods are usually sensitive to environmental dynamics-irrelevant information, e.g., white-noise. To handle such dynamics-irrelevant information, we propose a Dynamic Bottleneck (DB) model, which attains a dynamics-relevant representation based on the information-bottleneck principle. Based on the DB model, we further propose DB-bonus, which encourages the agent to explore state-action pairs with high information gain. We establish theoretical connections between the proposed DB-bonus, the upper confidence bound (UCB) for linear case, and the visiting count for tabular case. We evaluate the proposed method on Atari suits with dynamics-irrelevant noises. Our experiments show that exploration with DB bonus outperforms several state-of-the-art exploration methods in noisy environments.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

45 of 61 references.

References · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2016Deep Variational Information Bottleneckcited by 915
  11. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
33 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  3. 2016Deep Exploration via Bootstrapped DQNcited by 518
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  5. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  6. 2018Is Q-learning Provably Efficient?cited by 485
  7. 2020Reinforcement Learning with Augmented Datacited by 452
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  9. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  12. 2019Self-Supervised Exploration via Disagreementcited by 263
  13. 2017Count-Based Exploration with Neural Density Modelscited by 255
  14. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  15. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  16. 2021Reinforcement Learning with Prototypical Representationscited by 179
  17. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  18. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  19. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  20. 2021APS: Active Pretraining with Successor Featurescited by 109
  21. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  22. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  23. 2020Deep Reinforcement and InfoMax Learningcited by 79
  24. 2018Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Controlcited by 54
  25. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  26. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
  27. 2020BeBold: Exploration Beyond the Boundary of Explored Regionscited by 32
  28. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  29. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  30. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  31. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25
  32. 2020Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learningcited by 9
  33. 2019Implicit Generative Modeling for Efficient Explorationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 6; 2025: 4; 2026: 1.

21 papers.

Sort:

By area:

  1. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  2. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  3. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  4. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  5. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  6. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
  7. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10
  8. 2024Simplifying Latent Dynamics with Softly State-Invariant World Modelscited by 10
  9. 2025Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulationcited by 10
  10. 2023Bridging the Sim-to-Real Gap from the Information Bottleneck Perspectivecited by 8
  11. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  12. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
9 more
  1. 2024Effective Exploration Based on the Structural Information Principlescited by 4
  2. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  3. 2025Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibrationcited by 4
  4. 2022CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Controlcited by 1
  5. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  6. 2022Rollout Total Correlation for Deep Reinforcement Learningcited by 0
  7. 2025Maximum Total Correlation Reinforcement Learningcited by 0
  8. 2025Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Predictioncited by 0
  9. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0