mlsanityData to 14 Oct 2026

Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness

arXiv 2105.09992 ↗First posted 20 May 2021cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Sparse rewards are double-edged training signals in reinforcement learning: easy to design but hard to optimize. Intrinsic motivation guidances have thus been developed toward alleviating the resulting exploration problem. They usually incentivize agents to look for new states through novelty signals. Yet, such methods encourage exhaustive exploration of the state space rather than focusing on the environment's salient interaction opportunities. We propose a new exploration method, called Don't Do What Doesn't Matter (DoWhaM), shifting the emphasis from state novelty to state with relevant actions. While most actions consistently change the state when used, textite.g. moving the agent, some actions are only effective in specific states, textite.g., opening a door, grabbing an object. DoWhaM detects and rewards actions that seldom affect the environment. We evaluate DoWhaM on the procedurally-generated environment MiniGrid, against state-of-the-art methods and show that DoWhaM greatly reduces sample complexity.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

45 of 78 references.

References · 45

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Deep Exploration via Bootstrapped DQNcited by 527
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
33 more
  1. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  2. 2016Variational Intrinsic Controlcited by 267
  3. 2017Noisy Networks for Explorationcited by 266
  4. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  5. 2017Count-Based Exploration with Neural Density Modelscited by 257
  6. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  7. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  9. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  10. 2017Parameter Space Noise for Explorationcited by 161
  11. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  12. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  13. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  14. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  15. 2018Count-Based Exploration with the Successor Representationcited by 112
  16. 2018Episodic Curiosity through Reachabilitycited by 109
  17. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  18. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  19. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  20. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 83
  21. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  22. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
  23. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  24. 2021Adversarially Guided Actor-Criticcited by 47
  25. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  26. 2018Learning to Play with Intrinsically-Motivated Self-Aware Agentscited by 35
  27. 2020BeBold: Exploration Beyond the Boundary of Explored Regionscited by 33
  28. 2017Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learningcited by 15
  29. 2018Computational Theories of Curiosity-Driven Learningcited by 11
  30. 2019Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Explorationcited by 11
  31. 2014Kalman Temporal Differencescited by 10
  32. 2020Show me the Way: Intrinsic Motivation from Demonstrationscited by 7
  33. 2019MULEX: Disentangling Exploitation from Exploration in Deep RLcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 2.

7 papers.

  1. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  2. 2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
  3. 2022An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environmentscited by 8
  4. 2023Learning to Solve Tasks with Exploring Prior Behaviourscited by 4
  5. 2023Cyclophobic Reinforcement Learningcited by 1
  6. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  7. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0