# More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences

arXiv: https://arxiv.org/abs/2110.10632 · HTML: https://mlsanity.com/paper/2110.10632/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [A Rewriting System for Convex Optimization Problems](https://mlsanity.com/paper/1709.04494/) (2017)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [Investigating Human Priors for Playing Video Games](https://mlsanity.com/paper/1802.10217/) (2018)
- [Geometric Entropic Exploration](https://mlsanity.com/paper/2101.02055/) (2021)
- [Growing Action Spaces](https://mlsanity.com/paper/1906.12266/) (2019)
- [The Natural Language of Actions](https://mlsanity.com/paper/1902.01119/) (2019)
- [There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning](https://mlsanity.com/paper/2106.04480/) (2021)
- [Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field](https://mlsanity.com/paper/1908.04683/) (2019)
- [Symmetry Learning for Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1706.02999/) (2017)
- [Online Abstraction with MDP Homomorphisms for Deep Learning](https://mlsanity.com/paper/1811.12929/) (2018)
- [Monte-Carlo Graph Search for AlphaZero](https://mlsanity.com/paper/2012.11045/) (2020)
- [On the Sensory Commutativity of Action Sequences for Embodied Agents](https://mlsanity.com/paper/2002.05630/) (2020)

## Abstract (arXiv)

Incorporating prior knowledge in reinforcement learning algorithms is mainly an open question. Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch. In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect. We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations. We show that this strategy can be computed at little cost, by solving a convex optimization problem. By replacing the usual epsilon-greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures.