# Contingency-Aware Exploration in Reinforcement Learning

arXiv: https://arxiv.org/abs/1811.01483 · HTML: https://mlsanity.com/paper/1811.01483/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 26 of 26)

- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Learning Markov State Abstractions for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04379/) (2021)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [A Max-Min Entropy Framework for Reinforcement Learning](https://mlsanity.com/paper/2106.10517/) (2021)
- [Principled Exploration via Optimistic Bootstrapping and Backward Induction](https://mlsanity.com/paper/2105.06022/) (2021)
- [Dynamic Bottleneck for Robust Self-Supervised Exploration](https://mlsanity.com/paper/2110.10735/) (2021)
- [Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration](https://mlsanity.com/paper/2305.18258/) (2023)
- [Causal Action Influence Aware Counterfactual Data Augmentation](https://mlsanity.com/paper/2405.18917/) (2024)
- [ELDEN: Exploration via Local Dependencies](https://mlsanity.com/paper/2310.08702/) (2023)
- [Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning](https://mlsanity.com/paper/2512.16911/) (2025)
- [Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RL](https://mlsanity.com/paper/2203.11369/) (2022)
- [Learning Action-based Representations Using Invariance](https://mlsanity.com/paper/2403.16369/) (2024)
- [Behavioral Exploration: Learning to Explore via In-Context Adaptation](https://mlsanity.com/paper/2507.09041/) (2025)
- [Reinforcement Learning with Exogenous States and Rewards](https://mlsanity.com/paper/2303.12957/) (2023)
- [Generative Adversarial Exploration for Reinforcement Learning](https://mlsanity.com/paper/2201.11685/) (2022)
- [Did I do that? Blame as a means to identify controlled effects in reinforcement learning](https://mlsanity.com/paper/2106.00266/) (2021)
- [Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2107.06405/) (2021)
- [Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach](https://mlsanity.com/paper/2501.19128/) (2025)
- [APEX: Autonomous Policy Exploration for Self-Evolving LLM Agents](https://mlsanity.com/paper/2605.21240/) (2026)
- [Contextual Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2301.01298/) (2023)
- [Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning](https://mlsanity.com/paper/2404.02545/) (2024)
- [Attacking and Defending Deep Reinforcement Learning Policies](https://mlsanity.com/paper/2205.07626/) (2022)
- [Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL](https://mlsanity.com/paper/2407.01570/) (2024)
- [Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators](https://mlsanity.com/paper/2607.16858/) (2026)

## Abstract (arXiv)

This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.