# How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation

arXiv: https://arxiv.org/abs/2102.04399 · HTML: https://mlsanity.com/paper/2102.04399/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 8 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [Gotta Learn Fast: A New Benchmark for Generalization in RL](https://mlsanity.com/paper/1804.03720/) (2018)
- [Epistemic Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/1906.06273/) (2019)

## Built on by (most cited first, 27 of 27)

- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning](https://mlsanity.com/paper/2408.03029/) (2024)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [NAVIX: Scaling MiniGrid Environments with JAX](https://mlsanity.com/paper/2407.19396/) (2024)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [Approximate information state based convergence analysis of recurrent Q-learning](https://mlsanity.com/paper/2306.05991/) (2023)
- [Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environments](https://mlsanity.com/paper/2211.10515/) (2022)
- [Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration](https://mlsanity.com/paper/2509.20648/) (2025)
- [A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty](https://mlsanity.com/paper/2401.02914/) (2024)
- [Disentangling Uncertainties by Learning Compressed Data Representation](https://mlsanity.com/paper/2503.15801/) (2025)
- [TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learning](https://mlsanity.com/paper/2504.05585/) (2025)
- [Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring](https://mlsanity.com/paper/2509.25438/) (2025)
- [Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2408.10858/) (2024)
- [NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations](https://mlsanity.com/paper/2501.12668/) (2025)
- [ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World](https://mlsanity.com/paper/2505.19095/) (2025)
- [Exploration by Random Reward Perturbation](https://mlsanity.com/paper/2506.08737/) (2025)
- [DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration](https://mlsanity.com/paper/2603.07545/) (2026)
- [Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration](https://mlsanity.com/paper/2603.22273/) (2026)
- [Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning](https://mlsanity.com/paper/2408.02295/) (2024)
- [CIG: Exploration via Conditional Information Gain](https://mlsanity.com/paper/2605.20878/) (2026)
- [Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators](https://mlsanity.com/paper/2607.16858/) (2026)
- [Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification](https://mlsanity.com/paper/2609.03308/) (2026)
- [Escaping Local Views: Discovering Latent Concepts for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.34459/) (2026)

## Abstract (arXiv)

Exploration in environments with sparse rewards is difficult for artificial agents. Curiosity driven learning -- using feed-forward prediction errors as intrinsic rewards -- has achieved some success in these scenarios, but fails when faced with action-dependent noise sources. We present aleatoric mapping agents (AMAs), a neuroscience inspired solution modeled on the cholinergic system of the mammalian brain. AMAs aim to explicitly ascertain which dynamics of the environment are unpredictable, regardless of whether those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and variance of future states and reducing intrinsic rewards for those transitions with high aleatoric variance. We show AMAs are able to effectively circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents. The code for all experiments presented in this paper is open sourced: http://github.com/self-supervisor/Escaping-Stochastic-Traps-With-Aleatoric-Mapping-Agents.