# MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization

arXiv: https://arxiv.org/abs/2412.12098 · HTML: https://mlsanity.com/paper/2412.12098/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 63 of its 91 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 63)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Learning to Optimize via Information-Directed Sampling](https://mlsanity.com/paper/1403.5556/) (2014)
- [On Information Gain and Regret Bounds in Gaussian Process Bandits](https://mlsanity.com/paper/2009.06966/) (2020)

## Built on by (most cited first, 30 of 30)

- [Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.01601/) (2026)
- [Learning Sim-to-Real Humanoid Locomotion in 15 Minutes](https://mlsanity.com/paper/2512.01996/) (2025)
- [DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning](https://mlsanity.com/paper/2505.19850/) (2025)
- [Epistemically-guided forward-backward exploration](https://mlsanity.com/paper/2507.05477/) (2025)
- [SOMBRL: Scalable and Optimistic Model-Based RL](https://mlsanity.com/paper/2511.20066/) (2025)
- [How Should We Meta-Learn Reinforcement Learning Algorithms?](https://mlsanity.com/paper/2507.17668/) (2025)
- [Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow](https://mlsanity.com/paper/2601.19707/) (2026)
- [Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying](https://mlsanity.com/paper/2606.00151/) (2026)
- [Optimistic critics can empower small actors](https://mlsanity.com/paper/2506.01016/) (2025)
- [Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration](https://mlsanity.com/paper/2509.20648/) (2025)
- [Sample-efficient and Scalable Exploration in Continuous-Time RL](https://mlsanity.com/paper/2510.24482/) (2025)
- [Off-policy Reinforcement Learning with Model-based Exploration Augmentation](https://mlsanity.com/paper/2510.25529/) (2025)
- [Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models](https://mlsanity.com/paper/2505.07815/) (2025)
- [Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization](https://mlsanity.com/paper/2506.13345/) (2025)
- [Learning Soft Robotic Dynamics with Active Exploration](https://mlsanity.com/paper/2510.27428/) (2025)
- [Emergence of Physical Intelligence via Controllable Information Production](https://mlsanity.com/paper/2601.22449/) (2026)
- [ModalImmune: Immunity Driven Unlearning via Self Destructive Training](https://mlsanity.com/paper/2602.16197/) (2026)
- [Model-Based Reinforcement Learning for Control under Time-Varying Dynamics](https://mlsanity.com/paper/2604.02260/) (2026)
- [Retry Policy Gradients in Continuous Action Spaces](https://mlsanity.com/paper/2606.05888/) (2026)
- [Learning What Matters Now: A Dual-Critic Context-Aware RL Framework for Priority-Driven Information Gain](https://mlsanity.com/paper/2506.06786/) (2025)
- [Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning](https://mlsanity.com/paper/2509.20766/) (2025)
- [Distributional Active Inference](https://mlsanity.com/paper/2601.20985/) (2026)
- [Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration](https://mlsanity.com/paper/2605.22814/) (2026)
- [All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2606.01363/) (2026)
- [UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2606.19328/) (2026)
- [Information-Based Exploration via Random Features for Reinforcement Learning](https://mlsanity.com/paper/2607.17981/) (2026)
- [Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2608.07086/) (2026)
- [V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control](https://mlsanity.com/paper/2608.07870/) (2026)
- [Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning](https://mlsanity.com/paper/2609.29548/) (2026)
- [Tactile Curiosity Drives Robot Interaction](https://mlsanity.com/paper/2609.40134/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.