mlsanityData to 14 Oct 2026

MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization

arXiv 2412.12098 ↗First posted 16 Dec 2024cs.LG cs.AI cs.ROcited by 30

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 63 of its 91 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

63 of 91 references.

References · 63

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2016OpenAI Gymcited by 1,736
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2018DeepMind Control Suitecited by 956
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  6. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  10. 2017On Kernelized Multi-armed Banditscited by 352
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  13. 2019Self-Supervised Exploration via Disagreementcited by 265
  14. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  15. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  16. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  17. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  18. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 158
  19. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  20. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  21. 2021Discovering and Achieving Goals via World Modelscited by 131
  22. 2021Epistemic Neural Networkscited by 120
  23. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  24. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  25. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  26. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
  27. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  28. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  29. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  30. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 80
  31. 2017Boltzmann Exploration Done Rightcited by 69
  32. 2013Empowerment -- an Introductioncited by 68
  33. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  34. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  35. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  36. 2020Action and Perception as Divergence Minimizationcited by 32
  37. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  38. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 16.

30 papers.

Sort:

By area:

  1. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  2. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
  3. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  4. 2025Epistemically-guided forward-backward explorationcited by 8
  5. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  6. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  7. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  8. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  9. 2025Optimistic critics can empower small actorscited by 4
  10. 2025Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibrationcited by 4
  11. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  12. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
18 more
  1. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  2. 2025Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximizationcited by 2
  3. 2025Learning Soft Robotic Dynamics with Active Explorationcited by 2
  4. 2026Emergence of Physical Intelligence via Controllable Information Productioncited by 1
  5. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
  6. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
  7. 2026Retry Policy Gradients in Continuous Action Spacescited by 1
  8. 2025Learning What Matters Now: A Dual-Critic Context-Aware RL Framework for Priority-Driven Information Gaincited by 0
  9. 2025Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learningcited by 0
  10. 2026Distributional Active Inferencecited by 0
  11. 2026Remember to be Curious: Episodic Context and Persistent Worlds for 3D Explorationcited by 0
  12. 2026All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learningcited by 0
  13. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0
  14. 2026Information-Based Exploration via Random Features for Reinforcement Learningcited by 0
  15. 2026Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Controlcited by 0
  16. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  17. 2026Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learningcited by 0
  18. 2026Tactile Curiosity Drives Robot Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Exploration by Maximizing R'enyi Entropy for Reward-Free RL Framework
  2. 2025On Efficient Bayesian Exploration in Model-Based Reinforcement Learning
  3. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  4. 2024Beyond Optimism: Exploration With Partially Observable Rewards
  5. 2025CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning
  6. 2026Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity
  7. 2025Exploration by Random Reward Perturbation
  8. 2024Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration