mlsanityData to 7 Oct 2026

Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control

arXiv 2103.06859 ↗First posted 11 Mar 2021cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. While many approaches have been taken, one approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' which is often implemented in terms of maximizing the agents information gain about the world -- an approach which has been widely studied in machine learning and cognitive science. In this paper we mathematically investigate the nature and meaning of such approaches and demonstrate that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between evidence objectives, which correspond to well-known reward or utility maximizing objectives in the literature, and divergence objectives which instead seek to minimize the divergence between the agent's expected and desired futures, and argue that this new class of divergence objectives could form the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 24 of its 122 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

24 of 122 references.

References · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2016Variational Intrinsic Controlcited by 262
  10. 2019Efficient Exploration via State Marginal Matchingcited by 186
  11. 2013lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Banditscited by 148
  12. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
12 more
  1. 2018Model-Based Active Explorationcited by 105
  2. 2020Active inference on discrete state-spaces: a synthesiscited by 91
  3. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  4. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  5. 2019Deep Active Inference as Variational Policy Gradientscited by 52
  6. 2020Reinforcement Learning through Active Inferencecited by 47
  7. 2020On the Relationship Between Active Inference and Control as Inferencecited by 42
  8. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  9. 2020Relative Variational Intrinsic Controlcited by 34
  10. 2020Action and Perception as Divergence Minimizationcited by 31
  11. 2019Scaling active inferencecited by 30
  12. 2017Unsupervised Real-Time Control through Variational Empowermentcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 1; 2024: 0; 2025: 1; 2026: 2.

10 papers.

  1. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  2. 2021Active Inference in Robotics and Artificial Agents: Survey and Challengescited by 52
  3. 2022pymdp: A Python library for active inference in discrete state spacescited by 39
  4. 2021Applications of the Free Energy Principle to Machine Learning and Neurosciencecited by 9
  5. 2021Active Inference and Epistemic Value in Graphical Modelscited by 6
  6. 2023Realising Synthetic Active Inference Agents, Part I: Epistemic Objectives and Graphical Specification Languagecited by 4
  7. 2022Successor Representation Active Inferencecited by 3
  8. 2026Active Inference as a Convex Markov Decision Processcited by 1
  9. 2025Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspectivecited by 0
  10. 2026Active Inference as Context Acquisition for AI Agentscited by 0