mlsanityData to 7 Oct 2026

Exploration and preference satisfaction trade-off in reward-free learning

arXiv 2106.04316 ↗First posted 8 Jun 2021cs.AI q-bio.NCcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Biological agents have meaningful interactions with their environment despite the absence of immediate reward signals. In such instances, the agent can learn preferred modes of behaviour that lead to predictable states -- necessary for survival. In this paper, we pursue the notion that this learnt behaviour can be a consequence of reward-free preference learning that ensures an appropriate trade-off between exploration and preference satisfaction. For this, we introduce a model-based Bayesian agent equipped with a preference learning mechanism (pepper) using conjugate priors. These conjugate priors are used to augment the expected free energy planner for learning preferences over states (or outcomes) across time. Importantly, our approach enables the agent to learn preferences that encourage adaptive behaviour at test time. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Given a constant environment, these agents learn confident (i.e., precise) preferences and act to satisfy them. Conversely, in a volatile setting, perpetual preference uncertainty maintains exploratory behaviour. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction. Pepper offers a straightforward framework suitable for designing adaptive agents when reward functions cannot be predefined as in real environments.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 70 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

31 of 70 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2016OpenAI Gymcited by 1,684
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
19 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  2. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  3. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  4. 2019Contrastive Learning of Structured World Modelscited by 206
  5. 2019Efficient Exploration via State Marginal Matchingcited by 186
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  8. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  9. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  10. 2018Model-Based Active Explorationcited by 105
  11. 2020Active inference on discrete state-spaces: a synthesiscited by 91
  12. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  13. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  14. 2020Deep active inference agents using Monte-Carlo methodscited by 72
  15. 2017Deep Active Inferencecited by 44
  16. 2020Whence the Expected Free Energy?cited by 40
  17. 2018Neural Predictive Belief Representationscited by 39
  18. 2020Ready Policy One: World Building Through Active Learningcited by 33
  19. 2020Learning Perception and Planning with Deep Active Inferencecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 1; 2024: 2; 2025: 4; 2026: 1.

12 papers.

  1. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  2. 2023On efficient computation in active inferencecited by 15
  3. 2024Active Inference as a Model of Agencycited by 12
  4. 2021Branching Time Active Inference: empirical study and complexity class analysiscited by 9
  5. 2022Goal-directed Planning and Goal Understanding by Active Inference: Evaluation Through Simulated and Physical Robot Experimentscited by 4
  6. 2024Active Inference and Human--Computer Interactioncited by 4
  7. 2022Modelling non-reinforced preferences using selective attentioncited by 1
  8. 2025Emergence of Goal-Directed Behaviors via Active Inference with Self-Priorcited by 1
  9. 2025A Computational Model of Inclusive Pedagogy: From Understanding to Applicationcited by 0
  10. 2025Real-World Robot Control by Deep Active Inference With a Temporally Hierarchical World Modelcited by 0
  11. 2025Prior preferences in active inference agents: soft, hard, and goal shapingcited by 0
  12. 2026Distributional Active Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  2. 2021Prior Preference Learning from Experts:Designing a Reward with Active Inference
  3. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design
  4. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  5. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
  6. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
  7. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
  8. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learning