mlsanityData to 14 Oct 2026

Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors

arXiv 2507.19725 ↗First posted 26 Jul 2025cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

also in 1 more
Abstract (arXiv)

Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our results suggest that IM causes noticeable change by increasing the initial rewards, but also altering the way the agent plays; and that GRM mitigated reward hacking in some scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 28 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

21 of 28 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  7. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  8. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  9. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  10. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  11. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
  12. 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
9 more
  1. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  2. 2024Position: Benchmarking is Limited in Reinforcement Learning Researchcited by 15
  3. 2024Potential-Based Reward Shaping For Intrinsic Motivationcited by 12
  4. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  5. 2023DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewardscited by 7
  6. 2023Beyond Surprise: Improving Exploration Through Surprise Noveltycited by 5
  7. 2020Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Gamescited by 3
  8. 2024Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewardscited by 3
  9. 2025Action-Dependent Optimality-Preserving Reward Shapingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review
  2. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
  3. 2026Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization
  4. 2025LLM-Driven Intrinsic Motivation for Sparse Reward Reinforcement Learning
  5. 2026Intrinsic Motivation in Reinforcement Learning: A Research Agenda for Adaptive Self-Organisation
  6. 2024RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
  8. 2025Reward Models in Deep Reinforcement Learning: A Survey