# Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors

arXiv: https://arxiv.org/abs/2507.19725 · HTML: https://mlsanity.com/paper/2507.19725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 26 Jul 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [Redeeming Intrinsic Rewards via Constrained Optimization](https://mlsanity.com/paper/2211.07627/) (2022)
- [Position: Benchmarking is Limited in Reinforcement Learning Research](https://mlsanity.com/paper/2406.16241/) (2024)
- [Potential-Based Reward Shaping For Intrinsic Motivation](https://mlsanity.com/paper/2402.07411/) (2024)
- [The impact of intrinsic rewards on exploration in Reinforcement Learning](https://mlsanity.com/paper/2501.11533/) (2025)
- [DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards](https://mlsanity.com/paper/2304.10770/) (2023)
- [Beyond Surprise: Improving Exploration Through Surprise Novelty](https://mlsanity.com/paper/2308.04836/) (2023)
- [Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games](https://mlsanity.com/paper/2010.03956/) (2020)
- [Potential-Based Intrinsic Motivation: Preserving Optimality With Complex, Non-Markovian Shaping Rewards](https://mlsanity.com/paper/2410.12197/) (2024)
- [Action-Dependent Optimality-Preserving Reward Shaping](https://mlsanity.com/paper/2505.12611/) (2025)

## Abstract (arXiv)

Games are challenging for Reinforcement Learning~(RL) agents due to their reward-sparsity, as rewards are only obtainable after long sequences of deliberate actions. Intrinsic Motivation~(IM) methods -- which introduce exploration rewards -- are an effective solution to reward-sparsity. However, IM also causes an issue known as `reward hacking' where the agent optimizes for the new reward at the expense of properly playing the game. The larger problem is that reward hacking itself is largely unknown; there is no answer to whether, and to what extent, IM rewards change the behavior of RL agents. This study takes a first step by empirically evaluating the impact on behavior of three IM techniques on the MiniGrid game-like environment. We compare these IM models with Generalized Reward Matching~(GRM), a method that can be used with any intrinsic reward function to guarantee optimality. Our results suggest that IM causes noticeable change by increasing the initial rewards, but also altering the way the agent plays; and that GRM mitigated reward hacking in some scenarios.