# Avoiding Wireheading with Value Reinforcement Learning

arXiv: https://arxiv.org/abs/1605.03143 · HTML: https://mlsanity.com/paper/1605.03143/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2016.

## Built on by (most cited first, 15 of 15)

- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation](https://mlsanity.com/paper/2501.08617/) (2025)
- [Robust Learning from Observation with Model Misspecification](https://mlsanity.com/paper/2202.06003/) (2022)
- [Parameter Efficient Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2403.10704/) (2024)
- [Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models](https://mlsanity.com/paper/2507.07484/) (2025)
- [Morality, Machines and the Interpretation Problem: A Value-based, Wittgensteinian Approach to Building Moral Agents](https://mlsanity.com/paper/2103.02728/) (2021)
- [Concept Extrapolation: A Conceptual Primer](https://mlsanity.com/paper/2306.10999/) (2023)
- [Democratizing Reward Design for Personal and Representative Value-Alignment](https://mlsanity.com/paper/2410.22203/) (2024)
- [Belief-State Query Policies for User-Aligned POMDPs](https://mlsanity.com/paper/2405.15907/) (2024)
- [Does Self-Evaluation Enable Wireheading in Language Models?](https://mlsanity.com/paper/2511.23092/) (2025)
- [Modification-Considering Value Learning for Reward Hacking Mitigation in RL](https://mlsanity.com/paper/2606.28955/) (2026)

## Abstract (arXiv)

How can we design good goals for arbitrarily intelligent agents? Reinforcement learning (RL) is a natural approach. Unfortunately, RL does not work well for generally intelligent agents, as RL agents are incentivised to shortcut the reward sensor for maximum reward -- the so-called wireheading problem. In this paper we suggest an alternative to RL called value reinforcement learning (VRL). In VRL, agents use the reward signal to learn a utility function. The VRL setup allows us to remove the incentive to wirehead by placing a constraint on the agent's actions. The constraint is defined in terms of the agent's belief distributions, and does not require an explicit specification of which actions constitute wireheading.