# Against Proxy Optimization

arXiv: https://arxiv.org/abs/2606.23597 · HTML: https://mlsanity.com/paper/2606.23597/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking](https://mlsanity.com/paper/2403.03185/) (2024)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Goodhart's Law in Reinforcement Learning](https://mlsanity.com/paper/2310.09144/) (2023)
- [The Problem with Metrics is a Fundamental Problem for AI](https://mlsanity.com/paper/2002.08512/) (2020)
- [Off-Switching Not Guaranteed](https://mlsanity.com/paper/2502.08864/) (2025)

## Built on by (most cited first, 1 of 1)

- [Fragility of Value under Imperfect Alignment](https://mlsanity.com/paper/2607.28881/) (2026)

## Abstract (arXiv)

I discuss conditions under which maximizing a proxy utility function is harmful and suggest this poses problems for applying decision theory.