# Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

arXiv: https://arxiv.org/abs/2610.19832 · HTML: https://mlsanity.com/paper/2610.19832/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation

## Builds on (most cited first, 30 of 31)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://mlsanity.com/paper/2312.09244/) (2023)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [ODIN: Disentangled Reward Mitigates Hacking in RLHF](https://mlsanity.com/paper/2402.07319/) (2024)
- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [Theoretical guarantees on the best-of-n alignment policy](https://mlsanity.com/paper/2401.01879/) (2024)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [From Lists to Emojis: How Format Bias Affects Model Alignment](https://mlsanity.com/paper/2409.11704/) (2024)
- [AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors](https://mlsanity.com/paper/2602.22755/) (2026)
- [Uncertainty Estimation for Language Reward Models](https://mlsanity.com/paper/2203.07472/) (2022)
- [Spontaneous Reward Hacking in Iterative Self-Refinement](https://mlsanity.com/paper/2407.04549/) (2024)
- [Reward Model Overoptimisation in Iterated RLHF](https://mlsanity.com/paper/2505.18126/) (2025)
- [Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking](https://mlsanity.com/paper/2602.01750/) (2026)
- [Reward Hacking as Equilibrium under Finite Evaluation](https://mlsanity.com/paper/2603.28063/) (2026)
- [AI Alignment via Incentives and Correction](https://mlsanity.com/paper/2605.01643/) (2026)

## Abstract (arXiv)

Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.