mlsanityData to 14 Oct 2026

Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

arXiv 2610.19832 ↗First posted 14 Oct 2026cs.LGcited by 0

New in LLM alignment / RLHF / preference optimisation; no direction yet. LLM alignment / RLHF / preference optimisation

Abstract (arXiv)

Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 58 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

31 of 58 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  9. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  10. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  12. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
19 more
  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  2. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  3. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  4. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  5. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  6. 2018Categorizing Variants of Goodhart's Lawcited by 149
  7. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  8. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  9. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  10. 2018Incomplete Contracting and AI Alignmentcited by 62
  11. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  12. 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviorscited by 38
  13. 2022Uncertainty Estimation for Language Reward Modelscited by 37
  14. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  15. 2025Reward Model Overoptimisation in Iterated RLHFcited by 12
  16. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 11
  17. 2026Reward Hacking as Equilibrium under Finite Evaluationcited by 5
  18. 2026AI Alignment via Incentives and Correctioncited by 2
  19. 2026Alignment as Institutional Design: From Behavioral Correction to Transaction Structure in Intelligent Systemscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
  3. 2025Inference-Time Reward Hacking in Large Language Models
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHF
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
  6. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
  8. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling