mlsanityData to 14 Oct 2026

On Goodhart's law, with an application to value alignment

arXiv 2410.09638 ↗First posted 12 Oct 2024stat.ML cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

``When a measure becomes a target, it ceases to be a good measure'', this adage is known as it Goodhart's law. In this paper, we investigate formally this law and prove that it critically depends on the tail distribution of the discrepancy between the true goal and the measure that is optimized. Discrepancies with long-tail distributions favor a Goodhart's law, that is, the optimization of the measure can have a counter-productive effect on the goal. We provide a formal setting to assess Goodhart's law by studying the asymptotic behavior of the correlation between the goal and the measure, as the measure is optimized. Moreover, we introduce a distinction between a it weak Goodhart's law, when over-optimizing the metric is useless for the true goal, and a it strong Goodhart's law, when over-optimizing the metric is harmful for the true goal. A distinction which we prove to depend on the tail distribution. We stress the implications of this result to large-scale decision making and policies that are (and have to be) based on metrics, and propose numerous research directions to better assess the safety of such policies in general, and to the particularly concerning case where these policies are automated with algorithms.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 21 of its 50 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

21 of 50 references.

References · 21

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  7. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  8. 2019Two models of double descent for weak featurescited by 195
  9. 2018To understand deep learning we need to understand kernel learningcited by 187
  10. 2018Categorizing Variants of Goodhart's Lawcited by 149
  11. 2019Harmless interpolation of noisy data in regressioncited by 121
  12. 2019Optimal Policies Tend to Seek Powercited by 119
9 more
  1. 2018Does data interpolation contradict statistical optimality?cited by 95
  2. 2018Incomplete Contracting and AI Alignmentcited by 62
  3. 2017A Voting-Based System for Ethical Decision Makingcited by 55
  4. 2019Auditing Radicalization Pathways on YouTubecited by 53
  5. 2013Loss minimization and parameter estimation with heavy tailscited by 50
  6. 2019Robust subgaussian estimation of a mean vector in nearly linear timecited by 37
  7. 2020Adapting a Kidney Exchange Algorithm to Align with Human Valuescited by 36
  8. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  9. 2019Embedded Agencycited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  2. 2025The Reward Model Selection Crisis in Personalized Alignmentcited by 3
  3. 2025The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisationcited by 2
  4. 2026Consequentialist Objectives and Catastrophecited by 1
  5. 2026Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiabilitycited by 1
  6. 2025A Case for Specialisation in Non-Human Entitiescited by 0
  7. 2025Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimizationcited by 0
  8. 2026Why Code, Why Now: An Information-Theoretic Perspective on the Limits of Machine Learningcited by 0
  9. 2026A Cross Community Agenda for Speech AIcited by 0