mlsanityData to 7 Oct 2026

Extinction Risks from AI: Invisible to Science?

arXiv 2403.05540 ↗First posted 2 Feb 2024cs.CY cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart's Law as "Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity", and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart's Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that aims to be informative for evaluating specific arguments for Extinction-level Goodhart's Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 40 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

11 of 40 references.

References · 11

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2016Cooperative Inverse Reinforcement Learningcited by 298
  5. 2018Categorizing Variants of Goodhart's Lawcited by 132
  6. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  7. 2021Consequences of Misaligned AIcited by 79
  8. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  9. 2023Goodhart's Law in Reinforcement Learningcited by 37
  10. 2019Preferences Implicit in the State of the Worldcited by 33
  11. 2022On Avoiding Power-Seeking by Artificial Intelligencecited by 3

Built on by

Not cited yet.