A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?
Not on the map. Why?
Abstract (arXiv)
The value-loading problem is a significant challenge for researchers aiming to create artificial intelligence (AI) systems that align with human values and preferences. This problem requires a method to define and regulate safe and optimal limits of AI behaviors. In this work, we propose HALO (Hormetic ALignment via Opponent processes), a regulatory paradigm that uses hormetic analysis to regulate the behavioral patterns of AI. Behavioral hormesis is a phenomenon where low frequencies of a behavior have beneficial effects, while high frequencies are harmful. By modeling behaviors as allostatic opponent processes, we can use either Behavioral Frequency Response Analysis (BFRA) or Behavioral Count Response Analysis (BCRA) to quantify the hormetic limits of repeatable behaviors. We demonstrate how HALO can solve the 'paperclip maximizer' scenario, a thought experiment where an unregulated AI tasked with making paperclips could end up converting all matter in the universe into paperclips. Our approach may be used to help create an evolving database of 'values' based on the hedonic calculus of repeatable behaviors with decreasing marginal utility. This positions HALO as a promising solution for the value-loading problem, which involves embedding human-aligned values into an AI system, and the weak-to-strong generalization problem, which explores whether weak models can supervise stronger models as they become more intelligent. Hence, HALO opens several research avenues that may lead to the development of a computational value system that allows an AI algorithm to learn whether the decisions it makes are right or wrong.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 108 references that are in the map. First posted on arXiv 12 Feb 2024.
Builds on
24 of 108 references.
References · 24
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
- 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
- 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
- 2018Scalable agent alignment via reward modeling: a research directioncited by 438
- 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
12 more
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
- 2024A Roadmap to Pluralistic Alignmentcited by 175
- 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
- 2024Human-Instruction-Free LLM Self-Alignment with Limited Samplescited by 25
- 2023Algorithmic Pluralism: A Structural Approach To Equal Opportunitycited by 13
- 2023Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learningcited by 9
- 2022Provable Safe Reinforcement Learning with Binary Feedbackcited by 7
- 2020Distributional Reinforcement Learning with Ensemblescited by 3
- 2013Model Predictive Regulationcited by 1
- 2022A mathematical model of reward-mediated learning in drug addictioncited by 1
Built on by
Not cited yet.