mlsanityData to 7 Oct 2026

Superalignment with Dynamic Human Values

arXiv 2503.13621 ↗First posted 17 Mar 2025cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Two core challenges of alignment are 1) scalable oversight and 2) accounting for the dynamic nature of human values. While solutions like recursive reward modeling address 1), they do not simultaneously account for 2). We sketch a roadmap for a novel algorithmic framework that trains a superhuman reasoning model to decompose complex tasks into subtasks that are still amenable to human-level guidance. Our approach relies on what we call the part-to-complete generalization hypothesis, which states that the alignment of subtask solutions generalizes to the alignment of complete solutions. We advocate for the need to measure this generalization and propose ways to improve it in the future.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 18 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

16 of 18 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Weak-to-Strong Generalization Through the Data-Centric Lens5 Dec
  2. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision14 Mar
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  7. 2018AI safety via debatecited by 384
  8. 2023AI Alignment: A Comprehensive Surveycited by 256
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2018Supervising strong learners by amplifying weak expertscited by 139
  11. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 42
  12. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 33
1 more
  1. 2024Learning Task Decomposition to Assist Humans in Competitive Programmingcited by 14

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024The Superalignment of Superhuman Intelligence with Large Language Models
  2. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization
  3. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models
  4. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  5. 2024Position: Towards Bidirectional Human-AI Alignment
  6. 2023Concept Alignment as a Prerequisite for Value Alignment
  7. 2024MAP: Multi-Human-Value Alignment Palette
  8. 2026Thinking effort aligns between humans and reasoning models in abductive reasoning