mlsanityData to 7 Oct 2026

The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment

arXiv 2412.16468 ↗First posted 21 Dec 2024cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

The emergence of large language models (LLMs) has sparked discussion on Artificial Superintelligence (ASI), a hypothetical AI system that surpasses human intelligence. Although ASI remains hypothetical and far beyond current AI capabilities, discussing its potential and exploring its feasibility and potential risks is critical for the development of future AI systems. The idea of superalignment originates from scalable oversight, which studies how to supervise increasingly capable AI systems when direct human supervision becomes insufficient. In this paper, we focus on the superalignment problem: "The process of supervising, controlling, and governing artificial superintelligence." We first review scalable oversight paradigms-Sandwiching, Self-Enhancement, and Weak-to-Strong Generalization -- then analyze the limitations of current paradigms through the lens of possibility and impossibility, discuss key challenges, and propose pathways for the safe and continual improvement of future AI systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 100 of its 171 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

100 of 171 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Weak-to-Strong Generalization Through the Data-Centric Lens5 Dec
  2. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization17 Jun
  3. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision14 Mar
  4. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models7 Mar
  5. 2024Debating with More Persuasive LLMs Leads to More Truthful Answers9 Feb
  6. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models6 Feb
  7. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning1 Feb
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  9. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb

Other references · 91

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  13. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  14. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  15. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  16. 2018AI safety via debatecited by 384
  17. 2022Self-critiquing models for assisting human evaluatorscited by 372
  18. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  19. 2016Cooperative Inverse Reinforcement Learningcited by 298
  20. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  21. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  22. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  23. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  24. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  25. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  26. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  27. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  28. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  29. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  30. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  31. 2020CoMatch: Semi-supervised Learning with Contrastive Graph Regularizationcited by 167
  32. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  33. 2018Supervising strong learners by amplifying weak expertscited by 139
  34. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  35. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  36. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  37. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  38. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2026Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANPcited by 14
  2. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  3. 2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
  4. 2026The Last AI Built by Humans: Toward Genuine Recursive Self-Improvementcited by 5
  5. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  6. 2025ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Codecited by 2
  7. 2025Contrastive Weak-to-strong Generalizationcited by 2
  8. 2025Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curationcited by 2
  9. 2025Alice: Proactive Learning with Teacher's Demonstrations for Weak-to-Strong Generalizationcited by 1
  10. 2025A Criminology of Machinescited by 1
  11. 2026A three-dimensional typology of agency for advanced AI systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization
  2. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models
  3. 2024The Superalignment of Superhuman Intelligence with Large Language Models
  4. 2023The Alignment Problem in Context
  5. 2024Unlocking the Wisdom of Large Language Models: An Introduction to The Path to Artificial General Intelligence
  6. 2025Super Co-alignment of Human and AI for Sustainable Symbiotic Society
  7. 2024Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
  8. 2023Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models