mlsanityData to 14 Oct 2026

Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimization

arXiv 2503.07660 ↗First posted 8 Mar 2025cs.AI cs.CY cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

The recent leap in AI capabilities, driven by big generative models, has sparked the possibility of achieving Artificial General Intelligence (AGI) and further triggered discussions on Artificial Superintelligence (ASI)-a system surpassing all humans across measured domains. This gives rise to the critical research question of: As we approach ASI, how do we align it with human values, ensuring it benefits rather than harms human society, a.k.a., the Superalignment problem. Despite ASI being regarded by many as a hypothetical concept, in this position paper, we argue that superalignment is achievable and research on it should advance immediately, through simultaneous and alternating optimization of task competence and value conformity. We posit that superalignment is not merely a safeguard for ASI but also necessary for its responsible realization. To support this position, we first provide a formal definition of superalignment rooted in the gap between capability and capacity, delve into its perceived infeasibility by analyzing the limitations of existing paradigms, and then illustrate a conceptual path of superalignment to support its achievability, centered on two fundamental principles. This work frames a potential initiative for developing value-aligned next-generation AI in the future, which will garner greater benefits and reduce potential harm to humanity.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 126 of its 183 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

126 of 183 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization17 Jun
  2. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision14 Mar
  3. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models7 Mar
  4. 2024Investigating Cultural Alignment of Large Language Models20 Feb
  5. 2024Debating with More Persuasive LLMs Leads to More Truthful Answers9 Feb
  6. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning1 Feb
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  8. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec

Other references · 117

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  13. 2022Red Teaming Language Models with Language Modelscited by 1,026
  14. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  15. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  16. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  17. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  18. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  19. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  20. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  21. 2022Large Language Models Can Self-Improvecited by 664
  22. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  23. 2022Efficiently Scaling Transformer Inferencecited by 539
  24. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  25. 2018Behavioral Cloning from Observationcited by 492
  26. 2020Artificial Intelligence, Values and Alignmentcited by 448
  27. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  28. 2018AI safety via debatecited by 400
  29. 2022Self-critiquing models for assisting human evaluatorscited by 380
  30. 2024Alignment faking in large language modelscited by 360
  31. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  32. 2016Cooperative Inverse Reinforcement Learningcited by 300
  33. 2024Frontier Models are Capable of In-context Schemingcited by 287
  34. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  35. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  36. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  37. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  38. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251

50 most cited of 117 All 117 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Position: Align AI to Our Aspirations, Not Our Flawscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment
  2. 2025Super Co-alignment of Human and AI for Sustainable Symbiotic Society
  3. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Models
  4. 2025Neurodivergent Influenceability as a Contingent Solution to the AI Alignment Problem
  5. 2024Position: Towards Bidirectional Human-AI Alignment
  6. 2026Automated alignment is harder than you think
  7. 2025Superalignment with Dynamic Human Values
  8. 2024The Superalignment of Superhuman Intelligence with Large Language Models