mlsanityData to 7 Oct 2026

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

arXiv 2305.03047 ↗First posted 4 May 2023cs.LG cs.AI cs.CLcited by 318

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including < 200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 79 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

39 of 79 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
23 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2016Sequence-Level Knowledge Distillationcited by 761
  13. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  14. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  15. 2020Artificial Intelligence, Values and Alignmentcited by 443
  16. 2018Gender Bias in Coreference Resolutioncited by 438
  17. 2023Poisoning Language Models During Instruction Tuningcited by 254
  18. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  19. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  20. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  21. 2022Recitation-Augmented Language Modelscited by 65
  22. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  23. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 73; 2024: 152; 2025: 63; 2026: 30.

318 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023A Comprehensive Overview of Large Language Modelscited by 639
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  10. 2024Large Language Models: A Surveycited by 506
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
38 more
  1. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  4. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  8. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  9. 2023Large Language Model Alignment: A Surveycited by 241
  10. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  12. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  13. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  14. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  15. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  16. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  17. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  18. 2023Self-Alignment with Instruction Backtranslationcited by 149
  19. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  20. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  21. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  22. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  23. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  24. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  25. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  26. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  27. 2023Shepherd: A Critic for Language Model Generationcited by 96
  28. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  29. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  30. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  31. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  32. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  33. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  34. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  35. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  36. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  37. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  38. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63

50 most cited of 318 All 318 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  2. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2024Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
  5. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
  6. 2026HAL: Inducing Human-likeness in LLMs with Alignment
  7. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
  8. 2026Emergent Alignment