mlsanityData to 14 Oct 2026

Large Language Model Alignment: A Survey

arXiv 2309.15025 ↗First posted 26 Sep 2023cs.CL cs.AIcited by 243

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent years have witnessed remarkable progress made in large language models (LLMs). Such advancements, while garnering significant attention, have concurrently elicited various concerns. The potential of these models is undeniably vast; however, they may yield texts that are imprecise, misleading, or even detrimental. Consequently, it becomes paramount to employ alignment techniques to ensure these models to exhibit behaviors consistent with human values. This survey endeavors to furnish an extensive exploration of alignment methodologies designed for LLMs, in conjunction with the extant capability research in this domain. Adopting the lens of AI alignment, we categorize the prevailing methods and emergent proposals for the alignment of LLMs into outer and inner alignment. We also probe into salient issues including the models' interpretability, and potential vulnerabilities to adversarial attacks. To assess LLM alignment, we present a wide variety of benchmarks and evaluation methodologies. After discussing the state of alignment research for LLMs, we finally cast a vision toward the future, contemplating the promising avenues of research that lie ahead. Our aspiration for this survey extends beyond merely spurring research interests in this realm. We also envision bridging the gap between the AI alignment research community and the researchers engrossed in the capability exploration of LLMs for both capable and safe LLMs.

Key facts. Cited by 243 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 196 of its 287 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

196 of 287 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets20 Jul
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models24 May
  4. 2023Aligning Large Language Models through Synthetic Feedback23 May
  5. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  6. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 187

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,320
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  13. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  14. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  15. 2020Learning to summarize from human feedbackcited by 2,588
  16. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  17. 2020Extracting Training Data from Large Language Modelscited by 2,189
  18. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  19. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  20. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  21. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  22. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  23. 2016Concrete Problems in AI Safetycited by 1,642
  24. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  25. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  26. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  27. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  28. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  29. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  30. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  31. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  32. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  33. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  34. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  35. 2022Toy Models of Superpositioncited by 971
  36. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  37. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  38. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839

50 most cited of 187 All 187 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 105; 2025: 100; 2026: 29.

243 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  3. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  5. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 197
  6. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  7. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  9. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  10. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  11. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  12. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
38 more
  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  2. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  3. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 86
  4. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  5. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  6. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  7. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  8. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 64
  9. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  10. 2025Safety in Large Reasoning Models: A Surveycited by 64
  11. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 63
  12. 2024Large Language Model Safety: A Holistic Surveycited by 59
  13. 2024Determinants of LLM-assisted Decision-Makingcited by 56
  14. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  15. 2025Do as We Do, Not as You Think: the Conformity of Large Language Modelscited by 53
  16. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  17. 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
  18. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  19. 2023Learning from models beyond fine-tuningcited by 48
  20. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  21. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  22. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  23. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  24. 2024Personality Alignment of Large Language Modelscited by 36
  25. 2025A Survey of Direct Preference Optimizationcited by 36
  26. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  27. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  28. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  29. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  30. 2024A Survey of Small Language Modelscited by 31
  31. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  32. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agentscited by 30
  33. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  34. 2024Weaver: Foundation Models for Creative Writingcited by 27
  35. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  36. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 25
  37. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 24
  38. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24

50 most cited of 243 All 243 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2023Aligning Large Language Models with Human: A Survey
  3. 2026LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations
  4. 2025Aligning Multimodal LLM with Human Preference: A Survey
  5. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  6. 2024DeAL: Decoding-time Alignment for Large Language Models
  7. 2023The Alignment Problem in Context
  8. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety