mlsanityData to 7 Oct 2026

On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models

arXiv 2403.04204 ↗First posted 7 Mar 2024cs.AI cs.CLcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Big models have achieved revolutionary breakthroughs in the field of AI, but they might also pose potential concerns. Addressing such concerns, alignment technologies were introduced to make these models conform to human preferences and values. Despite considerable advancements in the past year, various challenges lie in establishing the optimal alignment strategy, such as data cost and scalable oversight, and how to align remains an open question. In this survey paper, we comprehensively investigate value alignment approaches. We first unpack the historical context of alignment tracing back to the 1920s (where it comes from), then delve into the mathematical essence of alignment (what it is), shedding light on the inherent challenges. Following this foundation, we provide a detailed examination of existing alignment methods, which fall into three categories: Reinforcement Learning, Supervised Fine-Tuning, and In-context Learning, and demonstrate their intrinsic connections, strengths, and limitations, helping readers better understand this research area. In addition, two emerging topics, personal alignment, and multimodal alignment, are also discussed as novel frontiers in this field. Looking forward, we discuss potential alignment paradigms and how they could handle remaining challenges, prospecting where future alignment will go.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 93 of its 157 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

93 of 157 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  2. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb
  3. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits1 Jan
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  13. 2023A Survey of Large Language Modelscited by 2,537
  14. 2022Emergent Abilities of Large Language Modelscited by 2,483
  15. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  16. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  17. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  18. 2023LIMA: Less Is More for Alignmentcited by 1,080
  19. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  20. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  21. 2021Ethical and social risks of harm from Language Modelscited by 957
  22. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  23. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  24. 2023Whose Opinions Do Language Models Reflect?cited by 759
  25. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  26. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  27. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  28. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  29. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  30. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  31. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  32. 2018Behavioral Cloning from Observationcited by 482
  33. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  34. 2020Artificial Intelligence, Values and Alignmentcited by 443
  35. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  36. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  37. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  38. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  4. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  6. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  7. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  8. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  9. 2024Can a large language model be a gaslighter?cited by 5
  10. 2025CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimizationcited by 5
  11. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  12. 2024REAL: Response Embedding-based Alignment for LLMscited by 4
9 more
  1. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignmentcited by 4
  2. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  3. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  4. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  5. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundariescited by 1
  6. 2026Labels have Human Values: Value Calibration of Subjective Taskscited by 1
  7. 2025Testing for Causal Fairnesscited by 0
  8. 2025Towards Efficient and Effective Alignment of Large Language Modelscited by 0
  9. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
  2. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  3. 2024Position: Towards Bidirectional Human-AI Alignment
  4. 2025The Specification Trap: Why Static Value Alignment Alone Is Insufficient for Robust Alignment
  5. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  6. 2026Toward a Theory of Value in AI Alignment
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback