mlsanityData to 14 Oct 2026

Understanding the Learning Dynamics of Alignment with Human Feedback

arXiv 2403.18742 ↗First posted 27 Mar 2024cs.LG cs.AIcited by 15

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human intentions has become a critical task for safely deploying models in real-world systems. While existing alignment approaches have seen empirical success, theoretically understanding how these methods affect model behavior remains an open question. Our work provides an initial attempt to theoretically analyze the learning dynamics of human preference alignment. We formally show how the distribution of preference datasets influences the rate of model updates and provide rigorous guarantees on the training accuracy. Our theory also reveals an intricate phenomenon where the optimization is prone to prioritizing certain behaviors with higher preference distinguishability. We empirically validate our findings on contemporary LLMs and alignment tasks, reinforcing our theoretical insights and shedding light on considerations for future alignment approaches. Disclaimer: This paper contains potentially offensive text; reader discretion is advised.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 59 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

53 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2023Fundamental Limitations of Alignment in Large Language Models19 Apr
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Mistral 7Bcited by 3,164
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
37 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  15. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  16. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  17. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  18. 2021Unsolved Problems in ML Safetycited by 292
  19. 2023AI Alignment: A Comprehensive Surveycited by 263
  20. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  21. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  22. 2023Nash Learning from Human Feedbackcited by 224
  23. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  24. 2023Model evaluation for extreme riskscited by 177
  25. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  26. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
  27. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  28. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  29. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  30. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  31. 2023Is RLHF More Difficult than Standard RL?cited by 96
  32. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  33. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 79
  34. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  35. 2018Fast convergence rates of deep neural networks for classificationcited by 46
  36. 2023Characterizing Manipulation from AI Systemscited by 43
  37. 2018Understanding the Loss Surface of Neural Networks for Binary Classificationcited by 27

Built on by

Cited by year

Citing papers per year: 2024: 10; 2025: 5.

15 papers.

Sort:

By area:

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  3. 2024Process Reward Model with Q-Value Rankingscited by 80
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  5. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  6. 2024On the Learnability of Out-of-distribution Detectioncited by 8
  7. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  8. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  9. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  10. 2024New Desiderata for Direct Preference Optimizationcited by 5
  11. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 5
  12. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
3 more
  1. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  2. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  3. 2025Intra-Trajectory Consistency for Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Human Preference Learning for Large Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
  4. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  5. 2023On Diversified Preferences of Large Language Model Alignment
  6. 2021A General Language Assistant as a Laboratory for Alignment
  7. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback