mlsanityData to 14 Oct 2026

A Survey on Human Preference Learning for Large Language Models

arXiv 2406.11191 ↗First posted 17 Jun 2024cs.CLcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning, enhancing LLMs with excellent applicability and effectiveness in a wide range of contexts. Despite the numerous related studies conducted, a perspective on how human preferences are introduced into LLMs remains limited, which may prevent a deeper comprehension of the relationships between human preferences and LLMs as well as the realization of their limitations. In this survey, we review the progress in exploring human preference learning for LLMs from a preference-centered perspective, covering the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs. We first categorize the human feedback according to data sources and formats. We then summarize techniques for human preferences modeling and compare the advantages and disadvantages of different schools of models. Moreover, we present various preference usage methods sorted by the objectives to utilize human preference signals. Finally, we summarize some prevailing approaches to evaluate LLMs in terms of alignment with human intentions and discuss our outlooks on the human intention alignment for LLMs.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 140 of its 198 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

140 of 198 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models7 Mar
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2023Reasons to Reject? Aligning Language Models with Judgments22 Dec
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  5. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  6. 2023Training Socially Aligned Language Models on Simulated Social Interactions26 May
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  8. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation1 May
  9. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits1 Jan
  10. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  11. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov

Other references · 129

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,320
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  13. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  14. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  15. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  16. 2023A Survey of Large Language Modelscited by 2,576
  17. 2018Deep contextualized word representationscited by 2,290
  18. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  19. 2024Mixtral of Expertscited by 1,823
  20. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  21. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  22. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  24. 2023PaLM 2 Technical Reportcited by 1,162
  25. 2023LIMA: Less Is More for Alignmentcited by 1,100
  26. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  27. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  28. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  29. 2021Ethical and social risks of harm from Language Modelscited by 966
  30. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  31. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  32. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  33. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  34. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  35. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  36. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  37. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  38. 2021Deduplicating Training Data Makes Language Models Bettercited by 714

50 most cited of 129 All 129 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 14; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  2. 2025A Survey of Direct Preference Optimizationcited by 36
  3. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 20
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  6. 2024Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generationcited by 13
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  8. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  9. 2024Goal Inference from Open-Ended Dialogcited by 4
  10. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  11. 2025Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoningcited by 4
  12. 2025Can LLMs Make (Personalized) Access Control Decisions?cited by 4
12 more
  1. 2024RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignmentcited by 3
  2. 2025Preference learning made easy: Everything should be understood through win ratecited by 3
  3. 2025Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignmentcited by 3
  4. 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedbackcited by 2
  5. 2025Listwise Preference Alignment Optimization for Tail Item Recommendationcited by 2
  6. 2025SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentationcited by 2
  7. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  8. 2025Preference Learning with Response Time: Robust Losses and Guaranteescited by 1
  9. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  10. 2026Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Modelscited by 0
  11. 2026Internal Pluralism and the Limits of Pairwise Comparisonscited by 0
  12. 2026An Investigation of Translationese in the Generations of Multilingual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  2. 2024Dissecting Human and LLM Preferences
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2024Aligning language models with human preferences
  5. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  6. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  7. 2024Investigating on RLHF methodology
  8. 2024Aligning to Thousands of Preferences via System Message Generalization