mlsanityData to 7 Oct 2026

Towards a Unified View of Preference Learning for Large Language Models: A Survey

arXiv 2409.02795 ↗First posted 4 Sep 2024cs.CLcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. This alignment process often requires only a small amount of data to efficiently enhance the LLM's performance. While effective, research in this area spans multiple domains, and the methods involved are relatively complex to understand. The relationships between different methods have been under-explored, limiting the development of the preference alignment. In light of this, we break down the existing popular alignment strategies into different components and provide a unified framework to study the current alignment strategies, thereby establishing connections among them. In this survey, we decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm. This unified view offers an in-depth understanding of existing alignment algorithms and also opens up possibilities to synergize the strengths of different strategies. Furthermore, we present detailed working examples of prevalent existing algorithms to facilitate a comprehensive understanding for the readers. Finally, based on our unified perspective, we explore the challenges and future research directions for aligning large language models with human preferences.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 143 of its 170 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

143 of 170 references.

Closest ancestors · 38

Same direction, newest first

  1. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward11 Jun
  2. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinement31 May
  3. 2024TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models30 May
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier28 May
  5. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024LIRE: listwise reward enhancement for preference alignment22 May
  8. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks23 Apr
  9. 2024Filtered Direct Preference Optimization22 Apr
  10. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  11. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  12. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment1 Apr
26 more
  1. 2024sDPO: Don't Use Your Data All at Once28 Mar
  2. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment17 Mar
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  5. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration25 Feb
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  8. 2024ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization14 Feb
  9. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  10. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  11. 2024Decoding-time Realignment of Language Models5 Feb
  12. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  13. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  14. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization12 Jan
  15. 2023Align on the Fly: Adapting Chatbot Behavior to Established Norms26 Dec
  16. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  17. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Training7 Nov
  18. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment25 Oct
  19. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  20. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  21. 2023Large Language Model Alignment: A Survey26 Sep
  22. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  23. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  24. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
  25. 2023Preference Ranking Optimization for Human Alignment30 Jun
  26. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 105

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  14. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  15. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  16. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  17. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  18. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  19. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  20. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  22. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  23. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  24. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  25. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  26. 2024Self-Rewarding Language Modelscited by 654
  27. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  28. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  29. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  30. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  31. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  32. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  33. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  34. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  35. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  36. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  37. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  38. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  2. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  5. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  6. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  7. 2025UltraIF: Advancing Instruction Following from the Wildcited by 18
  8. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decodingcited by 7
  9. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  10. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
  11. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 3
  12. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
5 more
  1. 2025Contrastive Weak-to-strong Generalizationcited by 2
  2. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  3. 2025RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedbackcited by 0
  4. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  5. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  2. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  5. 2024Investigating on RLHF methodology
  6. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  7. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques