mlsanityData to 7 Oct 2026

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey

arXiv 2409.11564 ↗First posted 17 Sep 2024cs.CL cs.AI cs.CVcited by 23

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference tuning is a crucial process for aligning deep generative models with human preferences. This survey offers a thorough overview of recent advancements in preference tuning and the integration of human feedback. The paper is organized into three main sections: 1) introduction and preliminaries: an introduction to reinforcement learning frameworks, preference tuning tasks, models, and datasets across various modalities: language, speech, and vision, as well as different policy approaches, 2) in-depth exploration of each preference tuning approach: a detailed analysis of the methods used in preference tuning, and 3) applications, discussion, and future directions: an exploration of the applications of preference tuning in downstream tasks, including evaluation methods for different modalities, and an outlook on future research directions. Our objective is to present the latest methodologies in preference tuning and model alignment, enhancing the understanding of this field for researchers and practitioners. We hope to encourage further engagement and innovation in this area.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 173 of its 224 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

173 of 224 references.

Closest ancestors · 30

Same direction, newest first

  1. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  2. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment18 Jun
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  4. 2024OPTune: Efficient Online Preference Tuning11 Jun
  5. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  8. 2024Model Extrapolation Expedites Alignment25 Apr
  9. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  10. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  11. 2024sDPO: Don't Use Your Data All at Once28 Mar
  12. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
18 more
  1. 2024InternLM2 Technical Report26 Mar
  2. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences14 Mar
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  6. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  7. 2024Direct Preference Optimization with an Offset16 Feb
  8. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  9. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  12. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization12 Jan
  13. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  14. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  15. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  16. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  17. 2023LIMA: Less Is More for Alignment18 May
  18. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 143

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2024The Llama 3 Herd of Modelscited by 14,192
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Mistral 7Bcited by 3,065
  13. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  14. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  15. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  16. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  17. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  18. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  19. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  20. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  21. 2024Mixtral of Expertscited by 1,785
  22. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  23. 2020GLU Variants Improve Transformercited by 1,724
  24. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  25. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  26. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  27. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  28. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  29. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  30. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  31. 2023PaLM 2 Technical Reportcited by 1,154
  32. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  33. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  34. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  35. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  36. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  37. 2023BloombergGPT: A Large Language Model for Financecited by 923
  38. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893

50 most cited of 143 All 143 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 6.

23 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  3. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  4. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  5. 2025DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learningcited by 17
  6. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  7. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  8. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 10
  9. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  10. 2024MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibrationcited by 6
  11. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual contextcited by 6
  12. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
11 more
  1. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  2. 2024HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuningcited by 4
  3. 2025RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferencescited by 4
  4. 2025Diffusion Generative Models Meet Compressed Sensing, with Applications to Imaging and Financecited by 3
  5. 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipelinecited by 3
  6. 2024Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Modelscited by 2
  7. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  8. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  9. 2025Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processescited by 0
  10. 2026Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Studycited by 0
  11. 2026DOHF: Online Diffusion Fine-tuning with Doob's h-transform Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
  4. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling
  5. 2024New Desiderata for Direct Preference Optimization
  6. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier