# LLM preference optimization methods

Area: [LLM alignment / RLHF / preference optimisation](https://mlsanity.com/alignment/index.md). HTML: https://mlsanity.com/alignment/llm-preference-optimization-methods/

## Key facts

- LLM preference optimization methods: 1,132 papers on arXiv since 2017 (data through 13 Oct 2026).
- 736 of its 1,132 mapped papers (65%) cite “Training language models to follow instructions with human feedback” (2022, arXiv 2203.02155).
- 359 of its 1,132 mapped papers (32%) cite “A General Theoretical Paradigm to Understand Learning from Human Preferences” (2023, arXiv 2310.12036).
- 21 new papers 17 Sep–8 Oct.

## Most cited here

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022), cited by 736 papers here
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023), cited by 359 papers here
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024), cited by 293 papers here
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024), cited by 177 papers here
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024), cited by 150 papers here
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023), cited by 131 papers here
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022), cited by 123 papers here
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021), cited by 113 papers here
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024), cited by 103 papers here
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024), cited by 94 papers here

## New (17 Sep–8 Oct)

- [Efficient Best-of-N policy evaluation for inference-time alignment](https://mlsanity.com/paper/2610.09250/) (2026-10-07)
- [CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning](https://mlsanity.com/paper/2610.09219/) (2026-10-06)
- [LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization](https://mlsanity.com/paper/2610.07592/) (2026-10-06)
- [Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution](https://mlsanity.com/paper/2610.04845/) (2026-10-04)
- [Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures](https://mlsanity.com/paper/2610.04328/) (2026-10-03)
- [The Asymptotics of Language Model Alignment with Memory](https://mlsanity.com/paper/2610.01828/) (2026-10-01)
- [GAW-PO: Preference Optimization with Gradient-Aligned Token Weights](https://mlsanity.com/paper/2610.01511/) (2026-10-01)
- [Multi-LLM Collaborative Alignment via Stackelberg Games](https://mlsanity.com/paper/2609.39076/) (2026-09-30)
- [Optimal Design for Active Preference Learning with Biased LLM Judges](https://mlsanity.com/paper/2609.38860/) (2026-09-30)
- [Uncertainty-Normalized Margins for Direct Preference Optimization](https://mlsanity.com/paper/2609.38647/) (2026-09-29)
- [Generalizable Lifelong Model Editing via Preference Optimization](https://mlsanity.com/paper/2609.36748/) (2026-09-29)
- [Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models](https://mlsanity.com/paper/2609.35695/) (2026-09-28)
- [BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2609.35044/) (2026-09-28)
- [Sample What You Say: Aligning Language Models to Sample the Distributions They State](https://mlsanity.com/paper/2609.34929/) (2026-09-28)
- [ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control](https://mlsanity.com/paper/2609.34374/) (2026-09-28)
- [LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations](https://mlsanity.com/paper/2609.32717/) (2026-09-26)
- [The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models](https://mlsanity.com/paper/2609.32617/) (2026-09-26)
- [DimPO: Dimensionality Reduction for Attention using Preference Optimization](https://mlsanity.com/paper/2609.32579/) (2026-09-26)
- [Masking Frequent Tokens Sharpens Direct Preference Optimization](https://mlsanity.com/paper/2609.32445/) (2026-09-26)
- [onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction](https://mlsanity.com/paper/2609.24983/) (2026-09-21)
- [ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment](https://mlsanity.com/paper/2609.21899/) (2026-09-18)