mlsanityData to 7 Oct 2026

LLM preference optimization methods

1,132 papers3 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: preference, alignment, language, optimization, llms

Key facts. LLM preference optimization methods: 1,132 papers on arXiv since 2017 (data through 7 Oct 2026). 736 of its 1,132 mapped papers (65%) cite “Training language models to follow instructions with human feedback” (2022, arXiv 2203.02155). 359 of its 1,132 mapped papers (32%) cite “A General Theoretical Paradigm to Understand Learning from Human Preferences” (2023, arXiv 2310.12036). 21 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

21 new papers, 17 Sep – 7 Oct; 3 this week.

  1. This week · 3
  2. Efficient Best-of-N policy evaluation for inference-time alignment
  3. CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning also fits: Process reward models for reasoning
  4. LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
  5. Earlier · 18
  6. Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
  7. Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures
  8. The Asymptotics of Language Model Alignment with Memory
  9. GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
  10. Multi-LLM Collaborative Alignment via Stackelberg Games also fits: Process reward models for reasoning
  11. Optimal Design for Active Preference Learning with Biased LLM Judges
  12. Uncertainty-Normalized Margins for Direct Preference Optimization also fits: Reward modeling and RLHF
All 21
  1. Generalizable Lifelong Model Editing via Preference Optimization
  2. Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models also fits: Preference alignment for recommenders and RAG
  3. BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  4. Sample What You Say: Aligning Language Models to Sample the Distributions They State
  5. ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control
  6. LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformations also fits: Human values and AI feedback alignment
  7. The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models also fits: Human values and AI feedback alignment
  8. DimPO: Dimensionality Reduction for Attention using Preference Optimization also fits: Preference alignment for recommenders and RAG
  9. Masking Frequent Tokens Sharpens Direct Preference Optimization
  10. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction also fits: Human values and AI feedback alignment
  11. ExpBoN: Exponential-Noise Best-of-n for Efficient Test-Time LLM Alignment also fits: Process reward models for reasoning

Foundations

Most cited by papers here

  1. Training language models to follow instructions with human feedback2022cited by 736 here
  2. A General Theoretical Paradigm to Understand Learning from Human Preferences2023cited by 359 here
  3. SimPO: Simple Preference Optimization with a Reference-Free Reward2024cited by 293 here
  4. ORPO: Monolithic Preference Optimization without Reference Model2024cited by 177 here
  5. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation2024cited by 150 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 410 here
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 273 here
  3. 2020Learning to summarize from human feedbackcited by 375 here
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 113 here
  5. 2022Training language models to follow instructions with human feedbackcited by 736 here
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 460 here
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 151 here
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 152 here
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 111 here
  10. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 103 here
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 293 here
13 more it builds on, not in this chain
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 99 here
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 194 here
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 123 here
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 133 here
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 174 here
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 804 here
  7. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 131 here
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 98 here
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 359 here
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 96 here
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 150 here
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 94 here
  13. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 177 here

The 24 most cited of 1,274.

Browse

30 of 1,132, ranked 6–35

  1. 6Statistical Rejection Sampling Improves Preference Optimization2023131
  2. 7Self-Instruct: Aligning Language Models with Self-Generated Instructions2022123
  3. 8A General Language Assistant as a Laboratory for Alignment2021113
  4. 9Disentangling Length from Quality in Direct Preference Optimization2024103
  5. 10Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive202494
  6. 11Preference Ranking Optimization for Human Alignment202393
  7. 12Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study202491
  8. 13LIMA: Less Is More for Alignment202387
  9. 14Direct Language Model Alignment from Online AI Feedback202485
  10. 15OpenAssistant Conversations -- Democratizing Large Language Model Alignment202383
All 30
  1. 16Zephyr: Direct Distillation of LM Alignment202382
  2. 17Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints202377
  3. 18Generalized Preference Optimization: A Unified Approach to Offline Alignment202474
  4. 19From r to Q^*: Your Language Model is Secretly a Q-Function202474
  5. 20Direct Preference Optimization with an Offset202459
  6. 21Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards202459
  7. 22Provably Robust DPO: Aligning Language Models with Noisy Feedback202458
  8. 23Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment202456
  9. 24Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision202353
  10. 25Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss202351
  11. 26KTO: Model Alignment as Prospect Theoretic Optimization202449
  12. 27Token-level Direct Preference Optimization202448
  13. 28Aligning Large Language Models with Human: A Survey202348
  14. 29LiPO: Listwise Preference Optimization through Learning-to-Rank202442
  15. 30Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization202342
  16. 31β-DPO: Direct Preference Optimization with Dynamic β202441
  17. 32Understanding the performance gap between online and offline alignment algorithms202440
  18. 33Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment202438
  19. 34The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning202338
  20. 35What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning202336