mlsanityData to 7 Oct 2026

Preference Ranking Optimization for Human Alignment

arXiv 2306.17492 ↗First posted 30 Jun 2023cs.CL cs.AIcited by 283

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, we propose Preference Ranking Optimization (PRO) as an efficient SFT algorithm to directly fine-tune LLMs for human alignment. PRO extends the pair-wise contrast to accommodate preference rankings of any length. By iteratively contrasting candidates, PRO instructs the LLM to prioritize the best response while progressively ranking the rest responses. In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses. Experiments have shown that PRO outperforms baseline algorithms, achieving comparable results to ChatGPT and human responses through automatic-based, reward-based, GPT-4, and human evaluations.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 52 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

40 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
24 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2023Scaling Data-Constrained Language Modelscited by 391
  13. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  14. 2023Reward Design with Language Modelscited by 257
  15. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  16. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  17. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  18. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  19. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  20. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  21. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  22. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  23. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  24. 2022Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policycited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 139; 2025: 77; 2026: 25.

283 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  9. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
38 more
  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  5. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  6. 2024Token-level Direct Preference Optimizationcited by 135
  7. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  8. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  9. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  10. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  11. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  12. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  13. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  14. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  15. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  16. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  17. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  18. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  19. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  20. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  21. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  22. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  23. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  24. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  25. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  26. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  27. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  28. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 62
  29. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  30. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  31. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  32. 2023Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observationscited by 58
  33. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  34. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  35. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  36. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  37. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  38. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  2. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  5. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  8. 2024CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models