mlsanityData to 14 Oct 2026

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

arXiv 2402.10958 ↗First posted 12 Feb 2024cs.CL cs.AI cs.LGcited by 36

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pairs of preferences derived from the same prompts, and it functions without needing an additional reward model. However, DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical but also similar questions. To overcome this shortfall, we propose Relative Preference Optimization (RPO). RPO is designed to discern between more and less preferred responses derived from both identical and related prompts. It introduces a contrastive weighting mechanism, enabling the tuning of LLMs using a broader range of preference data, including both paired and unpaired sets. This approach expands the learning capabilities of the model, allowing it to leverage insights from a more varied set of prompts. Through empirical tests, including dialogue and summarization tasks, and evaluations using the AlpacaEval2.0 leaderboard, RPO has demonstrated a superior ability to align LLMs with user preferences and to improve their adaptability during the training process. Our code can be viewed at https://github.com/yinyueqin/relative-preference-optimization

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 48 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

37 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Mistral 7Bcited by 3,164
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
21 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2018Computational Optimal Transportcited by 780
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  12. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  13. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  14. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  15. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  16. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  17. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  18. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  19. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  20. 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
  21. 2020Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributionscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 12.

36 papers.

Sort:

By area:

  1. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  2. 2025A Survey of Direct Preference Optimizationcited by 36
  3. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimizationcited by 32
  4. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 19
  5. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
  6. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 18
  7. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimizationcited by 15
  8. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  9. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  10. 2026Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Trainingcited by 11
  11. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  12. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
24 more
  1. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processcited by 8
  2. 2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignmentcited by 8
  3. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
  4. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  5. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignmentcited by 7
  6. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgmentscited by 7
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  8. 2025Contrastive Representations for Temporal Reasoningcited by 6
  9. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUscited by 4
  10. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  11. 2026Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graphcited by 2
  12. 2026AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectoriescited by 2
  13. 2024Integrating Emotional and Linguistic Models for Ethical Compliance in Large Language Modelscited by 1
  14. 2025SOLAR: Scalable Optimization of Large-scale Architecture for Reasoningcited by 1
  15. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPOcited by 1
  16. 2025STELAR-VISION: Self-Topology-Aware Efficient Learning for Aligned Reasoning in Visioncited by 0
  17. 2026Elo-Evolve: A Co-evolutionary Framework for Language Model Alignmentcited by 0
  18. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modelingcited by 0
  19. 2026RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflectioncited by 0
  20. 2026VisualDeltas: Learning Preferences from Visual Quality Perturbationscited by 0
  21. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  22. 2026DongYuan: An LLM-Based Framework for Integrative Chinese and Western Medicine Spleen-Stomach Disorders Diagnosiscited by 0
  23. 2026Generating Place-Based Compromises Between Two Points of Viewcited by 0
  24. 2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  2. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  3. 2025ComPO: Preference Alignment via Comparison Oracles
  4. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  5. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  6. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization