mlsanityData to 14 Oct 2026

ORPO: Monolithic Preference Optimization without Reference Model

arXiv 2403.07691 ↗First posted 12 Mar 2024cs.CL cs.AIcited by 562

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on AlpacaEval2.0 (Figure 1), 66.19% on IFEval (instruction-level loose, Table 6), and 7.32 in MT-Bench (Figure 12). We release code and model checkpoints for Mistral-ORPO-α (7B) and Mistral-ORPO-β (7B).

Key facts. Cited by 562 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 67 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

42 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference15 Nov
  3. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023DavIR: Data Selection via Implicit Reward for Large Language Models16 Oct
  6. 2023Preference Ranking Optimization for Human Alignment30 Jun
  7. 2023LIMA: Less Is More for Alignment18 May
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
22 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  13. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  14. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  15. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  16. 2023Fine-tuning Language Models for Factualitycited by 221
  17. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  18. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  19. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  20. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  21. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  22. 2024Reinforcement learning for question answering in programming domain using public community scoring as a human feedbackcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 134; 2025: 247; 2026: 181.

562 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  5. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  7. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  9. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 172
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
38 more
  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  3. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  5. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  6. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  7. 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilitiescited by 69
  8. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  9. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  10. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  11. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  12. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  13. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  14. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  15. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  16. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  17. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  18. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 46
  19. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  20. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  21. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 40
  22. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  23. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  24. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  25. 2025A Survey of Direct Preference Optimizationcited by 36
  26. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  27. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  28. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  29. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  30. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  31. 2024Weak-to-Strong Reasoningcited by 33
  32. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimizationcited by 32
  33. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 32
  34. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  35. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  36. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
  37. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  38. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29

50 most cited of 562 All 562 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  2. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  3. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  4. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  5. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  6. 2024ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization