mlsanityData to 7 Oct 2026

SimPO: Simple Preference Optimization with a Reference-Free Reward

arXiv 2405.14734 ↗First posted 23 May 2024cs.CL cs.LGcited by 951

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the average log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Gemma-2-9B-it, achieves a 72.4% length-controlled win rate on AlpacaEval 2, a 59.1% win rate on Arena-Hard, and ranks 1st on Chatbot Arena among <10B models with real user votes.

Key facts. Cited by 951 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 74 of its 99 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

74 of 99 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization31 Mar
  4. 2024sDPO: Don't Use Your Data All at Once28 Mar
  5. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  8. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  9. 2024Direct Preference Optimization with an Offset16 Feb
  10. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  11. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
8 more
  1. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  2. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2023Preference Ranking Optimization for Human Alignment30 Jun
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  7. 2022Training language models to follow instructions with human feedback4 Mar
  8. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2016An overview of gradient descent optimization algorithmscited by 1,021
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  14. 2024Self-Rewarding Language Modelscited by 654
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  16. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  17. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  18. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  19. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  20. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  21. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  22. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  23. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  24. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  25. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  26. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  27. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  28. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  29. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  30. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  31. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  32. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  33. 2023Pretraining Language Models with Human Preferencescited by 245
  34. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  35. 2023Fine-tuning Language Models for Factualitycited by 219
  36. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  37. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  38. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 211; 2025: 452; 2026: 288.

951 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  5. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  6. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  7. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  8. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  9. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  11. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 278
  12. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
38 more
  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  4. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  5. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  7. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 154
  8. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  9. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  10. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  11. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  12. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  13. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  14. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  15. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 115
  16. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  17. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  18. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  19. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  20. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  21. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  22. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  23. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  24. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  25. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  26. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  27. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  28. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  29. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  30. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  31. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  32. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  33. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  34. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  35. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  36. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  37. 2025Self-rewarding correction for mathematical reasoningcited by 67
  38. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66

50 most cited of 951 All 951 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  2. 2025Active Learning for Direct Preference Optimization
  3. 2026GOPO: Policy Optimization using Ranked Rewards
  4. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  5. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions
  6. 2024New Desiderata for Direct Preference Optimization
  7. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF
  8. 2024Filtered Direct Preference Optimization