mlsanityData to 7 Oct 2026

SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters

arXiv 2502.00883 ↗First posted 2 Feb 2025cs.LG cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Existing preference optimization objectives for language model alignment require additional hyperparameters that must be extensively tuned to achieve optimal performance, increasing both the complexity and time required for fine-tuning large language models. In this paper, we propose a simple yet effective hyperparameter-free preference optimization algorithm for alignment. We observe that promising performance can be achieved simply by optimizing inverse perplexity, which is calculated as the inverse of the exponentiated average log-likelihood of the chosen and rejected responses in the preference dataset. The resulting simple learning objective, SimPER, is easy to implement and eliminates the need for expensive hyperparameter tuning and a reference model, making it both computationally and memory efficient. Extensive experiments on widely used real-world benchmarks, including MT-Bench, AlpacaEval 2, and 10 key benchmarks of the Open LLM Leaderboard with 5 base models, demonstrate that SimPER consistently and significantly outperforms existing approaches-even without any hyperparameters or a reference model . For example, despite its simplicity, SimPER outperforms state-of-the-art methods by up to 5.7 points on AlpacaEval 2 and achieves the highest average ranking across 10 benchmarks on the Open LLM Leaderboard. The source code for SimPER is publicly available at: https://github.com/tengxiao1/SimPER.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 65 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

50 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  2. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective14 Oct
  3. 2024Understanding Reference Policies in Direct Preference Optimization18 Jul
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  8. 2024Decoding-time Realignment of Language Models5 Feb
  9. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Mistral 7Bcited by 3,065
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
26 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  8. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  11. 2023Scaling Data-Constrained Language Modelscited by 391
  12. 2024Iterative Reasoning Preference Optimizationcited by 244
  13. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  14. 2023Fine-tuning Language Models for Factualitycited by 219
  15. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  16. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  17. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  18. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  19. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  20. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  21. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  22. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  23. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
  24. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  25. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  26. 2024Cost-Effective Hallucination Detection for LLMscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 6.

25 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  3. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  4. 2025EXAONE Deep: Reasoning Enhanced Language Modelscited by 20
  5. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  6. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  7. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  8. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignmentcited by 7
  9. 2025Inference-time Alignment in Continuous Spacecited by 7
  10. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Usecited by 7
  11. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6
  12. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
13 more
  1. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  2. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  3. 2025Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimizationcited by 3
  4. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimizationcited by 3
  5. 2025Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMscited by 2
  6. 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignmentcited by 2
  7. 2026K-EXAONE Technical Reportcited by 2
  8. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shiftcited by 1
  9. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  10. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  11. 2026Intrinsic Mutual Information as a Modulator for Preference Optimizationcited by 0
  12. 2026xi-DPO: Direct Preference Optimization via Ratio Reward Margincited by 0
  13. 2026How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defensecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ComPO: Preference Alignment via Comparison Oracles
  2. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  3. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  4. 2024Aligner: Efficient Alignment by Learning to Correct
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  7. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  8. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO