mlsanityData to 7 Oct 2026

IPO: Your Language Model is Secretly a Preference Classifier

arXiv 2502.16182 ↗First posted 22 Feb 2025cs.CLcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. While it enables LLMs to achieve human-level alignment, it often incurs significant computational and financial costs due to its reliance on training external reward models or human-labeled preferences. In this work, we propose Implicit Preference Optimization (IPO), an alternative approach that leverages generative LLMs as preference classifiers, thereby reducing the dependence on external human feedback or reward models to obtain preferences. We conduct a comprehensive evaluation on the preference classification ability of LLMs using RewardBench, assessing models across different sizes, architectures, and training levels to validate our hypothesis. Furthermore, we investigate the self-improvement capabilities of LLMs by generating multiple responses for a given instruction and employing the model itself as a preference classifier for Direct Preference Optimization (DPO)-based training. Our findings demonstrate that models trained through IPO achieve performance comparable to those utilizing state-of-the-art reward models for obtaining preferences.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 68 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

51 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 50

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  12. 2022Large Language Models Can Self-Improvecited by 656
  13. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  14. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  15. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  16. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  17. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  18. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  19. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  20. 2023Llemma: An Open Language Model For Mathematicscited by 385
  21. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  22. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  23. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  24. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  25. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  26. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  27. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  28. 2024Can LLM be a Personalized Judge?cited by 99
  29. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  30. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  31. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  32. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  33. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  34. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  35. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  36. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  37. 2024Self-Evolved Reward Learning for LLMscited by 16
  38. 2024Dual Instruction Tuning with Large Language Models for Mathematical Reasoningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  2. 2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMscited by 5
  3. 2026Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoningcited by 5
  4. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  5. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Trainingcited by 2
  6. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Studycited by 1
  7. 2025Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Modelscited by 1
  8. 2026SIDiffAgent: Self-Improving Diffusion Agentcited by 1
  9. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  10. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignmentcited by 1
  11. 2026Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimizationcited by 1
  12. 2025GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMscited by 0
5 more
  1. 2026Analytic Score Optimization for Multi Dimension Video Quality Assessmentcited by 0
  2. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferencescited by 0
  3. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  4. 2026Learning from Saturated Data: Signals Beyond Correctness for LLM Trainingcited by 0
  5. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
  2. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  3. 2024TSO: Self-Training with Scaled Preference Optimization
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2025A Survey of Direct Preference Optimization
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  8. 2026ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment