mlsanityData to 14 Oct 2026

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

arXiv 2505.02363 ↗First posted 5 May 2025cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models with human preferences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off -policy data for preference learning, others indicate that the advantages of on-policy data may be task-dependent, highlighting the need for a systematic exploration of their interplay. In this work, we show that on-policy and off-policy data offer complementary strengths in preference optimization: on-policy data is particularly effective for reasoning tasks like math and coding, while off-policy data performs better on open-ended tasks such as creative writing and making personal recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach to combine the complementary strengths of on-policy and off-policy preference learning by simply mixing these two data sources. Our empirical results across diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it outperforms prior approaches that are much more complex in combining on- and off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 82 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

61 of 82 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following18 Dec
  2. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration13 Dec
  3. 2024Sample-Efficient Alignment for LLMs3 Nov
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  5. 2024The Crucial Role of Samplers in Online Direct Preference Optimization29 Sep
  6. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  7. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  9. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  11. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  12. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
3 more
  1. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
34 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  14. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  15. 2024Self-Rewarding Language Modelscited by 681
  16. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  17. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  18. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  19. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  20. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  21. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  23. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  24. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  25. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  26. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  27. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  28. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 48
  29. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  30. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  31. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  32. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  33. 2024Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPscited by 6
  34. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  2. 2026Raon-Speech Technical Reportcited by 7
  3. 2025ProteinZero: Self-Improving Protein Generation via Online Reinforcement Learningcited by 6
  4. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 1
  5. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  6. 2026Anchors Enhance Controllability and Reasoning of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Large Language Models by On-Policy Self-Judgment
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  4. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  5. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  6. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  7. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
  8. 2025Toward Preference-aligned Large Language Models via Residual-based Model Steering