mlsanityData to 13 Oct 2026

Learn Your Reference Model for Real Good Alignment

arXiv 2404.09656 ↗First posted 15 Apr 2024cs.LG cs.CLcited by 57

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Despite the fact that offline methods for Large Language Models (LLMs) alignment do not require a direct reward model, they remain susceptible to overoptimization. This issue arises when the trained model deviates excessively from the reference policy, leading to a decrease in sample quality. We propose a new paradigm of offline alignment methods, called Trust Region (including variants TR-DPO, TR-IPO, TR-KTO), which dynamically updates the reference policy throughout the training process. Our results show that TR alignment methods effectively mitigate overoptimization, enabling models to maintain strong performance even when substantially deviating from the initial reference policy. We demonstrate the efficacy of these approaches not only through toy examples that exhibit reduced overoptimization, but also through direct, side-by-side comparisons in specific tasks such as helpful and harmless dialogue, as well as summarization, where they surpass conventional methods. Additionally, we report significant improvements in general-purpose assistant setups with the Llama3 model on the AlpacaEval 2 and Arena-Hard benchmarks, highlighting the advantages of Trust Region methods over classical approaches.

Key facts. Cited by 57 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 69 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

51 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  3. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  4. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  5. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
30 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,804
  3. 2020Learning to summarize from human feedbackcited by 2,575
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  5. 2015Trust Region Policy Optimizationcited by 2,513
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  11. 2022Editing Models with Task Arithmeticcited by 1,101
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,077
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  14. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 877
  15. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  16. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  17. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 705
  18. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  19. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 556
  20. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 486
  21. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 390
  22. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  23. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  24. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 226
  25. 2023Nash Learning from Human Feedbackcited by 223
  26. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  27. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  28. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  29. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 59
  30. 2022Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Modelscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 21; 2025: 24; 2026: 12.

57 papers.

Sort:

By area:

  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  2. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 74
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  4. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  6. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 30
  9. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 28
  10. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  11. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  12. 2024How to Merge Your Multimodal Models Over Time?cited by 23
38 more
  1. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 23
  2. 2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
  3. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  5. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learningcited by 15
  6. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  7. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  8. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  9. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  10. 2025Reinforcement Learning Teachers of Test Time Scalingcited by 10
  11. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  12. 2024Aligning Diffusion Models with Noise-Conditioned Perceptioncited by 9
  13. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  14. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  15. 2024LIONs: An Empirically Optimized Approach to Align Language Modelscited by 7
  16. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 7
  17. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Modelcited by 6
  18. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  19. 2024New Desiderata for Direct Preference Optimizationcited by 5
  20. 2025KL Penalty Control via Perturbation for Direct Preference Optimizationcited by 5
  21. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 5
  22. 2024SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHFcited by 4
  23. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularizationcited by 4
  24. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  25. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  26. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
  27. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  28. 2025On the Interplay of Human-AI Alignment,Fairness, and Performance Trade-offs in Medical Imagingcited by 3
  29. 2025Rethinking Direct Preference Optimization in Diffusion Modelscited by 3
  30. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  31. 2024Online Joint Fine-tuning of Multi-Agent Flowscited by 2
  32. 2025Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Provingcited by 2
  33. 2025AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignmentcited by 2
  34. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  35. 2025Hybrid Explanation-Guided Learning for Transformer-Based Chest X-Ray Diagnosiscited by 1
  36. 2025Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimizationcited by 1
  37. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  38. 2026Reward-guided Fine-Tuning of One-Step Generative Models via Wasserstein Gradient Flowcited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Training-free Alignment of Large Language Models
  2. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?
  5. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
  6. 2024Aligner: Efficient Alignment by Learning to Correct
  7. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  8. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models