mlsanityData to 7 Oct 2026

Model Extrapolation Expedites Alignment

arXiv 2404.16792 ↗First posted 25 Apr 2024cs.LG cs.AI cs.CLcited by 37

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Given the high computational cost of preference alignment training of large language models (LLMs), exploring efficient methods to reduce the training overhead remains an important and compelling research problem. Motivated by the observation that alignment training typically involves only small parameter changes without injecting new knowledge into models, we propose a straightforward method called ExPO (model extrapolation) to expedite LLMs' alignment with human preferences. Given a partially-trained model and its initial SFT checkpoint, ExPO improves the implicit optimization objective of alignment training by simply amplifying the parameter change based on a first-order approximation, without any additional training overhead. Through controlled experiments, we demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one. Moreover, we show that ExPO notably improves existing open-source LLMs (ranging from 1.8B to 70B parameters) on the leading AlpacaEval 2.0 and MT-Bench benchmarks, which highlights ExPO's broader utility in efficiently enhancing LLM alignment.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 55 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

34 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024Decoding-time Realignment of Language Models5 Feb
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023Mitigating the Alignment Tax of RLHF12 Sep
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
16 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  12. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  13. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  14. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 130
  15. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  16. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 8; 2026: 8.

37 papers.

Sort:

By area:

  1. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  2. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  3. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  4. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  5. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  6. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  7. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  8. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  9. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  10. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  11. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  12. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
25 more
  1. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  2. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  3. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  4. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  5. 2024Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LMcited by 11
  6. 2024Yi-Lightning Technical Reportcited by 11
  7. 2024ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantationcited by 8
  8. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  9. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  10. 2024Accelerated Preference Optimization for Large Language Model Alignmentcited by 5
  11. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5
  12. 2026REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillationcited by 3
  13. 2024A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Modelscited by 2
  14. 2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 2
  15. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  16. 2026You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectoriescited by 2
  17. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
  18. 2024Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasetscited by 1
  19. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolationcited by 1
  20. 2025Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Modelscited by 1
  21. 2025Extrapolation Merging: Keep Improving With Extrapolation and Mergingcited by 1
  22. 2024Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Modelscited by 0
  23. 2025Adding Alignment Control to Language Modelscited by 0
  24. 2026Can LLM Safety Be Ensured by Constraining Parameter Regions?cited by 0
  25. 2026The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  2. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024Aligner: Efficient Alignment by Learning to Correct
  8. 2024Efficient Alignment of Large Language Models via Data Sampling