mlsanityData to 14 Oct 2026

Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective

arXiv 2410.15483 ↗First posted 20 Oct 2024cs.LG cs.AI cs.CLcited by 12

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The post-training of LLMs, which typically consists of the supervised fine-tuning (SFT) stage and the preference learning stage (RLHF or DPO), is crucial to effective and safe LLM applications. The widely adopted approach in post-training popular open-source LLMs is to sequentially perform SFT and RLHF/DPO. However, this is suboptimal in terms of SFT and RLHF/DPO trade-off: the LLM gradually forgets about the first stage's training when undergoing the second stage's training. This sequential paradigm persists largely due to its simplicity and modularity, which make it easier to implement and manage at scale despite its limitations. We theoretically prove the sub-optimality of sequential post-training and propose a practical joint post-training framework which has theoretical convergence guarantees and empirically outperforms sequential post-training framework, with up to 23% overall performance improvement across multiple LLM evaluation benchmarks, while having minimal computational overhead. Our code is available at https://github.com/heshandevaka/XRIGHT.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 73 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

59 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  6. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning7 Feb
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  9. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
37 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  5. 2023Instruction Tuning with GPT-4cited by 686
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  11. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  12. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  13. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  14. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  15. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  16. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  17. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  18. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  19. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  20. 2023Nash Learning from Human Feedbackcited by 224
  21. 2023Fine-tuning Language Models for Factualitycited by 221
  22. 2022Multi-Task Learning as a Bargaining Gamecited by 215
  23. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  24. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  25. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  26. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  27. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  28. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  29. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  30. 2023Is RLHF More Difficult than Standard RL?cited by 96
  31. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  32. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  33. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  34. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 35
  35. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  36. 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 30
  37. 2023Fast Trainable Projection for Robust Fine-Tuningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 6.

12 papers.

  1. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 13
  2. 2025Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Modelscited by 13
  3. 2025Emergent Slow Thinking in LLMs as Inverse Tree Freezingcited by 8
  4. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  5. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuningcited by 4
  6. 2025Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thoughtcited by 4
  7. 2026Netflix Artwork Personalization via LLM Post-trainingcited by 1
  8. 2025Joint Flashback Adaptation for Forgetting-Resistant Instruction Tuningcited by 0
  9. 2026Boundary Suppression Asymmetry in Post-trained Assistants: Over-expansion as a Controllability Costcited by 0
  10. 2026ARMOR: Adaptive Retriever Optimization for Low-Resource Telecom Question Answeringcited by 0
  11. 2026Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMscited by 0
  12. 2026Off-Target Effects of Response-Style Alignment in a Korean 27B Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
  3. 2024Reinforcement Learning for LLM Post-Training: A Survey
  4. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
  5. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
  6. 2026Dynamic Minimax Regret Optimization for Robust LLM Post-Training
  7. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  8. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training