Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Instruct LLM provide a paradigm used in large scale language model to align LLM to human preference. The paradigm contains supervised fine tuning and reinforce learning from human feedback. This paradigm is also used in downstream scenarios to adapt LLM to specific corpora and applications. Comparing to SFT, there are many efforts focused on RLHF and several algorithms being proposed, such as PPO, DPO, IPO, KTO, MinorDPO and etc. Meanwhile most efforts for SFT are focused on how to collect, filter and mix high quality data. In this article with insight from DPO and MinorDPO, we propose a training metric for SFT to measure the discrepancy between the optimized model and the original model, and a loss function MinorSFT that can increase the training effectiveness, and reduce the discrepancy between the optimized LLM and original LLM.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 13 references that are in the map. First posted on arXiv 20 Aug 2024.
Builds on
11 of 13 references.
Closest ancestors · 5
Same direction, newest first
- 2024Minor DPO reject penalty to increase training robustness19 Aug
- 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 6
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
- 2024Iterative Reasoning Preference Optimizationcited by 244
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
- 2025DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agentscited by 2
- 2025InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behaviorcited by 1
- 2026KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learningcited by 0
- 2026KALE: Enhancing Knowledge Manipulation in Large Language Models via Knowledge-aware Learning12 Jan 2026
- 2025DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents22 Oct 2025
- 2025InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior9 Jul 2025
Similar, not cited
Close in wording, no citation either way
- 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
- 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
- 2024PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning
- 2025Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M
- 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
- 2024Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities
- 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
- 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback