mlsanityData to 7 Oct 2026

PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning

arXiv 2406.17923 ↗First posted 25 Jun 2024cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) have shown remarkable abilities in diverse natural language processing (NLP) tasks. The LLMs generally undergo supervised fine-tuning (SFT) followed by preference alignment to be usable in downstream applications. However, this sequential training pipeline leads to alignment tax that degrades the LLM performance. This paper introduces PAFT, a new PArallel training paradigm for effective LLM Fine-Tuning, which independently performs SFT and preference alignment (e.g., DPO and ORPO, etc.) with the same pre-trained model on respective datasets. The model produced by SFT and the model from preference alignment are then merged into a final model by parameter fusing for use in downstream applications. This work reveals important findings that preference alignment like DPO naturally results in a sparse model while SFT leads to a natural dense model which needs to be sparsified for effective model merging. This paper introduces an effective interference resolution which reduces the redundancy by sparsifying the delta parameters. The LLM resulted from the new training paradigm achieved Rank #1 on the HuggingFace Open LLM Leaderboard. Comprehensive evaluation shows the effectiveness of the parallel training paradigm.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 35 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

26 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
11 more
  1. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  2. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  4. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  5. 2023A Survey on Model Compression for Large Language Modelscited by 263
  6. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  8. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  9. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  10. 2023Reward Collapse in Aligning Large Language Modelscited by 23
  11. 2021Sparse Structure Learning via Graph Neural Networks for Inductive Document Classificationcited by 15

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 5.

11 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  4. 2025A Survey of Direct Preference Optimizationcited by 35
  5. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  6. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  7. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  8. 2024UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Functioncited by 9
  9. 2024UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Typescited by 6
  10. 2025Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolationcited by 2
  11. 2025Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models
  2. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  3. 2024HFT: Half Fine-Tuning for Large Language Models
  4. 2024ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
  5. 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
  6. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  7. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  8. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks