mlsanityData to 14 Oct 2026

FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion

arXiv 2503.04222 ↗First posted 6 Mar 2025cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce FuseChat-3.0, a suite of large language models (LLMs) developed by integrating the strengths of heterogeneous source LLMs into more compact target LLMs. Our source models include the powerful Gemma-2-27B-it, Mistral-Large-Instruct-2407, Qwen-2.5-72B-Instruct, and Llama-3.1-70B-Instruct. For target models, we focus on three widely-used smaller variants-Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen-2.5-7B-Instruct-along with two ultra-compact options, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct. To leverage the diverse capabilities of these source models, we develop a specialized data construction protocol tailored to various tasks and domains. The FuseChat-3.0 training pipeline consists of two key stages: (1) supervised fine-tuning (SFT) to align the target and source model distributions, and (2) Direct Preference Optimization (DPO) to apply preferences from multiple source LLMs to fine-tune the target model. The resulting FuseChat-3.0 models exhibit significant performance gains across tasks such as instruction following, general knowledge, mathematics, and coding. As illustrated in Figure 1, using Llama-3.1-8B-Instruct as the target model, our fusion approach achieves an average improvement of 6.8 points across 14 benchmarks. Moreover, it demonstrates remarkable gains of 37.1 points and 30.1 points on the instruction-following benchmarks AlpacaEval-2 and Arena-Hard, respectively. Our code, models, and datasets are available at https://github.com/SLIT-AI/FuseChat-3.0.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 55 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

48 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Weighted-Reward Preference Optimization for Implicit Model Fusion4 Dec
  2. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Models30 Nov
  7. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
28 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  14. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  15. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  16. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  17. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  18. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  19. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  20. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  21. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  22. 2024Are We Done with MMLU?cited by 194
  23. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  24. 2024Knowledge Fusion of Large Language Modelscited by 115
  25. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 64
  26. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 55
  27. 2024Bridging the Gap between Different Vocabularies for LLM Ensemblecited by 45
  28. 2024Training-free Heterogeneous Model Mergingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  2. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
  3. 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Explorationcited by 3
  4. 2025DecIF: Improving Instruction-Following through Meta-Decompositioncited by 1
  5. 2026Dynamic Model Merging Made Slimcited by 1
  6. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion
  2. 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
  3. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
  4. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
  5. 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
  8. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey