mlsanityData to 7 Oct 2026

Knowledge Fusion of Large Language Models

arXiv 2401.10491 ↗First posted 19 Jan 2024cs.CLcited by 110

Not on the map. Why?

Abstract (arXiv)

While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at urlhttps://github.com/fanqiwan/FuseLLM.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

45 of 63 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
33 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2023StarCoder: may the source be with you!cited by 790
  13. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  14. 2023Instruction Tuning with GPT-4cited by 676
  15. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  16. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  17. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  18. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  19. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  20. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  21. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  22. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  23. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  24. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  25. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  26. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  27. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  28. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  29. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  30. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  31. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  32. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  33. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 42; 2026: 24.

110 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2024Large Language Models: A Surveycited by 506
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  5. 2024More Agents Is All You Needcited by 174
  6. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  7. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  8. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  9. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  10. 2024Parameter Competition Balancing for Model Mergingcited by 71
  11. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  12. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
38 more
  1. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  2. 2024What Matters for Model Merging at Scale?cited by 52
  3. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  4. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  5. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  6. 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
  7. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  8. 2024Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensemblingcited by 27
  9. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  10. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  11. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  12. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  13. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
  14. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  15. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  16. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  17. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  18. 2024Distilling Reasoning Ability from Large Language Models with Adaptive Thinkingcited by 13
  19. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  20. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 12
  21. 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
  22. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  23. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  24. 2024Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wildcited by 9
  25. 2024Efficient Dynamic Ensembling for Multiple LLM Expertscited by 9
  26. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  27. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  28. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 8
  29. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  30. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  31. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  32. 2025Batched Self-Consistency Improves LLM Relevance Assessment and Rankingcited by 8
  33. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  34. 2024Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Modelscited by 7
  35. 2024Unconstrained Model Merging for Enhanced LLM Reasoningcited by 7
  36. 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Modelscited by 7
  37. 2024Knowledge Fusion of Chat LLMs: A Preliminary Technical Reportcited by 6
  38. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 6

50 most cited of 110 All 110 (JSON)