mlsanityData to 7 Oct 2026

FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion

arXiv 2406.03280 ↗First posted 5 Jun 2024cs.LG cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Deep model fusion is an emerging technique that unifies the predictions or parameters of several deep neural networks into a single better-performing model in a cost-effective and data-efficient manner. Although a variety of deep model fusion techniques have been introduced, their evaluations tend to be inconsistent and often inadequate to validate their effectiveness and robustness. We present FusionBench, the first benchmark and a unified library designed specifically for deep model fusion. Our benchmark consists of multiple tasks, each with different settings of models and datasets. This variety allows us to compare fusion methods across different scenarios and model scales. Additionally, FusionBench serves as a unified library for easy implementation and testing of new fusion techniques. FusionBench is open source and actively maintained, with community contributions encouraged. Homepage https://github.com/tanganke/fusion_bench

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 65 of its 102 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

65 of 102 references.

References · 65

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2022Editing Models with Task Arithmeticcited by 1,076
38 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  9. 2021Fast Model Editing at Scalecited by 491
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  12. 2018Deep Learning for Classical Japanese Literaturecited by 415
  13. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  14. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  15. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  16. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  17. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  18. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  19. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  20. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  21. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  22. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  23. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  24. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  25. 2022Relative representations enable zero-shot latent space communicationcited by 175
  26. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  27. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  28. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  29. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  30. 2024Knowledge Fusion of Large Language Modelscited by 110
  31. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  32. 2023Deep Model Fusion: A Surveycited by 89
  33. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  34. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  35. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  36. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  37. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  38. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 12
  2. 2025RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Mergingcited by 10
  3. 2025Over-the-Air Edge Inference via End-to-End Metasurfaces-Integrated Artificial Neural Networkscited by 6
  4. 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
  5. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  6. 2026E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoringcited by 4
  7. 2024SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptationcited by 1
  8. 2026Unlocking the Potential of Continual Model Merging: An ODE Perspectivecited by 1
  9. 2026CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocationcited by 0
  10. 2026SMAT: Simple and Efficient Merge-Aware Trainingcited by 0