mlsanityData to 14 Oct 2026

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks

arXiv 2312.06795 ↗First posted 11 Dec 2023cs.LGcited by 133

Not on the map. Why?

Abstract (arXiv)

The rapid development of AI systems has been greatly influenced by the emergence of foundation models. A common approach for targeted problems involves fine-tuning these pre-trained foundation models for specific target tasks, resulting in a rapid spread of models fine-tuned across a diverse array of tasks. This work focuses on the problem of merging multiple fine-tunings of the same foundation model derived from a spectrum of auxiliary tasks. We introduce a new simple method, Model Breadcrumbs, which consists of a sparsely defined weight set that guides model adaptation within the weight space of a pre-trained model. These breadcrumbs are constructed by subtracting the weights from a pre-trained model before and after fine-tuning, followed by a sparsification process that eliminates weight outliers and negligible perturbations. Our experiments demonstrate the effectiveness of Model Breadcrumbs to simultaneously improve performance across multiple tasks. This contribution aligns with the evolving paradigm of updatable machine learning, reminiscent of the collaborative principles underlying open-source software development, fostering a community-driven effort to reliably update machine learning models. Our method is shown to be more efficient and unlike previous proposals does not require hyperparameter tuning for each new task added. Through extensive experimentation involving various models, tasks, and modalities we establish that integrating Model Breadcrumbs offers a simple, efficient, and highly effective approach for constructing multi-task models and facilitating updates to foundation models.

Key facts. Cited by 133 papers in the map (data through 14 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

39 of 60 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
27 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  7. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  8. 2021Robust fine-tuning of zero-shot modelscited by 797
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  11. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  12. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  13. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
  14. 2020What is being transferred in transfer learning?cited by 368
  15. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  16. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  17. 2022Patching open-vocabulary models by interpolating weightscited by 190
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  19. 2022Fusing finetuned models for better pretrainingcited by 110
  20. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  21. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  22. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 76
  23. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  24. 2023Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learningcited by 35
  25. 2021C5T5: Controllable Generation of Organic Molecules with Transformerscited by 16
  26. 2023Guiding The Last Layer in Federated Learning with Pre-Trained Modelscited by 13
  27. 2023ToxBuster: In-game Chat Toxicity Buster with BERTcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 31; 2025: 55; 2026: 47.

133 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  3. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 147
  4. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  5. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  6. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  7. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  8. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
  9. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  10. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 32
  11. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  12. 2025Activation-Informed Merging of Large Language Modelscited by 30
38 more
  1. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 29
  2. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  3. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  4. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  5. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  6. 2025Steering Language Models with Weight Arithmeticcited by 20
  7. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 19
  8. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 18
  9. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 18
  10. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  11. 2024SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Mergingcited by 16
  12. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  13. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  14. 2024Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfercited by 13
  15. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 13
  16. 2024Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMscited by 12
  17. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  18. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  19. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  20. 2025Overcoming the Communication-Performance Tradeoff in LLM Pretrainingcited by 12
  21. 2025Model Merging Scaling Laws in Large Language Modelscited by 12
  22. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11
  23. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 11
  24. 2025STAR: Spectral Truncation and Rescale for Model Mergingcited by 11
  25. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  26. 2025MERGE3: Efficient Evolutionary Merging on Consumer-grade GPUscited by 10
  27. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 10
  28. 2025Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspectivecited by 10
  29. 2024Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy?cited by 9
  30. 2025Subspace-Boosted Model Mergingcited by 9
  31. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  32. 2025CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Mergingcited by 8
  33. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 8
  34. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusioncited by 8
  35. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  36. 2024MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Modelscited by 7
  37. 2024Unconstrained Model Merging for Enhanced LLM Reasoningcited by 7
  38. 2025LED-Merging: Mitigating Safety-Utility Conflicts in Model Merging with Location-Election-Disjointcited by 7

50 most cited of 133 All 133 (JSON)