mlsanityData to 7 Oct 2026

BYOM: Building Your Own Multi-Task Model For Free

arXiv 2310.01886 ↗First posted 3 Oct 2023cs.LG cs.CL cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining. However, existing methods suffer from a large performance deterioration compared to using multiple task-specific models. In this paper, we propose to inject task-specific knowledge into the merged model and design two parameter-efficient approaches (BYOM-FFT and BYOM-LoRA) to Build Your Own Multi-task model. BYOM-FFT is for merging fully finetuned models, while BYOM-LoRA is for LoRA-finetuned models. Both methods are data-free and computation-efficient. Extensive experiments on computer vision and natural language processing tasks show that the proposed BYOM methods outperform existing merging methods by a large margin. Moreover, BYOM-FFT is general and can be integrated into existing merging methods to further boost performance.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

37 of 57 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022A ConvNet for the 2020scited by 3,545
25 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2017A Survey on Multi-Task Learningcited by 838
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2018Rethinking the Value of Network Pruningcited by 595
  13. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  14. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  15. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  16. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  17. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  18. 2021Meta-learning via Language Model In-context Tuningcited by 163
  19. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  20. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  21. 2019Pruning from Scratchcited by 72
  22. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  23. 2021Multi-Objective Meta Learningcited by 39
  24. 2023Dual-Balancing for Multi-Task Learningcited by 27
  25. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  2. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  3. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  4. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 18
  5. 2024Continuous Language Model Interpolation for Dynamic and Controllable Text Generationcited by 2