mlsanityData to 7 Oct 2026

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

arXiv 2311.03099 ↗First posted 6 Nov 2023cs.CL cs.LGcited by 679

Not on the map. Why?

Cited by 42 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio p And REscales the remaining ones by 1 / (1 - p) to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.

Key facts. Cited by 679 papers in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

52 of 74 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
  13. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  14. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  15. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  16. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  17. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  18. 2022Editing Models with Task Arithmeticcited by 1,076
  19. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  20. 2018Neural Network Acceptability Judgmentscited by 952
  21. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  22. 2017A Survey on Multi-Task Learningcited by 838
  23. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  24. 2021GPT Understands, Toocited by 715
  25. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  26. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  27. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  28. 2018Rethinking the Value of Network Pruningcited by 595
  29. 2019The State of Sparsity in Deep Neural Networkscited by 471
  30. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  31. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  32. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  33. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  34. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  35. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  36. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  37. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  38. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 169; 2025: 238; 2026: 267.

679 papers.

Sort:

By area:

  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  5. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  6. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  7. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  8. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  9. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  10. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  11. 2024Model merging with SVD to tie the Knotscited by 118
  12. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
38 more
  1. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 108
  2. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  3. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  4. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 90
  5. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  6. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  7. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  8. 2024Parameter Competition Balancing for Model Mergingcited by 71
  9. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  10. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  11. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
  12. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  13. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  14. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  15. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  16. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  17. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  18. 2024What Matters for Model Merging at Scale?cited by 52
  19. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
  20. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  21. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  22. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
  23. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  24. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  25. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  26. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  27. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  28. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  29. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  30. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  31. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  32. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  33. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  34. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
  35. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  36. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  37. 2024Model Extrapolation Expedites Alignmentcited by 37
  38. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37

50 most cited of 679 All 679 (JSON)