mlsanityData to 13 Oct 2026

DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling

arXiv 2406.11617 ↗First posted 17 Jun 2024cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

With the proliferation of domain-specific models, model merging has emerged as a set of techniques that combine the capabilities of multiple models into one that can multitask without the cost of additional training. In this paper, we propose a new model merging technique, Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE, which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the parameters in order of their magnitude and assigns higher dropout probabilities (p) to parameters with lower ranks corresponding to lower magnitudes. To approximate the original embeddings, MAGPRUNE employs a rescaling operation on the parameters that survive the random dropping by 1/(1 - p). On three different expert models considered for merging (LM, Math, Code) and corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an average improvement of 2.4 points over baseline methods employing delta parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over DARE), and 11.1 points over the no-pruning baseline (TA). We release the source code at: https://github.com/declare-lab/della.

Key facts. Cited by 80 papers in the map (data through 13 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

22 of 34 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
  8. 2021Program Synthesis with Large Language Modelscited by 3,573
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,257
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 731
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 687
10 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 621
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 617
  3. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 602
  4. 2018Rethinking the Value of Network Pruningcited by 597
  5. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 381
  7. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  8. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 123
  9. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  10. 2024DPPA: Pruning Method for Large Language Model to Model Mergingcited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 18; 2025: 25; 2026: 37.

80 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 251
  2. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  3. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  4. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  5. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  6. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  7. 2024Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Mergingcited by 18
  8. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  9. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 12
  10. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  11. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  12. 2026AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agentscited by 12
38 more
  1. 2025From Task-Specific Models to Unified Systems: A Review of Model Merging Approachescited by 11
  2. 2024It's Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimizationcited by 10
  3. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 10
  4. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
  5. 2025Subspace-Boosted Model Mergingcited by 9
  6. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  7. 2024MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Modelscited by 7
  8. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  9. 2024ChipAlign: Instruction Alignment in Large Language Models for Chip Design via Geodesic Interpolationcited by 6
  10. 2025Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Modelscited by 6
  11. 2025Training-free LLM Merging for Multi-task Learningcited by 6
  12. 2026Demystifying Mergeability: Interpretable Properties to Predict Model Merging Successcited by 6
  13. 2025FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuningcited by 5
  14. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 5
  15. 2026ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimationcited by 5
  16. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
  17. 2025The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Mergingcited by 4
  18. 2026AttnDiff: Attention-based Differential Fingerprinting for Large Language Modelscited by 4
  19. 2024Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automationcited by 3
  20. 2025LEWIS (LayEr WIse Sparsity) -- A Training Free Guided Model Merging Approachcited by 3
  21. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 3
  22. 2024A Unified View of Delta Parameter Editing in Post-Trained Large-Scale Modelscited by 2
  23. 2025Optimal Brain Iterative Merging: Mitigating Interference in LLM Mergingcited by 2
  24. 2025Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolationcited by 2
  25. 2025ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Mergingcited by 2
  26. 2025Model Assembly Learning with Heterogeneous Layer Weight Mergingcited by 2
  27. 2025AMELIA: A Family of Multi-task End-to-end Language Models for Argumentationcited by 2
  28. 2025From Construction to Injection: Edit-Based Fingerprints for Large Language Modelscited by 2
  29. 2025Defending Unauthorized Model Merging via Dual-Stage Weight Protectioncited by 2
  30. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  31. 2026Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropycited by 2
  32. 2026Can Heterogeneous Language Models Be Fused?cited by 2
  33. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
  34. 2026Conflict-Free Replicated Data Types for Neural Network Model Merging: A Two-Layer Architecture Enabling CRDT-Compliant Model Merging Across 26 Strategiescited by 2
  35. 2026Predicting Mergeability of Parameter-Efficient Fine-Tuning Updatescited by 2
  36. 2024SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptationcited by 1
  37. 2026Multi-task Code LLMs: Data Mix or Model Merge?cited by 1
  38. 2026GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioningcited by 1

50 most cited of 80 All 80 (JSON)