mlsanityData to 7 Oct 2026

DPPA: Pruning Method for Large Language Model to Model Merging

arXiv 2403.02799 ↗First posted 5 Mar 2024cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Model merging is to combine fine-tuned models derived from multiple domains, with the intent of enhancing the model's proficiency across various domains. The principal concern is the resolution of parameter conflicts. A substantial amount of existing research remedy this issue during the merging stage, with the latest study focusing on resolving this issue throughout the pruning stage. The DARE approach has exhibited promising outcomes when applied to a simplistic fine-tuned model. However, the efficacy of this method tends to wane when employed on complex fine-tuned models that show a significant parameter bias relative to the baseline model. In this paper, we introduce a dual-stage method termed Dynamic Pruning Partition Amplification (DPPA), devised to tackle the challenge of merging complex fine-tuned models. Initially, we introduce Dynamically Pruning (DP), an improved approach based on magnitude pruning, which aim is to enhance performance at higher pruning rates. Subsequently, we propose Dynamically Partition Amplification (DPA), a rescaling strategy, is designed to dynamically amplify parameter partitions in relation to their significance levels. The experimental results show that our method maintains a mere 20% of domain-specific parameters and yet delivers a performance comparable to other methodologies that preserve up to 90% of parameters. Furthermore, our method displays outstanding performance post-pruning, leading to a significant improvement of nearly 20% performance in model merging. We make our code on Github.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

28 of 49 references.

References · 28

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  11. 2020Federated Learning with Matched Averagingcited by 564
  12. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
16 more
  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  2. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  3. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  4. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  5. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  6. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  7. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  8. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  9. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  10. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  11. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  12. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  13. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  14. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  15. 2023Task Arithmetic with LoRA for Continual Learningcited by 31
  16. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 2.

7 papers.

  1. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 77
  2. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  3. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  4. 2024Obliviate: Neutralizing Task-agnostic Backdoors within the Parameter-efficient Fine-tuning Paradigmcited by 7
  5. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Modelscited by 4
  6. 2026Sparsity-Aware Evolution for Model Mergingcited by 0
  7. 2026Defense Against LLM Backdoors using Critical Neuron Isolation Pruningcited by 0