mlsanityData to 7 Oct 2026

COPAL: Continual Pruning in Large Language Generative Models

arXiv 2405.02347 ↗First posted 2 May 2024cs.LG cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Adapting pre-trained large language models to different domains in natural language processing requires two key considerations: high computational demands and model's inability to continual adaptation. To simultaneously address both issues, this paper presents COPAL (COntinual Pruning in Adaptive Language settings), an algorithm developed for pruning large language generative models under a continual model adaptation setting. While avoiding resource-heavy finetuning or retraining, our pruning process is guided by the proposed sensitivity analysis. The sensitivity effectively measures model's ability to withstand perturbations introduced by the new dataset and finds model's weights that are relevant for all encountered datasets. As a result, COPAL allows seamless model adaptation to new domains while enhancing the resource efficiency. Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

30 of 36 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
18 more
  1. 2016Progressive Neural Networkscited by 1,332
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2018Experience Replay for Continual Learningcited by 1,074
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  9. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  10. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  13. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  14. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  15. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  16. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  17. 2022Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learningcited by 27
  18. 2022Continual Prune-and-Select: Class-incremental learning with specialized subnetworkscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  2. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  3. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  4. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsitycited by 1
  5. 2025Efficient Large Language Models with Zero-Shot Adjustable Accelerationcited by 0
  6. 2026FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inferencecited by 0