mlsanityData to 7 Oct 2026

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

arXiv 2401.12200 ↗First posted 22 Jan 2024cs.CL cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Fine-tuning and inference with large Language Models (LM) are generally known to be expensive. Parameter-efficient fine-tuning over pretrained LMs reduces training memory by updating a small number of LM parameters but does not improve inference efficiency. Structured pruning improves LM inference efficiency by removing consistent parameter blocks, yet often increases training memory and time. To improve both training and inference efficiency, we introduce APT that adaptively prunes and tunes parameters for the LMs. At the early stage of fine-tuning, APT dynamically adds salient tuning parameters for fast and accurate convergence while discarding unimportant parameters for efficiency. Compared to baselines, our experiments show that APT maintains up to 98% task performance when pruning RoBERTa and T5 models with 40% parameters left while keeping 86.4% LLaMA models' performance with 70% parameters remained. Furthermore, APT speeds up LMs fine-tuning by up to 8x and reduces large LMs memory training footprint by up to 70%.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 50 of its 70 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

50 of 70 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  14. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  16. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  17. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  18. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  19. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  20. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  21. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  22. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  23. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  24. 2021Block Pruning For Faster Transformerscited by 176
  25. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  26. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  27. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  28. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  29. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  30. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  31. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  32. 2022Structural Pruning via Latency-Saliency Knapsackcited by 40
  33. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  34. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  35. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  36. 2018Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmarkcited by 21
  37. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  38. 2023CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devicescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 15; 2026: 3.

29 papers.

Sort:

By area:

  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  3. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  5. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  6. 2024Federated LoRA with Sparse Communicationcited by 26
  7. 2025Pack and Force Your Memory: Long-form and Consistent Video Generationcited by 26
  8. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
  9. 2024Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruningcited by 15
  10. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  11. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  12. 2025EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Modelscited by 7
17 more
  1. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  2. 2025Olica: Efficient Structured Pruning of Large Language Models without Retrainingcited by 6
  3. 2024Sparsity-Accelerated Training for Large Language Modelscited by 5
  4. 2024FineGates: LLMs Finetuning with Compression using Stochastic Gatescited by 3
  5. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  6. 2025LoRAFusion: Efficient LoRA Fine-Tuning for LLMscited by 3
  7. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  8. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  9. 2025AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Researchcited by 2
  10. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  11. 2025MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMscited by 1
  12. 2025High-Layer Attention Pruning with Rescalingcited by 1
  13. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsitycited by 1
  14. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 1
  15. 2026An Adapter-free Fine-tuning Approach for Tuning 3D Foundation Modelscited by 1
  16. 2024FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routingcited by 0
  17. 2025CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devicescited by 0