mlsanityData to 14 Oct 2026

TuneComp: Joint Fine-tuning and Compression for Large Foundation Models

arXiv 2505.21835 ↗First posted 27 May 2025cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

To reduce model size during post-training, compression methods, including knowledge distillation, low-rank approximation, and pruning, are often applied after fine-tuning the model. However, sequential fine-tuning and compression sacrifices performance, while creating a larger than necessary model as an intermediate step. In this work, we aim to reduce this gap, by directly constructing a smaller model while guided by the downstream task. We propose to jointly fine-tune and compress the model by gradually distilling it to a pruned low-rank structure. Experiments demonstrate that joint fine-tuning and compression significantly outperforms other sequential compression methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

33 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation13 Jun

Other references · 32

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Rethinking Attention with Performerscited by 1,601
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  5. 2018Rethinking the Value of Network Pruningcited by 599
  6. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  8. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  9. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  10. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  11. 2023Reward Design with Language Modelscited by 261
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
20 more
  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  2. 2022Language model compression with weighted low-rank factorizationcited by 195
  3. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  4. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  5. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  6. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  7. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  8. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  9. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  10. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  11. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  12. 2020GAN Slimming: All-in-One GAN Compression by A Unified Optimization Frameworkcited by 44
  13. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  14. 2021Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillationcited by 23
  15. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  16. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  17. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformerscited by 14
  18. 2017Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimizationcited by 8
  19. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  20. 2024Computer Vision Model Compression Techniques for Embedded Systems: A Surveycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
  2. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation
  3. 2023CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
  4. 2024Pruning Foundation Models for High Accuracy without Retraining
  5. 2025HPM-KD: Hierarchical Progressive Multi-Teacher Framework for Knowledge Distillation and Efficient Model Compression
  6. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
  7. 2024Self-Data Distillation for Recovering Quality in Pruned Large Language Models
  8. 2024Enhanced Sparsification via Stimulative Training