mlsanityData to 7 Oct 2026

PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation

arXiv 2406.09117 ↗First posted 13 Jun 2024cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Low-rank adaption (LoRA) is a prominent method that adds a small number of learnable parameters to the frozen pre-trained weights for parameter-efficient fine-tuning. Prompted by the question, ``Can we make its representation enough with LoRA weights solely at the final phase of finetuning without the pre-trained weights?'' In this work, we introduce Progressive Compression LoRA~(PC-LoRA), which utilizes low-rank adaptation (LoRA) to simultaneously perform model compression and fine-tuning. The PC-LoRA method gradually removes the pre-trained weights during the training process, eventually leaving only the low-rank adapters in the end. Thus, these low-rank adapters replace the whole pre-trained weights, achieving the goals of compression and fine-tuning at the same time. Empirical analysis across various models demonstrates that PC-LoRA achieves parameter and FLOPs compression rates of 94.36%/89.1% for vision models, e.g., ViT-B, and 93.42%/84.2% parameters and FLOPs compressions for language models, e.g., BERT.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

16 of 23 references.

References · 16

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
4 more
  1. 2018Rethinking the Value of Network Pruningcited by 595
  2. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  3. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  4. 2022Language model compression with weighted low-rank factorizationcited by 192

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 6
  2. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  3. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3
  4. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  5. 2025TuneComp: Joint Fine-tuning and Compression for Large Foundation Modelscited by 0
  6. 2026Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inferencecited by 0
  7. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  8. 2026SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillationcited by 0