mlsanityData to 14 Oct 2026

Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals

arXiv 2305.18425 ↗First posted 28 May 2023cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we present an efficient method for storing fine-tuned models by leveraging the low-rank properties of weight residuals. Our key observation is that weight residuals in large overparameterized models exhibit even stronger low-rank characteristics. Based on this insight, we propose Efficient Residual Encoding (ERE), a novel approach that achieves efficient storage of fine-tuned model weights by approximating the low-rank weight residuals. Furthermore, we analyze the robustness of weight residuals and push the limit of storage efficiency by utilizing additional quantization and layer-wise rank allocation. Our experimental results demonstrate that our method significantly reduces memory footprint while preserving performance in various tasks and modalities. We release our code.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

42 of 51 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
30 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
  13. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  14. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  15. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  16. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  17. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  18. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  19. 2017Variational Continual Learningcited by 339
  20. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  21. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  22. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  23. 2020Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANscited by 132
  24. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  25. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  26. 2023Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMscited by 38
  27. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  28. 2020Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Networkcited by 20
  29. 2023A Comparative Study between Full-Parameter and LoRA-based Fine-Tuning on Chinese Instruction Data for Instruction Following Large Language Modelcited by 19
  30. 2017Learned Perceptual Image Enhancementcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 2.

12 papers.

  1. 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 39
  2. 2024Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Modelscited by 17
  3. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  4. 2025ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMscited by 5
  5. 2025Towards Reversible Model Merging For Low-rank Weightscited by 3
  6. 2024ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Modelscited by 2
  7. 2026D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximationcited by 1
  8. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  9. 2025RanDeS: Randomized Delta Superposition for Multi-Model Compressioncited by 0
  10. 2025SineLoRADelta: Sine-Activated Delta Compressioncited by 0
  11. 2025Enhancing Delta Compression in LLMs via SVD-based Quantization Error Minimizationcited by 0
  12. 2025Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices
  2. 2026Kilobyte Models: Neural Networks as a Seed and a Quantized Latent
  3. 2026Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
  4. 2023WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models
  5. 2026Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
  6. 2026ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
  7. 2026Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs
  8. 2026Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices