Memory-Efficient Fine-Tuning via Low-Rank Activation Compression
Not on the map. Why?
Abstract (arXiv)
The parameter-efficient fine-tuning paradigm has garnered significant attention with the advancement of foundation models. Although numerous methods have been proposed to reduce the number of trainable parameters, their substantial memory overhead remains a critical bottleneck that hinders practical deployment. In this paper, we observe that model activations constitute a major source of memory consumption, especially under large batch sizes and long context lengths; however, the rank of the activations remains consistently low. Motivated by this insight, we propose a memory-efficient fine-tuning approach Low-Rank Activation Compression (LoRAct). Unlike prior work, LoRAct provides a more flexible and versatile compressing strategy that can be applied online during the forward pass without the need for any calibration data. Moreover, LoRAct incorporates a novel sampling-based orthogonal decomposition algorithm specifically designed for low-rank matrices, offering improved computational efficiency and a tighter error bound compared to the widely used RSVD. Experiments on both vision and language tasks demonstrate the effectiveness of LoRAct. Notably, LoRAct further reduces activation memory by approximately 80% in comparison with the widely adopted LoRA method, while maintaining competitive performance. The source code is available at https://github.com/shijxcs/meft.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 27 Sep 2025.
Builds on
42 of 60 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
30 more
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 2015An Introduction to Matrix Concentration Inequalitiescited by 397
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2022Language model compression with weighted low-rank factorizationcited by 195
- 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
- 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
- 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
- 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
- 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
- 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
- 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
- 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
- 2015Optimal approximate matrix product in terms of stable rankcited by 37
- 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
- 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
- 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
- 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
- 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
- 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
- 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
- 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
- 2024COAP: Memory-Efficient Training with Correlation-Aware Gradient Projectioncited by 8
- 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 7
- 2025Memory-Efficient Fine-Tuning of Transformers via Token Selectioncited by 4
- 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
- 2020WOR and p's: Sketches for ℓp-Sampling Without Replacementcited by 3
- 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.