# CompAct: Compressed Activations for Memory-Efficient LLM Training

arXiv: https://arxiv.org/abs/2410.15352 · HTML: https://mlsanity.com/paper/2410.15352/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [ReLoRA: High-Rank Training Through Low-Rank Updates](https://mlsanity.com/paper/2307.05695/) (2023)
- [Flora: Low-Rank Adapters Are Secretly Gradient Compressors](https://mlsanity.com/paper/2402.03293/) (2024)
- [SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining](https://mlsanity.com/paper/2406.02214/) (2024)
- [Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients](https://mlsanity.com/paper/2407.08296/) (2024)
- [GACT: Activation Compressed Training for Generic Network Architectures](https://mlsanity.com/paper/2206.11357/) (2022)
- [VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections](https://mlsanity.com/paper/2405.17991/) (2024)
- [Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients](https://mlsanity.com/paper/2406.17660/) (2024)
- [Mesa: A Memory-saving Training Framework for Transformers](https://mlsanity.com/paper/2111.11124/) (2021)
- [Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation](https://mlsanity.com/paper/2406.16282/) (2024)
- [Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction](https://mlsanity.com/paper/2210.10246/) (2022)

## Built on by (most cited first, 15 of 15)

- [CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation](https://mlsanity.com/paper/2502.10940/) (2025)
- [On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://mlsanity.com/paper/2602.15322/) (2026)
- [Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking](https://mlsanity.com/paper/2505.22922/) (2025)
- [QKV Projections Require a Fraction of Their Memory](https://mlsanity.com/paper/2506.02939/) (2025)
- [Memory-Efficient Fine-Tuning via Low-Rank Activation Compression](https://mlsanity.com/paper/2509.23472/) (2025)
- [BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models](https://mlsanity.com/paper/2512.12131/) (2025)
- [PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training](https://mlsanity.com/paper/2602.23111/) (2026)
- [Rethinking Token Prediction: Tree-Structured Diffusion Language Model](https://mlsanity.com/paper/2604.03537/) (2026)
- [OASIS: Online Activation Subspace Learning for Memory-Efficient Training](https://mlsanity.com/paper/2604.09406/) (2026)
- [DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training](https://mlsanity.com/paper/2606.28932/) (2026)
- [PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training](https://mlsanity.com/paper/2505.18313/) (2025)
- [It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs](https://mlsanity.com/paper/2506.00486/) (2025)
- [AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs](https://mlsanity.com/paper/2605.00539/) (2026)
- [Efficient Pre-Training of LLMs through Truncated SVD Representations](https://mlsanity.com/paper/2605.28573/) (2026)
- [CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA](https://mlsanity.com/paper/2607.11940/) (2026)

## Abstract (arXiv)

We introduce CompAct, a technique that reduces peak memory utilization on GPU by 25-30% for pretraining and 50% for fine-tuning of LLMs. Peak device memory is a major limiting factor in training LLMs, with various recent works aiming to reduce model memory. However most works don't target the largest component of allocated memory during training: the model's compute graph, which is stored for the backward pass. By storing low-rank, compressed activations to be used in the backward pass we greatly reduce the required memory, unlike previous methods which only reduce optimizer overheads or the number of trained parameters. Our compression uses random projection matrices, thus avoiding additional memory overheads. Comparisons with previous techniques for either pretraining or fine-tuning show that CompAct substantially improves existing compute-performance tradeoffs. We expect CompAct's savings to scale even higher for larger models.