mlsanityData to 14 Oct 2026

CompAct: Compressed Activations for Memory-Efficient LLM Training

arXiv 2410.15352 ↗First posted 20 Oct 2024cs.LG cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce CompAct, a technique that reduces peak memory utilization on GPU by 25-30% for pretraining and 50% for fine-tuning of LLMs. Peak device memory is a major limiting factor in training LLMs, with various recent works aiming to reduce model memory. However most works don't target the largest component of allocated memory during training: the model's compute graph, which is stored for the backward pass. By storing low-rank, compressed activations to be used in the backward pass we greatly reduce the required memory, unlike previous methods which only reduce optimizer overheads or the number of trained parameters. Our compression uses random projection matrices, thus avoiding additional memory overheads. Comparisons with previous techniques for either pretraining or fine-tuning show that CompAct substantially improves existing compute-performance tradeoffs. We expect CompAct's savings to scale even higher for larger models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

22 of 33 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2017Mixed Precision Trainingcited by 1,156
  11. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
10 more
  1. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  2. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  3. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  4. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  5. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  6. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  7. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  8. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16
  9. 2024Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagationcited by 5
  10. 2022Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reductioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  2. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 7
  3. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  4. 2025QKV Projections Require a Fraction of Their Memorycited by 2
  5. 2025Memory-Efficient Fine-Tuning via Low-Rank Activation Compressioncited by 2
  6. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  7. 2026PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Trainingcited by 1
  8. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Modelcited by 1
  9. 2026OASIS: Online Activation Subspace Learning for Memory-Efficient Trainingcited by 1
  10. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  11. 2025PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Trainingcited by 0
  12. 2025It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMscited by 0
3 more
  1. 2026AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMscited by 0
  2. 2026Efficient Pre-Training of LLMs through Truncated SVD Representationscited by 0
  3. 2026CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRAcited by 0