mlsanityData to 14 Oct 2026

GACT: Activation Compressed Training for Generic Network Architectures

arXiv 2206.11357 ↗First posted 22 Jun 2022cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Training large neural network (NN) models requires extensive memory resources, and Activation Compressed Training (ACT) is a promising approach to reduce training memory footprint. This paper presents GACT, an ACT framework to support a broad range of machine learning tasks for generic NN architectures with limited domain knowledge. By analyzing a linearized version of ACT's approximate gradient, we prove the convergence of GACT without prior knowledge on operator type or model architecture. To make training stable, we propose an algorithm that decides the compression ratio for each tensor by estimating its impact on the gradient at run time. We implement GACT as a PyTorch library that readily applies to any NN architecture. GACT reduces the activation memory for convolutional NNs, transformers, and graph NNs by up to 8.1x, enabling training with a 4.2x to 24.7x larger batch size, with negligible accuracy loss. We implement GACT as a PyTorch library at https://github.com/LiuXiaoxuanPKU/GACT-ICML.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 22 Jun 2022.

Builds on

31 of 51 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2017Graph Attention Networkscited by 8,923
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
19 more
  1. 2017Mixed Precision Trainingcited by 1,156
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  3. 2020Simple and Deep Graph Convolutional Networkscited by 860
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  5. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  9. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  10. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  11. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  12. 2018Training and Inference with Integers in Deep Neural Networkscited by 86
  13. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  14. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  15. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  16. 2020Dynamic Tensor Rematerializationcited by 46
  17. 2019Backprop with Approximate Activations for Memory-efficient Network Trainingcited by 19
  18. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16
  19. 2020A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compressioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 9; 2026: 3.

31 papers.

Sort:

  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  3. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  4. 2023A Survey on Efficient Training of Transformerscited by 39
  5. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  6. 2023TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edgecited by 18
  7. 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
  8. 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
  9. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  10. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  11. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  12. 2024DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagationcited by 7
19 more
  1. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  2. 2024SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantizationcited by 6
  3. 2025PaCA: Partial Connection Adaptation for Efficient Fine-Tuningcited by 6
  4. 2023SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamicscited by 5
  5. 2024Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagationcited by 5
  6. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  7. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 4
  8. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  9. 2025DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Trainingcited by 3
  10. 2023OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagationcited by 2
  11. 2025QKV Projections Require a Fraction of Their Memorycited by 2
  12. 2025FedHybrid: Breaking the Memory Wall of Federated Learning via Hybrid Tensor Managementcited by 1
  13. 2023An Evaluation of Memory Optimization Methods for Training Neural Networkscited by 0
  14. 2024Face processing emerges from object-trained convolutional neural networkscited by 0
  15. 2025BurTorch: Revisiting Training from First Principles by Coupling Autodiff, Math Optimization, and Systemscited by 0
  16. 2025Rethinking the Potential of Layer Freezing for Efficient DNN Trainingcited by 0
  17. 2026On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passescited by 0
  18. 2026AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMscited by 0
  19. 2026Backward-State Policy Is Part of the Learning Algorithmcited by 0