mlsanityData to 7 Oct 2026

Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction

arXiv 2202.00441 ↗First posted 1 Feb 2022cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Memory footprint is one of the main limiting factors for large neural network training. In backpropagation, one needs to store the input to each operation in the computational graph. Every modern neural network model has quite a few pointwise nonlinearities in its architecture, and such operation induces additional memory costs which -- as we show -- can be significantly reduced by quantization of the gradients. We propose a systematic approach to compute optimal quantization of the retained gradients of the pointwise nonlinear functions with only a few bits per each element. We show that such approximation can be achieved by computing optimal piecewise-constant approximation of the derivative of the activation function, which can be done by dynamic programming. The drop-in replacements are implemented for all popular nonlinearities and can be used in any existing pipeline. We confirm the memory reduction and the same convergence on several open benchmarks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

21 of 33 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  11. 2021A White Paper on Neural Network Quantizationcited by 453
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
9 more
  1. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  2. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  3. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  4. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  5. 2019Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memorycited by 22
  6. 2019Active Subspace of Neural Networks: Structural Analysis and Universal Attackscited by 12
  7. 2020Addressing the Memory Bottleneck in AI Model Trainingcited by 3
  8. 2019Reduced-Order Modeling of Deep Neural Networkscited by 2
  9. 2021Emojich -- zero-shot emoji generation using Russian language: a technical reportcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 7; 2025: 2; 2026: 3.

14 papers.

Sort:

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  2. 2022Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AIcited by 58
  3. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  4. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  5. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  6. 2022Survey on Large Scale Neural Network Trainingcited by 4
  7. 2024Moonwalk: Inverse-Forward Differentiationcited by 1
  8. 2024Quantization of Large Language Models with an Overdetermined Basiscited by 1
  9. 2026Layer-wise Derivative Controlled Networkscited by 1
  10. 2024LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reductioncited by 0
  11. 2024Inverted Activations: Reducing Memory Footprint in Neural Network Trainingcited by 0
  12. 2025S2A: A Unified Framework for Parameter and Memory Efficient Transfer Learningcited by 0
2 more
  1. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0
  2. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0