# Backprop with Approximate Activations for Memory-efficient Network Training

arXiv: https://arxiv.org/abs/1901.07988 · HTML: https://mlsanity.com/paper/1901.07988/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 19 of 19)

- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training](https://mlsanity.com/paper/2104.14129/) (2021)
- [GACT: Activation Compressed Training for Generic Network Architectures](https://mlsanity.com/paper/2206.11357/) (2022)
- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers](https://mlsanity.com/paper/2302.02596/) (2023)
- [Mesa: A Memory-saving Training Framework for Transformers](https://mlsanity.com/paper/2111.11124/) (2021)
- [Enabling Binary Neural Network Training on the Edge](https://mlsanity.com/paper/2102.04270/) (2021)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training](https://mlsanity.com/paper/2202.13808/) (2022)
- [Do We Really Need a Large Number of Visual Prompts?](https://mlsanity.com/paper/2305.17223/) (2023)
- [Operating critical machine learning models in resource constrained regimes](https://mlsanity.com/paper/2303.10181/) (2023)
- [SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics](https://mlsanity.com/paper/2305.18513/) (2023)
- [Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation](https://mlsanity.com/paper/2406.16282/) (2024)
- [DIVISION: Memory Efficient Training via Dual Activation Precision](https://mlsanity.com/paper/2208.04187/) (2022)
- [Block Selective Reprogramming for On-device Training of Vision Transformers](https://mlsanity.com/paper/2405.10951/) (2024)
- [TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network](https://mlsanity.com/paper/2506.01352/) (2025)
- [OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation](https://mlsanity.com/paper/2310.02422/) (2023)
- [Moonwalk: Inverse-Forward Differentiation](https://mlsanity.com/paper/2402.14212/) (2024)
- [Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients](https://mlsanity.com/paper/2509.03503/) (2025)

## Abstract (arXiv)

Training convolutional neural network models is memory intensive since back-propagation requires storing activations of all intermediate layers. This presents a practical concern when seeking to deploy very deep architectures in production, especially when models need to be frequently re-trained on updated datasets. In this paper, we propose a new implementation for back-propagation that significantly reduces memory usage, by enabling the use of approximations with negligible computational cost and minimal effect on training performance. The algorithm reuses common buffers to temporarily store full activations and compute the forward pass exactly. It also stores approximate per-layer copies of activations, at significant memory savings, that are used in the backward pass. Compared to simply approximating activations within standard back-propagation, our method limits accumulation of errors across layers. This allows the use of much lower-precision approximations without affecting training accuracy. Experiments on CIFAR-10, CIFAR-100, and ImageNet show that our method yields performance close to exact training, while storing activations compactly with as low as 4-bit precision.