# Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization

arXiv: https://arxiv.org/abs/2309.11856 · HTML: https://mlsanity.com/paper/2309.11856/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models](https://mlsanity.com/paper/2007.03051/) (2020)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training](https://mlsanity.com/paper/2104.14129/) (2021)
- [8-Bit Approximations for Parallelism in Deep Learning](https://mlsanity.com/paper/1511.04561/) (2015)
- [A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking](https://mlsanity.com/paper/2210.07494/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Distributed Scheduling using Graph Neural Networks](https://mlsanity.com/paper/2011.09430/) (2020)
- [Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI](https://mlsanity.com/paper/2309.02065/) (2023)
- [Matrix Completion With Variational Graph Autoencoders: Application in Hyperlocal Air Quality Inference](https://mlsanity.com/paper/1811.01662/) (2018)
- [Multi-Channel Speech Enhancement using Graph Neural Networks](https://mlsanity.com/paper/2102.06934/) (2021)
- [Training Graph Neural Networks on Growing Stochastic Graphs](https://mlsanity.com/paper/2210.15567/) (2022)
- [Improved stochastic rounding](https://mlsanity.com/paper/2006.00489/) (2020)

## Built on by (most cited first, 4 of 4)

- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Activation Map Compression through Tensor Decomposition for Deep Learning](https://mlsanity.com/paper/2411.06346/) (2024)
- [An Empirical Study of the Influence of Adversarial Fine-Tuning on Compressed Neural Networks](https://mlsanity.com/paper/2403.09441/) (2024)
- [Memory Constrained Dynamic Subnetwork Update for Transfer Learning](https://mlsanity.com/paper/2510.20979/) (2025)

## Abstract (arXiv)

Efficient training of large-scale graph neural networks (GNNs) has been studied with a specific focus on reducing their memory consumption. Work by Liu et al. (2022) proposed extreme activation compression (EXACT) which demonstrated drastic reduction in memory consumption by performing quantization of the intermediate activation maps down to using INT2 precision. They showed little to no reduction in performance while achieving large reductions in GPU memory consumption. In this work, we present an improvement to the EXACT strategy by using block-wise quantization of the intermediate activation maps. We experimentally analyze different block sizes and show further reduction in memory consumption (>15%), and runtime speedup per epoch (about 5%) even when performing extreme extents of quantization with similar performance trade-offs as with the original EXACT. Further, we present a correction to the assumptions on the distribution of intermediate activation maps in EXACT (assumed to be uniform) and show improved variance estimations of the quantization and dequantization steps.