# Faster Neural Network Training with Approximate Tensor Operations

arXiv: https://arxiv.org/abs/1805.08079 · HTML: https://mlsanity.com/paper/1805.08079/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 14 of 14)

- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Training Transformers with 4-bit Integers](https://mlsanity.com/paper/2306.11987/) (2023)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization](https://mlsanity.com/paper/2110.14807/) (2021)
- [GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning](https://mlsanity.com/paper/2502.12913/) (2025)
- [RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computations](https://mlsanity.com/paper/2210.10737/) (2022)
- [Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures](https://mlsanity.com/paper/2407.19580/) (2024)
- [Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model](https://mlsanity.com/paper/2305.15265/) (2023)
- [Memory-Efficient Backpropagation through Large Linear Layers](https://mlsanity.com/paper/2201.13195/) (2022)
- [TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learning](https://mlsanity.com/paper/2308.09201/) (2023)
- [QKV Projections Require a Fraction of Their Memory](https://mlsanity.com/paper/2506.02939/) (2025)
- [On the Duality between Gradient Transformations and Adapters](https://mlsanity.com/paper/2502.13811/) (2025)
- [Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing](https://mlsanity.com/paper/2512.23200/) (2025)
- [Reformulating KV Cache Eviction Problem for Long-Context LLM Inference](https://mlsanity.com/paper/2605.07234/) (2026)

## Abstract (arXiv)

We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.