mlsanityData to 7 Oct 2026

Faster Neural Network Training with Approximate Tensor Operations

arXiv 1805.08079 ↗First posted 21 May 2018cs.LG cs.AI cs.NEcited by 14

Not on the map. Why?

Abstract (arXiv)

We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same convergence guarantees when applied to SGD training. We apply approximate tensor operations to single and multi-node training of MLP and CNN networks on MNIST, CIFAR-10 and ImageNet datasets. We demonstrate up to 66% reduction in the amount of computations and communication, and up to 1.37x faster training time while maintaining negligible or no impact on the final test accuracy.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 1; 2025: 4; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2023Training Transformers with 4-bit Integerscited by 58
  3. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  4. 2021L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimizationcited by 21
  5. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21
  6. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  7. 2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
  8. 2023Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Modelcited by 9
  9. 2022Memory-Efficient Backpropagation through Large Linear Layerscited by 4
  10. 2023TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learningcited by 4
  11. 2025QKV Projections Require a Fraction of Their Memorycited by 2
  12. 2025On the Duality between Gradient Transformations and Adapterscited by 1
2 more
  1. 2025Energy and Memory-Efficient Federated Learning With Ordered Layer Freezingcited by 1
  2. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inferencecited by 1