# Memory-efficient training with streaming dimensionality reduction

arXiv: https://arxiv.org/abs/2004.12041 · HTML: https://mlsanity.com/paper/2004.12041/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 1 of 1)

- [Non-Volatile Memory Accelerated Geometric Multi-Scale Resolution Analysis](https://mlsanity.com/paper/2202.11518/) (2022)

## Abstract (arXiv)

The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads. To minimize this overhead, especially on the movement and calculation of gradient information, we introduce streaming batch principal component analysis as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations to generate a stochastic k-rank approximation of the network gradient. We demonstrate that the low rank updates produced by streaming batch principal component analysis can effectively train convolutional neural networks on a variety of common datasets, with performance comparable to standard mini batch gradient descent. These results can lead to both improvements in the design of application specific integrated circuits for deep learning and in the speed of synchronization of machine learning models trained with data parallelism.