# On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method

arXiv: https://arxiv.org/abs/2104.03420 · HTML: https://mlsanity.com/paper/2104.03420/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2021.

## Built on by (most cited first, 9 of 9)

- [Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networks](https://mlsanity.com/paper/2308.09858/) (2023)
- [TedNet: A Pytorch Toolkit for Tensor Decomposition Networks](https://mlsanity.com/paper/2104.05018/) (2021)
- [TensorKrowch: Smooth integration of tensor networks in machine learning](https://mlsanity.com/paper/2306.08595/) (2023)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration](https://mlsanity.com/paper/2105.06250/) (2021)
- [Low-Rank+Sparse Tensor Compression for Neural Networks](https://mlsanity.com/paper/2111.01697/) (2021)
- [Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding](https://mlsanity.com/paper/2306.01076/) (2023)
- [Tensor Shape Search for Optimum Data Compression](https://mlsanity.com/paper/2205.10651/) (2022)
- [Ultrafast On-Chip Online Learning via Spline Locality in Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2602.02056/) (2026)

## Abstract (arXiv)

Various hardware accelerators have been developed for energy-efficient and real-time inference of neural networks on edge devices. However, most training is done on high-performance GPUs or servers, and the huge memory and computing costs prevent training neural networks on edge devices. This paper proposes a novel tensor-based training framework, which offers orders-of-magnitude memory reduction in the training process. We propose a novel rank-adaptive tensorized neural network model, and design a hardware-friendly low-precision algorithm to train this model. We present an FPGA accelerator to demonstrate the benefits of this training method on edge devices. Our preliminary FPGA implementation achieves 59× speedup and 123× energy reduction compared to embedded CPU, and 292× memory reduction over a standard full-size training.