# 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

arXiv: https://arxiv.org/abs/2105.06250 · HTML: https://mlsanity.com/paper/2105.06250/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM](https://mlsanity.com/paper/1707.09870/) (2017)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edge](https://mlsanity.com/paper/1904.04421/) (2019)
- [E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings](https://mlsanity.com/paper/1910.13349/) (2019)
- [Ternary Neural Networks for Resource-Efficient AI Applications](https://mlsanity.com/paper/1609.00222/) (2016)
- [Compressing Recurrent Neural Network with Tensor Train](https://mlsanity.com/paper/1705.08052/) (2017)
- [Tensorized Embedding Layers for Efficient Model Compression](https://mlsanity.com/paper/1901.10787/) (2019)
- [Bayesian Tensorized Neural Networks with Automatic Rank Selection](https://mlsanity.com/paper/1905.10478/) (2019)
- [Hardware-oriented Approximation of Convolutional Neural Networks](https://mlsanity.com/paper/1604.03168/) (2016)
- [Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination](https://mlsanity.com/paper/2010.08689/) (2020)
- [VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization](https://mlsanity.com/paper/2005.08501/) (2020)
- [Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design](https://mlsanity.com/paper/2103.15750/) (2021)
- [Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition](https://mlsanity.com/paper/2005.04366/) (2020)
- [On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method](https://mlsanity.com/paper/2104.03420/) (2021)
- [FPGA Based Implementation of Deep Neural Networks Using On-chip Memory Only](https://mlsanity.com/paper/1602.01616/) (2016)

## Built on by (most cited first, 3 of 3)

- [Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networks](https://mlsanity.com/paper/2308.09858/) (2023)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Enabling All In-Edge Deep Learning: A Literature Review](https://mlsanity.com/paper/2204.03326/) (2022)

## Abstract (arXiv)

The deep neural network (DNN) based AI applications on the edge require both low-cost computing platforms and high-quality services. However, the limited memory, computing resources, and power budget of the edge devices constrain the effectiveness of the DNN algorithms. Developing edge-oriented AI algorithms and implementations (e.g., accelerators) is challenging. In this paper, we summarize our recent efforts for efficient on-device AI development from three aspects, including both training and inference. First, we present on-device training with ultra-low memory usage. We propose a novel rank-adaptive tensor-based tensorized neural network model, which offers orders-of-magnitude memory reduction during training. Second, we introduce an ultra-low bitwidth quantization method for DNN model compression, achieving the state-of-the-art accuracy under the same compression ratio. Third, we introduce an ultra-low latency DNN accelerator design, practicing the software/hardware co-design methodology. This paper emphasizes the importance and efficacy of training, quantization and accelerator design, and calls for more research breakthroughs in the area for AI on the edge.