# A 3 TOPS/W RISC-V Parallel Cluster for Inference of Fine-Grain Mixed-Precision Quantized Neural Networks

arXiv: https://arxiv.org/abs/2307.01056 · HTML: https://mlsanity.com/paper/2307.01056/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 3 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs](https://mlsanity.com/paper/2008.07127/) (2020)
- [Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers](https://mlsanity.com/paper/1905.13082/) (2019)
- [Vega: A 10-Core SoC for IoT End-Nodes with DNN Acceleration and Cognitive Wake-Up From MRAM-Based State-Retentive Sleep Mode](https://mlsanity.com/paper/2110.09101/) (2021)
- [A near-threshold RISC-V core with DSP extensions for scalable IoT Endpoint Devices](https://mlsanity.com/paper/1608.08376/) (2016)
- [A Mixed-Precision RISC-V Processor for Extreme-Edge DNN Inference](https://mlsanity.com/paper/2010.04073/) (2020)
- [Enabling Mixed-Precision Quantized Neural Networks in Extreme-Edge Devices](https://mlsanity.com/paper/2007.07759/) (2020)
- [XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Network on RISC-V based IoT End Nodes](https://mlsanity.com/paper/2011.14325/) (2020)

## Built on by (most cited first, 3 of 3)

- [Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO](https://mlsanity.com/paper/2311.01057/) (2023)
- [A Heterogeneous RISC-V based SoC for Secure Nano-UAV Navigation](https://mlsanity.com/paper/2401.03531/) (2024)
- [David and Goliath: An Empirical Evaluation of Attacks and Defenses for QNNs at the Deep Edge](https://mlsanity.com/paper/2404.05688/) (2024)

## Abstract (arXiv)

The emerging trend of deploying complex algorithms, such as Deep Neural Networks (DNNs), increasingly poses strict memory and energy efficiency requirements on Internet-of-Things (IoT) end-nodes. Mixed-precision quantization has been proposed as a technique to minimize a DNN's memory footprint and maximize its execution efficiency, with negligible end-to-end precision degradation. In this work, we present a novel hardware and software stack for energy-efficient inference of mixed-precision Quantized Neural Networks (QNNs). We introduce Flex-V, a processor based on the RISC-V Instruction Set Architecture (ISA) that features fused Mac&Load mixed-precision dot product instructions; to avoid the exponential growth of the encoding space due to mixed-precision variants, we encode formats into the Control-Status Registers (CSRs). Flex-V core is integrated into a tightly-coupled cluster of eight processors; in addition, we provide a full framework for the end-to-end deployment of DNNs including a compiler, optimized libraries, and a memory-aware deployment flow. Our results show up to 91.5 MAC/cycle and 3.26 TOPS/W on the cluster, implemented in a commercial 22nm FDX technology, with up to 8.5x speed-up, and an area overhead of only 5.6% with respect to the baseline. To demonstrate the capabilities of the architecture, we benchmark it with end-to-end real-life QNNs, improving performance by 2x - 2.5x with respect to existing solutions using fully flexible programmable processors.