# SMOF: Streaming Modern CNNs on FPGAs with Smart Off-Chip Eviction

arXiv: https://arxiv.org/abs/2403.18921 · HTML: https://mlsanity.com/paper/2403.18921/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [Memory-Efficient Dataflow Inference for Deep CNNs on FPGA](https://mlsanity.com/paper/2011.07317/) (2020)
- [SAMO: Optimised Mapping of Convolutional Neural Networks to Streaming Architectures](https://mlsanity.com/paper/2112.00170/) (2021)
- [HARFLOW3D: A Latency-Oriented 3D-CNN Accelerator Toolflow for HAR on FPGA Devices](https://mlsanity.com/paper/2303.17218/) (2023)
- [SATAY: A Streaming Architecture Toolflow for Accelerating YOLO Models on FPGA Devices](https://mlsanity.com/paper/2309.01587/) (2023)
- [Snowflake: A Model Agnostic Accelerator for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1708.02579/) (2017)
- [PASS: Exploiting Post-Activation Sparsity in Streaming Architectures for CNN Acceleration](https://mlsanity.com/paper/2307.07821/) (2023)
- [DeepBurning-MixQ: An Open Source Mixed-Precision Neural Network Accelerator Design Framework for FPGAs](https://mlsanity.com/paper/2308.11334/) (2023)
- [fpgaHART: A toolflow for throughput-oriented acceleration of 3D CNNs for HAR onto FPGAs](https://mlsanity.com/paper/2305.19896/) (2023)
- [AutoWS: Automate Weights Streaming in Layer-wise Pipelined DNN Accelerators](https://mlsanity.com/paper/2311.04764/) (2023)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) have demonstrated their effectiveness in numerous vision tasks. However, their high processing requirements necessitate efficient hardware acceleration to meet the application's performance targets. In the space of FPGAs, streaming-based dataflow architectures are often adopted by users, as significant performance gains can be achieved through layer-wise pipelining and reduced off-chip memory access by retaining data on-chip. However, modern topologies, such as the UNet, YOLO, and X3D models, utilise long skip connections, requiring significant on-chip storage and thus limiting the performance achieved by such system architectures. The paper addresses the above limitation by introducing weight and activation eviction mechanisms to off-chip memory along the computational pipeline, taking into account the available compute and memory resources. The proposed mechanism is incorporated into an existing toolflow, expanding the design space by utilising off-chip memory as a buffer. This enables the mapping of such modern CNNs to devices with limited on-chip memory, under the streaming architecture design approach. SMOF has demonstrated the capacity to deliver competitive and, in some cases, state-of-the-art performance across a spectrum of computer vision tasks, achieving up to 10.65 X throughput improvement compared to previous works.