# Layer-specific Optimization for Mixed Data Flow with Mixed Precision in FPGA Design for CNN-based Object Detectors

arXiv: https://arxiv.org/abs/2009.01588 · HTML: https://mlsanity.com/paper/2009.01588/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [Towards Efficient In-memory Computing Hardware for Quantized Neural Networks: State-of-the-art, Open Challenges and Perspectives](https://mlsanity.com/paper/2307.03936/) (2023)
- [SATAY: A Streaming Architecture Toolflow for Accelerating YOLO Models on FPGA Devices](https://mlsanity.com/paper/2309.01587/) (2023)
- [Design of High-Throughput Mixed-Precision CNN Accelerators on FPGA](https://mlsanity.com/paper/2208.04854/) (2022)
- [Binarizing by Classification: Is soft function really necessary?](https://mlsanity.com/paper/2205.07433/) (2022)
- [Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deployment](https://mlsanity.com/paper/2407.12070/) (2024)
- [CORDIC Is All You Need](https://mlsanity.com/paper/2503.11685/) (2025)
- [ASPO: Constraint-Aware Bayesian Optimization for FPGA-based Soft Processors](https://mlsanity.com/paper/2506.06817/) (2025)
- [Efficient SAR Vessel Detection for FPGA-Based On-Satellite Sensing](https://mlsanity.com/paper/2507.04842/) (2025)

## Abstract (arXiv)

Convolutional neural networks (CNNs) require both intensive computation and frequent memory access, which lead to a low processing speed and large power dissipation. Although the characteristics of the different layers in a CNN are frequently quite different, previous hardware designs have employed common optimization schemes for them. This paper proposes a layer-specific design that employs different organizations that are optimized for the different layers. The proposed design employs two layer-specific optimizations: layer-specific mixed data flow and layer-specific mixed precision. The mixed data flow aims to minimize the off-chip access while demanding a minimal on-chip memory (BRAM) resource of an FPGA device. The mixed precision quantization is to achieve both a lossless accuracy and an aggressive model compression, thereby further reducing the off-chip access. A Bayesian optimization approach is used to select the best sparsity for each layer, achieving the best trade-off between the accuracy and compression. This mixing scheme allows the entire network model to be stored in BRAMs of the FPGA to aggressively reduce the off-chip access, and thereby achieves a significant performance enhancement. The model size is reduced by 22.66-28.93 times compared to that in a full-precision network with a negligible degradation of accuracy on VOC, COCO, and ImageNet datasets. Furthermore, the combination of mixed dataflow and mixed precision significantly outperforms the previous works in terms of both throughput, off-chip access, and on-chip memory requirement.