# Memory-Efficient CNN Accelerator Based on Interlayer Feature Map Compression

arXiv: https://arxiv.org/abs/2110.06155 · HTML: https://mlsanity.com/paper/2110.06155/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [CREST: Convolutional Residual Learning for Visual Tracking](https://mlsanity.com/paper/1708.00225/) (2017)
- [DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework](https://mlsanity.com/paper/1803.05788/) (2018)
- [Feature Map Transform Coding for Energy-Efficient CNN Inference](https://mlsanity.com/paper/1905.10830/) (2019)
- [A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things](https://mlsanity.com/paper/1707.02973/) (2017)
- [DCT-Conv: Coding filters in convolutional networks with Discrete Cosine Transform](https://mlsanity.com/paper/2001.08517/) (2020)
- [Chain-NN: An Energy-Efficient 1D Chain Architecture for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1703.01457/) (2017)

## Built on by (most cited first, 4 of 4)

- [Improved Projection Learning for Lower Dimensional Feature Maps](https://mlsanity.com/paper/2210.15170/) (2022)
- [Frequency Regularization: Restricting Information Redundancy of Convolutional Neural Networks](https://mlsanity.com/paper/2304.07973/) (2023)
- [ASC: Adaptive Scale Feature Map Compression for Deep Neural Network](https://mlsanity.com/paper/2312.08176/) (2023)
- [SF-MMCN: Low-Power Sever Flow Multi-Mode Diffusion Model Accelerator](https://mlsanity.com/paper/2403.10542/) (2024)

## Abstract (arXiv)

Existing deep convolutional neural networks (CNNs) generate massive interlayer feature data during network inference. To maintain real-time processing in embedded systems, large on-chip memory is required to buffer the interlayer feature maps. In this paper, we propose an efficient hardware accelerator with an interlayer feature compression technique to significantly reduce the required on-chip memory size and off-chip memory access bandwidth. The accelerator compresses interlayer feature maps through transforming the stored data into frequency domain using hardware-implemented 8x8 discrete cosine transform (DCT). The high-frequency components are removed after the DCT through quantization. Sparse matrix compression is utilized to further compress the interlayer feature maps. The on-chip memory allocation scheme is designed to support dynamic configuration of the feature map buffer size and scratch pad size according to different network-layer requirements. The hardware accelerator combines compression, decompression, and CNN acceleration into one computing stream, achieving minimal compressing and processing delay. A prototype accelerator is implemented on an FPGA platform and also synthesized in TSMC 28-nm COMS technology. It achieves 403GOPS peak throughput and 1.4x~3.3x interlayer feature map reduction by adding light hardware area overhead, making it a promising hardware accelerator for intelligent IoT devices.