# An Energy-Efficient Edge Computing Paradigm for Convolution-based Image Upsampling

arXiv: https://arxiv.org/abs/2107.07647 · HTML: https://mlsanity.com/paper/2107.07647/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Accelerating the Super-Resolution Convolutional Neural Network](https://mlsanity.com/paper/1608.00367/) (2016)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [On-Device Machine Learning: An Algorithms and Learning Theory Perspective](https://mlsanity.com/paper/1911.00623/) (2019)
- [Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize](https://mlsanity.com/paper/1707.02937/) (2017)
- [Is the deconvolution layer the same as a convolutional layer?](https://mlsanity.com/paper/1609.07009/) (2016)
- [An Energy-Efficient FPGA-based Deconvolutional Neural Networks Accelerator for Single Image Super-Resolution](https://mlsanity.com/paper/1801.05997/) (2018)
- [Are FPGAs Suitable for Edge Computing?](https://mlsanity.com/paper/1804.06404/) (2018)
- [A Competitive Edge: Can FPGAs Beat GPUs at DCNN Inference Acceleration in Resource-Limited Edge Computing Applications?](https://mlsanity.com/paper/2102.00294/) (2021)
- [Deep Learning Techniques for Super-Resolution in Video Games](https://mlsanity.com/paper/2012.09810/) (2020)
- [A Design Methodology for Efficient Implementation of Deconvolutional Neural Networks on an FPGA](https://mlsanity.com/paper/1705.02583/) (2017)

## Built on by (most cited first, 4 of 4)

- [A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2308.13504/) (2023)
- [Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations](https://mlsanity.com/paper/2110.08271/) (2021)
- [A2Q+: Improving Accumulator-Aware Weight Quantization](https://mlsanity.com/paper/2401.10432/) (2024)
- [Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2301.13376/) (2023)

## Abstract (arXiv)

A novel energy-efficient edge computing paradigm is proposed for real-time deep learning-based image upsampling applications. State-of-the-art deep learning solutions for image upsampling are currently trained using either resize or sub-pixel convolution to learn kernels that generate high fidelity images with minimal artifacts. However, performing inference with these learned convolution kernels requires memory-intensive feature map transformations that dominate time and energy costs in real-time applications. To alleviate this pressure on memory bandwidth, we confine the use of resize or sub-pixel convolution to training in the cloud by transforming learned convolution kernels to deconvolution kernels before deploying them for inference as a functionally equivalent deconvolution. These kernel transformations, intended as a one-time cost when shifting from training to inference, enable a systems designer to use each algorithm in their optimal context by preserving the image fidelity learned when training in the cloud while minimizing data transfer penalties during inference at the edge. We also explore existing variants of deconvolution inference algorithms and introduce a novel variant for consideration. We analyze and compare the inference properties of convolution-based upsampling algorithms using a quantitative model of incurred time and energy costs and show that using deconvolution for inference at the edge improves both system latency and energy efficiency when compared to their sub-pixel or resize convolution counterparts.