# CNNs for JPEGs: A Study in Computational Cost

arXiv: https://arxiv.org/abs/2309.11417 · HTML: https://mlsanity.com/paper/2309.11417/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Learning Filter Basis for Convolutional Neural Network Compression](https://mlsanity.com/paper/1908.08932/) (2019)
- [Quantization Guided JPEG Artifact Correction](https://mlsanity.com/paper/2004.09320/) (2020)
- [Deep Residual Learning in the JPEG Transform Domain](https://mlsanity.com/paper/1812.11690/) (2018)
- [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017)
- [Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1809.06323/) (2018)
- [DCT-CompCNN: A Novel Image Classification Network Using JPEG Compressed DCT Coefficients](https://mlsanity.com/paper/1907.11503/) (2019)
- [Learning a Single Model with a Wide Range of Quality Factors for JPEG Image Artifacts Removal](https://mlsanity.com/paper/2009.06912/) (2020)
- [Fast object detection in compressed JPEG Images](https://mlsanity.com/paper/1904.08408/) (2019)
- [Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks](https://mlsanity.com/paper/1712.07493/) (2017)
- [Exploring Semantic Segmentation on the DCT Representation](https://mlsanity.com/paper/1907.10015/) (2019)
- [Less is More: Accelerating Faster Neural Networks Straight from JPEG](https://mlsanity.com/paper/2104.00185/) (2021)
- [Analyzing and Mitigating JPEG Compression Defects in Deep Learning](https://mlsanity.com/paper/2011.08932/) (2020)
- [Faster and Accurate Compressed Video Action Recognition Straight from the Frequency Domain](https://mlsanity.com/paper/2012.13726/) (2020)
- [Faster and Accurate Classification for JPEG2000 Compressed Images in Networked Applications](https://mlsanity.com/paper/1909.05638/) (2019)
- [Object Detection in the DCT Domain: is Luminance the Solution?](https://mlsanity.com/paper/2006.05732/) (2020)
- [Deep Learning Based Image Retrieval in the JPEG Compressed Domain](https://mlsanity.com/paper/2107.03648/) (2021)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have achieved astonishing advances over the past decade, defining state-of-the-art in several computer vision tasks. CNNs are capable of learning robust representations of the data directly from the RGB pixels. However, most image data are usually available in compressed format, from which the JPEG is the most widely used due to transmission and storage purposes demanding a preliminary decoding process that have a high computational load and memory usage. For this reason, deep learning methods capable of learning directly from the compressed domain have been gaining attention in recent years. Those methods usually extract a frequency domain representation of the image, like DCT, by a partial decoding, and then make adaptation to typical CNNs architectures to work with them. One limitation of these current works is that, in order to accommodate the frequency domain data, the modifications made to the original model increase significantly their amount of parameters and computational complexity. On one hand, the methods have faster preprocessing, since the cost of fully decoding the images is avoided, but on the other hand, the cost of passing the images though the model is increased, mitigating the possible upside of accelerating the method. In this paper, we propose a further study of the computational cost of deep models designed for the frequency domain, evaluating the cost of decoding and passing the images through the network. We also propose handcrafted and data-driven techniques for reducing the computational complexity and the number of parameters for these models in order to keep them similar to their RGB baselines, leading to efficient models with a better trade off between computational cost and accuracy.