# Exploring the Rate-Distortion-Complexity Optimization in Neural Image Compression

arXiv: https://arxiv.org/abs/2305.07678 · HTML: https://mlsanity.com/paper/2305.07678/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 59 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Variable Rate Deep Image Compression With a Conditional Autoencoder](https://mlsanity.com/paper/1909.04802/) (2019)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)
- [End-to-end optimization of nonlinear transform codes for perceptual quality](https://mlsanity.com/paper/1607.05006/) (2016)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform](https://mlsanity.com/paper/2108.09551/) (2021)
- [Soft then Hard: Rethinking the Quantization in Neural Image Compression](https://mlsanity.com/paper/2104.05168/) (2021)
- [Learning Accurate Entropy Model with Global Reference for Image Compression](https://mlsanity.com/paper/2010.08321/) (2020)
- [Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression](https://mlsanity.com/paper/1906.10057/) (2019)
- [Learned Block-based Hybrid Image Compression](https://mlsanity.com/paper/2012.09550/) (2020)
- [Multi-scale and Context-adaptive Entropy Model for Image Compression](https://mlsanity.com/paper/1910.07844/) (2019)
- [DeepQTMT: A Deep Learning Approach for Fast QTMT-based CU Partition of Intra-mode VVC](https://mlsanity.com/paper/2006.13125/) (2020)

## Built on by (most cited first, 2 of 2)

- [Balanced Rate-Distortion Optimization in Learned Image Compression](https://mlsanity.com/paper/2502.20161/) (2025)
- [Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence](https://mlsanity.com/paper/2601.20769/) (2026)

## Abstract (arXiv)

Despite a short history, neural image codecs have been shown to surpass classical image codecs in terms of rate-distortion performance. However, most of them suffer from significantly longer decoding times, which hinders the practical applications of neural image codecs. This issue is especially pronounced when employing an effective yet time-consuming autoregressive context model since it would increase entropy decoding time by orders of magnitude. In this paper, unlike most previous works that pursue optimal RD performance while temporally overlooking the coding complexity, we make a systematical investigation on the rate-distortion-complexity (RDC) optimization in neural image compression. By quantifying the decoding complexity as a factor in the optimization goal, we are now able to precisely control the RDC trade-off and then demonstrate how the rate-distortion performance of neural image codecs could adapt to various complexity demands. Going beyond the investigation of RDC optimization, a variable-complexity neural codec is designed to leverage the spatial dependencies adaptively according to industrial demands, which supports fine-grained complexity adjustment by balancing the RDC tradeoff. By implementing this scheme in a powerful base model, we demonstrate the feasibility and flexibility of RDC optimization for neural image codecs.