# Latent-Shift: Gradient of Entropy Helps Neural Codecs

arXiv: https://arxiv.org/abs/2308.00725 · HTML: https://mlsanity.com/paper/2308.00725/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Content Adaptive Optimization for Neural Image Compression](https://mlsanity.com/paper/1906.01223/) (2019)
- [Reducing The Amortization Gap of Entropy Bottleneck In End-to-End Image Compression](https://mlsanity.com/paper/2209.00964/) (2022)
- [AIVC: Artificial Intelligence based Video Codec](https://mlsanity.com/paper/2202.04365/) (2022)

## Built on by (most cited first, 3 of 3)

- [Improved Positional Encoding for Implicit Neural Representation based Compact Data Representation](https://mlsanity.com/paper/2311.06059/) (2023)
- [Robustly overfitting latents for flexible neural image compression](https://mlsanity.com/paper/2401.17789/) (2024)
- [Exploiting Latent Properties to Optimize Neural Codecs](https://mlsanity.com/paper/2501.01231/) (2025)

## Abstract (arXiv)

End-to-end image/video codecs are getting competitive compared to traditional compression techniques that have been developed through decades of manual engineering efforts. These trainable codecs have many advantages over traditional techniques such as easy adaptation on perceptual distortion metrics and high performance on specific domains thanks to their learning ability. However, state of the art neural codecs does not take advantage of the existence of gradient of entropy in decoding device. In this paper, we theoretically show that gradient of entropy (available at decoder side) is correlated with the gradient of the reconstruction error (which is not available at decoder side). We then demonstrate experimentally that this gradient can be used on various compression methods, leading to a 1-2% rate savings for the same quality. Our method is orthogonal to other improvements and brings independent rate savings.