# End-to-end optimization of nonlinear transform codes for perceptual quality

arXiv: https://arxiv.org/abs/1607.05006 · HTML: https://mlsanity.com/paper/1607.05006/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2016.

## Built on by (most cited first, 30 of 73)

- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [An Introduction to Neural Data Compression](https://mlsanity.com/paper/2202.06533/) (2022)
- [End-to-End Rate-Distortion Optimized 3D Gaussian Representation](https://mlsanity.com/paper/2406.01597/) (2024)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning by Reconstruction Produces Uninformative Features For Perception](https://mlsanity.com/paper/2402.11337/) (2024)
- [Image coding for machines: an end-to-end learned approach](https://mlsanity.com/paper/2108.09993/) (2021)
- [Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning](https://mlsanity.com/paper/2505.12477/) (2025)
- [Learned Image Compression with Gaussian-Laplacian-Logistic Mixture Model and Concatenated Residual Modules](https://mlsanity.com/paper/2107.06463/) (2021)
- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [Extending Neural P-frame Codecs for B-frame Coding](https://mlsanity.com/paper/2104.00531/) (2021)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Framework](https://mlsanity.com/paper/2106.02782/) (2021)
- [4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video](https://mlsanity.com/paper/2503.18421/) (2025)
- [Computationally-Efficient Neural Image Compression with Shallow Decoders](https://mlsanity.com/paper/2304.06244/) (2023)
- [LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks](https://mlsanity.com/paper/2111.08988/) (2021)
- [MobileCodec: Neural Inter-frame Video Compression on Mobile Devices](https://mlsanity.com/paper/2207.08338/) (2022)
- [M2T: Masking Transformers Twice for Faster Decoding](https://mlsanity.com/paper/2304.07313/) (2023)
- [Frequency Disentangled Features in Neural Image Compression](https://mlsanity.com/paper/2308.02620/) (2023)
- [IICNet: A Generic Framework for Reversible Image Conversion](https://mlsanity.com/paper/2109.04242/) (2021)
- [HPC: Hierarchical Progressive Coding Framework for Volumetric Video](https://mlsanity.com/paper/2407.09026/) (2024)
- [Scale-arbitrary Invertible Image Downscaling](https://mlsanity.com/paper/2201.12576/) (2022)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [Insights from Generative Modeling for Neural Video Compression](https://mlsanity.com/paper/2107.13136/) (2021)
- [Image Compression with Recurrent Neural Network and Generalized Divisive Normalization](https://mlsanity.com/paper/2109.01999/) (2021)
- [Self-Organized Variational Autoencoders (Self-VAE) for Learned Image Compression](https://mlsanity.com/paper/2105.12107/) (2021)
- [Attention-Based Generative Neural Image Compression on Solar Dynamics Observatory](https://mlsanity.com/paper/2210.06478/) (2022)
- [Generalized Gaussian Model for Learned Image Compression](https://mlsanity.com/paper/2411.19320/) (2024)
- [OSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression](https://mlsanity.com/paper/2107.09179/) (2021)
- [Rate Distortion Characteristic Modeling for Neural Image Compression](https://mlsanity.com/paper/2106.12954/) (2021)

## Abstract (arXiv)

We introduce a general framework for end-to-end optimization of the rate--distortion performance of nonlinear transform codes assuming scalar quantization. The framework can be used to optimize any differentiable pair of analysis and synthesis transforms in combination with any differentiable perceptual metric. As an example, we consider a code built from a linear transform followed by a form of multi-dimensional local gain control. Distortion is measured with a state-of-the-art perceptual metric. When optimized over a large database of images, this representation offers substantial improvements in bitrate and perceptual appearance over fixed (DCT) codes, and over linear transform codes optimized for mean squared error.