# Entroformer: A Transformer-based Entropy Model for Learned Image Compression

arXiv: https://arxiv.org/abs/2202.05492 · HTML: https://mlsanity.com/paper/2202.05492/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Learning Convolutional Networks for Content-weighted Image Compression](https://mlsanity.com/paper/1703.10553/) (2017)
- [End-to-end optimization of nonlinear transform codes for perceptual quality](https://mlsanity.com/paper/1607.05006/) (2016)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [KVT: k-NN Attention for Boosting Vision Transformers](https://mlsanity.com/paper/2106.00515/) (2021)
- [Learning Accurate Entropy Model with Global Reference for Image Compression](https://mlsanity.com/paper/2010.08321/) (2020)
- [Ada-NETS: Face Clustering via Adaptive Neighbour Discovery in the Structure Space](https://mlsanity.com/paper/2202.03800/) (2022)
- [Interpolation variable rate image compression](https://mlsanity.com/paper/2109.09280/) (2021)
- [Learning Context-Based Non-local Entropy Modeling for Image Compression](https://mlsanity.com/paper/2005.04661/) (2020)

## Built on by (most cited first, 30 of 107)

- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [MLIC: Multi-Reference Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2211.07273/) (2022)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior](https://mlsanity.com/paper/2404.18820/) (2024)
- [MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression](https://mlsanity.com/paper/2307.15421/) (2023)
- [Frequency-Aware Transformer for Learned Image Compression](https://mlsanity.com/paper/2310.16387/) (2023)
- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression](https://mlsanity.com/paper/2203.02452/) (2022)
- [Lossy Image Compression with Foundation Diffusion Models](https://mlsanity.com/paper/2404.08580/) (2024)
- [Image Compression with Product Quantized Masked Image Modeling](https://mlsanity.com/paper/2212.07372/) (2022)
- [StableCodec: Taming One-Step Diffusion for Extreme Image Compression](https://mlsanity.com/paper/2506.21977/) (2025)
- [HEMGS: A Hybrid Entropy Model for 3D Gaussian Splatting Data Compression](https://mlsanity.com/paper/2411.18473/) (2024)
- [Learned Image Compression with Dictionary-based Entropy Model](https://mlsanity.com/paper/2504.00496/) (2025)
- [MambaVC: Learned Visual Compression with Selective State Spaces](https://mlsanity.com/paper/2405.15413/) (2024)
- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [RDEIC: Accelerating Diffusion-Based Extreme Image Compression with Relay Residual Diffusion](https://mlsanity.com/paper/2410.02640/) (2024)
- [Linear Attention Modeling for Learned Image Compression](https://mlsanity.com/paper/2502.05741/) (2025)
- [Deep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression](https://mlsanity.com/paper/2209.04847/) (2022)
- [CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer](https://mlsanity.com/paper/2405.03376/) (2024)
- [Context-Based Trit-Plane Coding for Progressive Image Compression](https://mlsanity.com/paper/2303.05715/) (2023)
- [Learned Image Compression with Hierarchical Progressive Context Modeling](https://mlsanity.com/paper/2507.19125/) (2025)
- [High-Efficiency Lossy Image Coding Through Adaptive Neighborhood Information Aggregation](https://mlsanity.com/paper/2204.11448/) (2022)
- [MambaIC: State Space Models for High-Performance Learned Image Compression](https://mlsanity.com/paper/2503.12461/) (2025)
- [Causal Context Adjustment Loss for Learned Image Compression](https://mlsanity.com/paper/2410.04847/) (2024)
- [Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model](https://mlsanity.com/paper/2405.16817/) (2024)
- [Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptation](https://mlsanity.com/paper/2406.00758/) (2024)
- [A Comprehensive Survey of Transformers for Computer Vision](https://mlsanity.com/paper/2211.06004/) (2022)
- [M2T: Masking Transformers Twice for Faster Decoding](https://mlsanity.com/paper/2304.07313/) (2023)

## Abstract (arXiv)

One critical component in lossy deep image compression is the entropy model, which predicts the probability distribution of the quantized latent representation in the encoding and decoding modules. Previous works build entropy models upon convolutional neural networks which are inefficient in capturing global dependencies. In this work, we propose a novel transformer-based entropy model, termed Entroformer, to capture long-range dependencies in probability distribution estimation effectively and efficiently. Different from vision transformers in image classification, the Entroformer is highly optimized for image compression, including a top-k self-attention and a diamond relative position encoding. Meanwhile, we further expand this architecture with a parallel bidirectional context model to speed up the decoding process. The experiments show that the Entroformer achieves state-of-the-art performance on image compression while being time-efficient.