# Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression

arXiv: https://arxiv.org/abs/2203.02452 · HTML: https://mlsanity.com/paper/2203.02452/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Density Modeling of Images using a Generalized Normalization Transformation](https://mlsanity.com/paper/1511.06281/) (2015)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Involution: Inverting the Inherence of Convolution for Visual Recognition](https://mlsanity.com/paper/2103.06255/) (2021)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Learning Accurate Entropy Model with Global Reference for Image Compression](https://mlsanity.com/paper/2010.08321/) (2020)

## Built on by (most cited first, 30 of 45)

- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting](https://mlsanity.com/paper/2403.08551/) (2024)
- [MLIC: Multi-Reference Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2211.07273/) (2022)
- [Multi-Realism Image Compression with a Conditional Generator](https://mlsanity.com/paper/2212.13824/) (2022)
- [MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression](https://mlsanity.com/paper/2307.15421/) (2023)
- [Frequency-Aware Transformer for Learned Image Compression](https://mlsanity.com/paper/2310.16387/) (2023)
- [MambaVC: Learned Visual Compression with Selective State Spaces](https://mlsanity.com/paper/2405.15413/) (2024)
- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2305.19185/) (2023)
- [CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer](https://mlsanity.com/paper/2405.03376/) (2024)
- [Causal Context Adjustment Loss for Learned Image Compression](https://mlsanity.com/paper/2410.04847/) (2024)
- [Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model](https://mlsanity.com/paper/2405.16817/) (2024)
- [M2T: Masking Transformers Twice for Faster Decoding](https://mlsanity.com/paper/2304.07313/) (2023)
- [Efficient Contextformer: Spatio-Channel Window Attention for Fast Context Modeling in Learned Image Compression](https://mlsanity.com/paper/2306.14287/) (2023)
- [Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation](https://mlsanity.com/paper/2309.02529/) (2023)
- [ECSIC: Epipolar Cross Attention for Stereo Image Compression](https://mlsanity.com/paper/2307.10284/) (2023)
- [Region-Adaptive Transform with Segmentation Prior for Image Compression](https://mlsanity.com/paper/2403.00628/) (2024)
- [LLIC: Large Receptive Field Transform Coding with Adaptive Weights for Learned Image Compression](https://mlsanity.com/paper/2304.09571/) (2023)
- [Another Way to the Top: Exploit Contextual Clustering in Learned Image Coding](https://mlsanity.com/paper/2401.11615/) (2024)
- [GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression](https://mlsanity.com/paper/2405.01170/) (2024)
- [Multistage Spatial Context Models for Learned Image Compression](https://mlsanity.com/paper/2302.09263/) (2023)
- [Multi-Scale Invertible Neural Network for Wide-Range Variable-Rate Learned Image Compression](https://mlsanity.com/paper/2503.21284/) (2025)
- [Laplacian-guided Entropy Model in Neural Codec with Blur-dissipated Synthesis](https://mlsanity.com/paper/2403.16258/) (2024)
- [Exploring Effective Mask Sampling Modeling for Neural Image Compression](https://mlsanity.com/paper/2306.05704/) (2023)
- [ConvNeXt-ChARM: ConvNeXt-based Transform for Efficient Neural Image Compression](https://mlsanity.com/paper/2307.06342/) (2023)
- [Corner-to-Center Long-range Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2311.18103/) (2023)
- [Bit Distribution Study and Implementation of Spatial Quality Map in the JPEG-AI Standardization](https://mlsanity.com/paper/2402.17470/) (2024)
- [Bit Rate Matching Algorithm Optimization in JPEG-AI Verification Model](https://mlsanity.com/paper/2402.17487/) (2024)

## Abstract (arXiv)

Entropy modeling is a key component for high-performance image compression algorithms. Recent developments in autoregressive context modeling helped learning-based methods to surpass their classical counterparts. However, the performance of those models can be further improved due to the underexploited spatio-channel dependencies in latent space, and the suboptimal implementation of context adaptivity. Inspired by the adaptive characteristics of the transformers, we propose a transformer-based context model, named Contextformer, which generalizes the de facto standard attention mechanism to spatio-channel attention. We replace the context model of a modern compression framework with the Contextformer and test it on the widely used Kodak, CLIC2020, and Tecnick image datasets. Our experimental results show that the proposed model provides up to 11% rate savings compared to the standard Versatile Video Coding (VVC) Test Model (VTM) 16.2, and outperforms various learning-based models in terms of PSNR and MS-SSIM.