# Efficient Contextformer: Spatio-Channel Window Attention for Fast Context Modeling in Learned Image Compression

arXiv: https://arxiv.org/abs/2306.14287 · HTML: https://mlsanity.com/paper/2306.14287/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 75 references that are in the map.
- First posted on arXiv 25 Jun 2023.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Density Modeling of Images using a Generalized Normalization Transformation](https://mlsanity.com/paper/1511.06281/) (2015)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)

## Built on by (most cited first, 13 of 13)

- [DeepHQ: Learned Hierarchical Quantizer for Progressive Deep Image Coding](https://mlsanity.com/paper/2408.12150/) (2024)
- [FViT: A Focal Vision Transformer with Gabor Filter](https://mlsanity.com/paper/2402.11303/) (2024)
- [Customizable ROI-Based Deep Image Compression](https://mlsanity.com/paper/2507.00373/) (2025)
- [Context Guided Transformer Entropy Modeling for Video Compression](https://mlsanity.com/paper/2508.01852/) (2025)
- [Bit Distribution Study and Implementation of Spatial Quality Map in the JPEG-AI Standardization](https://mlsanity.com/paper/2402.17470/) (2024)
- [Bit Rate Matching Algorithm Optimization in JPEG-AI Verification Model](https://mlsanity.com/paper/2402.17487/) (2024)
- [Lightweight Embedded FPGA Deployment of Learned Image Compression with Knowledge Distillation and Hybrid Quantization](https://mlsanity.com/paper/2503.04832/) (2025)
- [FIPER: Factorized Features for Robust Image Super-Resolution and Compression](https://mlsanity.com/paper/2410.18083/) (2024)
- [Deeply-Conditioned Image Compression via Self-Generated Priors](https://mlsanity.com/paper/2510.24437/) (2025)
- [Adapting Learned Image Codecs to Screen Content via Adjustable Transformations](https://mlsanity.com/paper/2402.17544/) (2024)
- [Multi-Scale Feature Prediction with Auxiliary-Info for Neural Image Compression](https://mlsanity.com/paper/2409.12719/) (2024)
- [Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding](https://mlsanity.com/paper/2608.08832/) (2026)
- [You've Seen Enough: Quality-Constrained Image Coding for Machines](https://mlsanity.com/paper/2609.25108/) (2026)

## Abstract (arXiv)

Entropy estimation is essential for the performance of learned image compression. It has been demonstrated that a transformer-based entropy model is of critical importance for achieving a high compression ratio, however, at the expense of a significant computational effort. In this work, we introduce the Efficient Contextformer (eContextformer) - a computationally efficient transformer-based autoregressive context model for learned image compression. The eContextformer efficiently fuses the patch-wise, checkered, and channel-wise grouping techniques for parallel context modeling, and introduces a shifted window spatio-channel attention mechanism. We explore better training strategies and architectural designs and introduce additional complexity optimizations. During decoding, the proposed optimization techniques dynamically scale the attention span and cache the previous attention computations, drastically reducing the model and runtime complexity. Compared to the non-parallel approach, our proposal has ~145x lower model complexity and ~210x faster decoding speed, and achieves higher average bit savings on Kodak, CLIC2020, and Tecnick datasets. Additionally, the low complexity of our context model enables online rate-distortion algorithms, which further improve the compression performance. We achieve up to 17% bitrate savings over the intra coding of Versatile Video Coding (VVC) Test Model (VTM) 16.2 and surpass various learning-based compression models.