# Transformer-based Image Compression

arXiv: https://arxiv.org/abs/2111.06707 · HTML: https://mlsanity.com/paper/2111.06707/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [A Unified End-to-End Framework for Efficient Deep Image Compression](https://mlsanity.com/paper/2002.03370/) (2020)

## Built on by (most cited first, 30 of 62)

- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [MLIC: Multi-Reference Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2211.07273/) (2022)
- [MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression](https://mlsanity.com/paper/2307.15421/) (2023)
- [Frequency-Aware Transformer for Learned Image Compression](https://mlsanity.com/paper/2310.16387/) (2023)
- [Lossy Image Compression with Quantized Hierarchical VAEs](https://mlsanity.com/paper/2208.13056/) (2022)
- [QARV: Quantization-Aware ResNet VAE for Lossy Image Compression](https://mlsanity.com/paper/2302.08899/) (2023)
- [MambaVC: Learned Visual Compression with Selective State Spaces](https://mlsanity.com/paper/2405.15413/) (2024)
- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [Towards Robust Neural Image Compression: Adversarial Attack and Model Finetuning](https://mlsanity.com/paper/2112.08691/) (2021)
- [Linear Attention Modeling for Learned Image Compression](https://mlsanity.com/paper/2502.05741/) (2025)
- [High-Efficiency Lossy Image Coding Through Adaptive Neighborhood Information Aggregation](https://mlsanity.com/paper/2204.11448/) (2022)
- [MambaIC: State Space Models for High-Performance Learned Image Compression](https://mlsanity.com/paper/2503.12461/) (2025)
- [Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion](https://mlsanity.com/paper/2505.08281/) (2025)
- [Causal Context Adjustment Loss for Learned Image Compression](https://mlsanity.com/paper/2410.04847/) (2024)
- [Efficient Visual Computing with Camera RAW Snapshots](https://mlsanity.com/paper/2212.07778/) (2022)
- [All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation](https://mlsanity.com/paper/2409.19660/) (2024)
- [End-to-End Learnable Multi-Scale Feature Compression for VCM](https://mlsanity.com/paper/2306.16670/) (2023)
- [FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing](https://mlsanity.com/paper/2302.10681/) (2023)
- [Rate-Distortion-Cognition Controllable Versatile Neural Image Compression](https://mlsanity.com/paper/2407.11700/) (2024)
- [Rate-Distortion Optimized Post-Training Quantization for Learned Image Compression](https://mlsanity.com/paper/2211.02854/) (2022)
- [ROI-based Deep Image Compression with Swin Transformers](https://mlsanity.com/paper/2305.07783/) (2023)
- [Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression](https://mlsanity.com/paper/2308.07733/) (2023)
- [Transformer-based Variable-rate Image Compression with Region-of-interest Control](https://mlsanity.com/paper/2305.10807/) (2023)
- [Image Coding for Machines with Object Region Learning](https://mlsanity.com/paper/2308.13984/) (2023)
- [Efficient LiDAR Point Cloud Geometry Compression Through Neighborhood Point Attention](https://mlsanity.com/paper/2208.12573/) (2022)
- [LLIC: Large Receptive Field Transform Coding with Adaptive Weights for Learned Image Compression](https://mlsanity.com/paper/2304.09571/) (2023)
- [Large Language Models as Computable Approximations to Solomonoff Induction](https://mlsanity.com/paper/2505.15784/) (2025)
- [Content-Aware Mamba for Learned Image Compression](https://mlsanity.com/paper/2508.02192/) (2025)
- [Progressive Learning with Visual Prompt Tuning for Variable-Rate Image Compression](https://mlsanity.com/paper/2311.13846/) (2023)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)

## Abstract (arXiv)

A Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. Both main and hyper encoders are comprised of a sequence of neural transformation units (NTUs) to analyse and aggregate important information for more compact representation of input image, while the decoders mirror the encoder-side operations to generate pixel-domain image reconstruction from the compressed bitstream. Each NTU is consist of a Swin Transformer Block (STB) and a convolutional layer (Conv) to best embed both long-range and short-range information; In the meantime, a casual attention module (CAM) is devised for adaptive context modeling of latent features to utilize both hyper and autoregressive priors. The TIC rivals with state-of-the-art approaches including deep convolutional neural networks (CNNs) based learnt image coding (LIC) methods and handcrafted rules-based intra profile of recently-approved Versatile Video Coding (VVC) standard, and requires much less model parameters, e.g., up to 45% reduction to leading-performance LIC.