# Towards End-to-End Image Compression and Analysis with Transformers

arXiv: https://arxiv.org/abs/2112.09300 · HTML: https://mlsanity.com/paper/2112.09300/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Visformer: The Vision-friendly Transformer](https://mlsanity.com/paper/2104.12533/) (2021)
- [Density Modeling of Images using a Generalized Normalization Transformation](https://mlsanity.com/paper/1511.06281/) (2015)
- [Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics](https://mlsanity.com/paper/2001.03569/) (2020)
- [Learning Convolutional Networks for Content-weighted Image Compression](https://mlsanity.com/paper/1703.10553/) (2017)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Learning End-to-End Lossy Image Compression: A Benchmark](https://mlsanity.com/paper/2002.03711/) (2020)

## Built on by (most cited first, 26 of 26)

- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [Recent Advances in Vision Transformer: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2203.01536/) (2022)
- [A Coding Framework and Benchmark towards Low-Bitrate Video Understanding](https://mlsanity.com/paper/2202.02813/) (2022)
- [End-to-End Learnable Multi-Scale Feature Compression for VCM](https://mlsanity.com/paper/2306.16670/) (2023)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Semantic Segmentation in Learned Compressed Domain](https://mlsanity.com/paper/2209.01355/) (2022)
- [Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling](https://mlsanity.com/paper/2211.06797/) (2022)
- [LLIC: Large Receptive Field Transform Coding with Adaptive Weights for Learned Image Compression](https://mlsanity.com/paper/2304.09571/) (2023)
- [GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression](https://mlsanity.com/paper/2405.01170/) (2024)
- [CALLIC: Content Adaptive Learning for Lossless Image Compression](https://mlsanity.com/paper/2412.17464/) (2024)
- [An Efficient Adaptive Compression Method for Human Perception and Machine Vision Tasks](https://mlsanity.com/paper/2501.04329/) (2025)
- [Towards Point Cloud Compression for Machine Perception: A Simple and Strong Baseline by Learning the Octree Depth Level Predictor](https://mlsanity.com/paper/2406.00791/) (2024)
- [ROI-Guided Point Cloud Geometry Compression Towards Human and Machine Vision](https://mlsanity.com/paper/2504.14240/) (2025)
- [You Can Mask More For Extremely Low-Bitrate Image Compression](https://mlsanity.com/paper/2306.15561/) (2023)
- [Multiscale Feature Importance-based Bit Allocation for End-to-End Feature Coding for Machines](https://mlsanity.com/paper/2503.19278/) (2025)
- [Wavelet-Like Transform-Based Technology in Response to the Call for Proposals on Neural Network-Based Image Coding](https://mlsanity.com/paper/2403.05937/) (2024)
- [Multi-spectral Entropy Constrained Neural Compression of Solar Imagery](https://mlsanity.com/paper/2309.10791/) (2023)
- [All-in-One Transferring Image Compression from Human Perception to Multi-Machine Perception](https://mlsanity.com/paper/2504.12997/) (2025)
- [DynaQuant: Dynamic Mixed-Precision Quantization for Learned Image Compression](https://mlsanity.com/paper/2511.07903/) (2025)
- [Machines Serve Human: A Novel Variable Human-machine Collaborative Compression Framework](https://mlsanity.com/paper/2511.08915/) (2025)
- [CIC: Circular Image Compression](https://mlsanity.com/paper/2407.15870/) (2024)
- [RDD: Pareto Analysis of the Rate-Distortion-Distinguishability Trade-off](https://mlsanity.com/paper/2509.24805/) (2025)
- [Spectral and Spatial Graph Learning for Multispectral Solar Image Compression](https://mlsanity.com/paper/2512.24463/) (2025)
- [What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters](https://mlsanity.com/paper/2604.10017/) (2026)
- [DCVC-MB: Neural B-Frame Video Compression using State Space Models](https://mlsanity.com/paper/2607.14305/) (2026)
- [CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression](https://mlsanity.com/paper/2608.25568/) (2026)

## Abstract (arXiv)

We propose an end-to-end image compression and analysis model with Transformers, targeting to the cloud-based image classification application. Instead of placing an existing Transformer-based image classification model directly after an image codec, we aim to redesign the Vision Transformer (ViT) model to perform image classification from the compressed features and facilitate image compression with the long-term information from the Transformer. Specifically, we first replace the patchify stem (i.e., image splitting and embedding) of the ViT model with a lightweight image encoder modelled by a convolutional neural network. The compressed features generated by the image encoder are injected convolutional inductive bias and are fed to the Transformer for image classification bypassing image reconstruction. Meanwhile, we propose a feature aggregation module to fuse the compressed features with the selected intermediate features of the Transformer, and feed the aggregated features to a deconvolutional neural network for image reconstruction. The aggregated features can obtain the long-term information from the self-attention mechanism of the Transformer and improve the compression performance. The rate-distortion-accuracy optimization problem is finally solved by a two-step training strategy. Experimental results demonstrate the effectiveness of the proposed model in both the image compression and the classification tasks.