# Frequency-Aware Transformer for Learned Image Compression

arXiv: https://arxiv.org/abs/2310.16387 · HTML: https://mlsanity.com/paper/2310.16387/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [Fast Vision Transformers with HiLo Attention](https://mlsanity.com/paper/2205.13213/) (2022)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Transformer-based Image Compression](https://mlsanity.com/paper/2111.06707/) (2021)
- [A Unified End-to-End Framework for Efficient Deep Image Compression](https://mlsanity.com/paper/2002.03370/) (2020)
- [Motion-aware Contrastive Video Representation Learning via Foreground-background Merging](https://mlsanity.com/paper/2109.15130/) (2021)
- [Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression](https://mlsanity.com/paper/2203.02452/) (2022)
- [Pose-Oriented Transformer with Uncertainty-Guided Refinement for 2D-to-3D Human Pose Estimation](https://mlsanity.com/paper/2302.07408/) (2023)
- [Dual Contrastive Learning for Spatio-temporal Representation](https://mlsanity.com/paper/2207.05340/) (2022)
- [Frequency Disentangled Features in Neural Image Compression](https://mlsanity.com/paper/2308.02620/) (2023)

## Built on by (most cited first, 30 of 58)

- [Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance](https://mlsanity.com/paper/2402.02149/) (2024)
- [Learned Image Compression with Dictionary-based Entropy Model](https://mlsanity.com/paper/2504.00496/) (2025)
- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [RDEIC: Accelerating Diffusion-Based Extreme Image Compression with Relay Residual Diffusion](https://mlsanity.com/paper/2410.02640/) (2024)
- [Linear Attention Modeling for Learned Image Compression](https://mlsanity.com/paper/2502.05741/) (2025)
- [Learned Image Compression with Hierarchical Progressive Context Modeling](https://mlsanity.com/paper/2507.19125/) (2025)
- [Causal Context Adjustment Loss for Learned Image Compression](https://mlsanity.com/paper/2410.04847/) (2024)
- [Large Language Model for Lossless Image Compression with Visual Prompts](https://mlsanity.com/paper/2502.16163/) (2025)
- [Rate-Distortion-Cognition Controllable Versatile Neural Image Compression](https://mlsanity.com/paper/2407.11700/) (2024)
- [Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation](https://mlsanity.com/paper/2311.17893/) (2023)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [HiPART: Hierarchical Pose AutoRegressive Transformer for Occluded 3D Human Pose Estimation](https://mlsanity.com/paper/2503.23331/) (2025)
- [Content-Aware Mamba for Learned Image Compression](https://mlsanity.com/paper/2508.02192/) (2025)
- [Balanced Rate-Distortion Optimization in Learned Image Compression](https://mlsanity.com/paper/2502.20161/) (2025)
- [Dual-Representation Image Compression at Ultra-Low Bitrates via Explicit Semantics and Implicit Textures](https://mlsanity.com/paper/2602.05213/) (2026)
- [Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression](https://mlsanity.com/paper/2604.10546/) (2026)
- [High Frequency Matters: Uncertainty Guided Image Compression with Wavelet Diffusion](https://mlsanity.com/paper/2407.12538/) (2024)
- [A Lightweight Model for Perceptual Image Compression via Implicit Priors](https://mlsanity.com/paper/2502.13988/) (2025)
- [Synonymous Variational Inference for Perceptual Image Compression](https://mlsanity.com/paper/2505.22438/) (2025)
- [Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior](https://mlsanity.com/paper/2511.22549/) (2025)
- [Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors](https://mlsanity.com/paper/2603.15129/) (2026)
- [3DM-WeConvene: Learned Image Compression with 3D Multi-Level Wavelet-Domain Convolution and Entropy Model](https://mlsanity.com/paper/2504.04658/) (2025)
- [Neural Stereo Video Compression with Hybrid Disparity Compensation](https://mlsanity.com/paper/2504.20383/) (2025)
- [Scaling Learned Image Compression Models up to 1 Billion](https://mlsanity.com/paper/2508.09075/) (2025)
- [Switchable Token-Specific Codebook Quantization For Face Image Compression](https://mlsanity.com/paper/2510.22943/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence](https://mlsanity.com/paper/2601.20769/) (2026)
- [Rethinking Learned Image Compression: Context is All You Need](https://mlsanity.com/paper/2407.11590/) (2024)
- [Leveraging Diffusion Knowledge for Generative Image Compression with Fractal Frequency-Aware Band Learning](https://mlsanity.com/paper/2503.11321/) (2025)
- [All-in-One Transferring Image Compression from Human Perception to Multi-Machine Perception](https://mlsanity.com/paper/2504.12997/) (2025)

## Abstract (arXiv)

Learned image compression (LIC) has gained traction as an effective solution for image storage and transmission in recent years. However, existing LIC methods are redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these challenges, we propose a novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LIC. The FAT block comprises frequency-decomposition window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Additionally, we introduce frequency-modulation feed-forward network (FMFFN) to adaptively modulate different frequency components, improving rate-distortion performance. Furthermore, we present a transformer-based channel-wise autoregressive (T-CA) model that effectively exploits channel dependencies. Experiments show that our method achieves state-of-the-art rate-distortion performance compared to existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 by 14.5%, 15.1%, 13.0% in BD-rate on the Kodak, Tecnick, and CLIC datasets.