# Linear Attention Modeling for Learned Image Compression

arXiv: https://arxiv.org/abs/2502.05741 · HTML: https://mlsanity.com/paper/2502.05741/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 30 of 33)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](https://mlsanity.com/paper/2402.15648/) (2024)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence](https://mlsanity.com/paper/2404.05892/) (2024)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures](https://mlsanity.com/paper/2403.02308/) (2024)
- [VL-Mamba: Exploring State Space Models for Multimodal Learning](https://mlsanity.com/paper/2403.13600/) (2024)
- [Transformer-based Image Compression](https://mlsanity.com/paper/2111.06707/) (2021)
- [MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression](https://mlsanity.com/paper/2307.15421/) (2023)
- [Frequency-Aware Transformer for Learned Image Compression](https://mlsanity.com/paper/2310.16387/) (2023)
- [Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models](https://mlsanity.com/paper/2404.04478/) (2024)
- [Restore-RWKV: Efficient and Effective Medical Image Restoration with RWKV](https://mlsanity.com/paper/2407.11087/) (2024)
- [MambaVC: Learned Visual Compression with Selective State Spaces](https://mlsanity.com/paper/2405.15413/) (2024)
- [Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model](https://mlsanity.com/paper/2406.19369/) (2024)

## Built on by (most cited first, 23 of 23)

- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [Content-Aware Mamba for Learned Image Compression](https://mlsanity.com/paper/2508.02192/) (2025)
- [ProGIC: Progressive and Lightweight Generative Image Compression with Residual Vector Quantization](https://mlsanity.com/paper/2603.02897/) (2026)
- [Scaling Learned Image Compression Models up to 1 Billion](https://mlsanity.com/paper/2508.09075/) (2025)
- [Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence](https://mlsanity.com/paper/2601.20769/) (2026)
- [TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data](https://mlsanity.com/paper/2602.09893/) (2026)
- [CADC: Content Adaptive Diffusion-Based Generative Image Compression](https://mlsanity.com/paper/2602.21591/) (2026)
- [OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data](https://mlsanity.com/paper/2602.22286/) (2026)
- [Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compression](https://mlsanity.com/paper/2603.05756/) (2026)
- [Adaptive Learned Image Compression with Graph Neural Networks](https://mlsanity.com/paper/2603.25316/) (2026)
- [Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2606.01608/) (2026)
- [Variable Rate Image Compression via N-Gram Context based Swin-transformer](https://mlsanity.com/paper/2510.00058/) (2025)
- [A Compact Hybrid Convolution--Frequency State Space Network for Learned Image Compression](https://mlsanity.com/paper/2511.20151/) (2025)
- [Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulations](https://mlsanity.com/paper/2512.01213/) (2025)
- [SDGIC: A Semantic Disambiguation-Guided Generative Image Compression Method for Ultra-Low Bitrates](https://mlsanity.com/paper/2512.06344/) (2025)
- [HiDE: Hierarchical Dictionary-Based Entropy Modeling for Learned Image Compression](https://mlsanity.com/paper/2603.06766/) (2026)
- [Control Your View: High-Resolution Global Semantic Manipulation in Learned Image Compression](https://mlsanity.com/paper/2605.08727/) (2026)
- [Efficient Learned Image Compression without Entropy Coding](https://mlsanity.com/paper/2605.23323/) (2026)
- [MixCompress: Mixture of Experts for Variable Rate Learned Image Compression](https://mlsanity.com/paper/2607.14334/) (2026)
- [Hadamard-Domain Model Quantization for Learned Image Coding](https://mlsanity.com/paper/2608.01653/) (2026)
- [Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding](https://mlsanity.com/paper/2608.08832/) (2026)
- [Mixture-of-Experts-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2608.10947/) (2026)
- [Multiple Scale Latents for Learned Image Compression](https://mlsanity.com/paper/2608.10952/) (2026)

## Abstract (arXiv)

Recent years, learned image compression has made tremendous progress to achieve impressive coding efficiency. Its coding gain mainly comes from non-linear neural network-based transform and learnable entropy modeling. However, most studies focus on a strong backbone, and few studies consider a low complexity design. In this paper, we propose LALIC, a linear attention modeling for learned image compression. Specially, we propose to use Bi-RWKV blocks, by utilizing the Spatial Mix and Channel Mix modules to achieve more compact feature extraction, and apply the Conv based Omni-Shift module to adapt to two-dimensional latent representation. Furthermore, we propose a RWKV-based Spatial-Channel ConTeXt model (RWKV-SCCTX), that leverages the Bi-RWKV to modeling the correlation between neighboring features effectively. To our knowledge, our work is the first work to utilize efficient Bi-RWKV models with linear attention for learned image compression. Experimental results demonstrate that our method achieves competitive RD performances by outperforming VTM-9.1 by -15.26%, -15.41%, -17.63% in BD-rate on Kodak, CLIC and Tecnick datasets. The code is available at https://github.com/sjtu-medialab/RwkvCompress .