# FEDS: Feature and Entropy-Based Distillation Strategy for Efficient Learned Image Compression

arXiv: https://arxiv.org/abs/2503.06399 · HTML: https://mlsanity.com/paper/2503.06399/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Focal and Global Knowledge Distillation for Detectors](https://mlsanity.com/paper/2111.11837/) (2021)
- [Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution](https://mlsanity.com/paper/1904.05049/) (2019)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [MLIC: Multi-Reference Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2211.07273/) (2022)
- [Learning End-to-End Lossy Image Compression: A Benchmark](https://mlsanity.com/paper/2002.03711/) (2020)
- [Joint Global and Local Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/2112.04487/) (2021)
- [Distilling portable Generative Adversarial Networks for Image Translation](https://mlsanity.com/paper/2003.03519/) (2020)
- [Learned Image Compression with Gaussian-Laplacian-Logistic Mixture Model and Concatenated Residual Modules](https://mlsanity.com/paper/2107.06463/) (2021)
- [A Comprehensive Benchmark for Single Image Compression Artifacts Reduction](https://mlsanity.com/paper/1909.03647/) (2019)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [Learned Multi-Resolution Variable-Rate Image Compression with Octave-based Residual Blocks](https://mlsanity.com/paper/2012.15463/) (2020)
- [Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation](https://mlsanity.com/paper/2309.02529/) (2023)

## Built on by (most cited first, 1 of 1)

- [KD-NVC: A Search-and-Distill Framework to Accelerate Neural Video Coding](https://mlsanity.com/paper/2606.04595/) (2026)

## Abstract (arXiv)

Learned image compression (LIC) methods have recently outperformed traditional codecs such as VVC in rate-distortion performance. However, their large models and high computational costs have limited their practical adoption. In this paper, we first construct a high-capacity teacher model by integrating Swin-Transformer V2-based attention modules, additional residual blocks, and expanded latent channels, thus achieving enhanced compression performance. Building on this foundation, we propose a underlineFeature and underlineEntropy-based underlineDistillation underlineStrategy (FEDS) that transfers key knowledge from the teacher to a lightweight student model. Specifically, we align intermediate feature representations and emphasize the most informative latent channels through an entropy-based loss. A staged training scheme refines this transfer in three phases: feature alignment, channel-level distillation, and final fine-tuning. Our student model nearly matches the teacher across Kodak (1.24% BD-Rate increase), Tecnick (1.17%), and CLIC (0.55%) while cutting parameters by about 63% and accelerating encoding/decoding by around 73%. Moreover, ablation studies indicate that FEDS generalizes effectively to transformer-based networks. The experimental results demonstrate our approach strikes a compelling balance among compression performance, speed, and model parameters, making it well-suited for real-time or resource-limited scenarios.