# Neural Video Compression with Feature Modulation

arXiv: https://arxiv.org/abs/2402.17414 · HTML: https://mlsanity.com/paper/2402.17414/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Lossy Compression with Gaussian Diffusion](https://mlsanity.com/paper/2206.08889/) (2022)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [CANF-VC: Conditional Augmented Normalizing Flows for Video Compression](https://mlsanity.com/paper/2207.05315/) (2022)
- [Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression](https://mlsanity.com/paper/2203.02452/) (2022)
- [Conditional Entropy Coding for Efficient Video Compression](https://mlsanity.com/paper/2008.09180/) (2020)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models](https://mlsanity.com/paper/2211.07793/) (2022)
- [Conditional Coding for Flexible Learned Video Compression](https://mlsanity.com/paper/2104.07930/) (2021)
- [Neural Compression-Based Feature Learning for Video Restoration](https://mlsanity.com/paper/2203.09208/) (2022)
- [Computationally-Efficient Neural Image Compression with Shallow Decoders](https://mlsanity.com/paper/2304.06244/) (2023)
- [Optical Flow and Mode Selection for Learning-based Video Coding](https://mlsanity.com/paper/2008.02580/) (2020)

## Built on by (most cited first, 30 of 123)

- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s](https://mlsanity.com/paper/2409.08481/) (2024)
- [One-Step Diffusion-Based Image Compression with Semantic Distillation](https://mlsanity.com/paper/2505.16687/) (2025)
- [Generative Latent Coding for Ultra-Low Bitrate Image and Video Compression](https://mlsanity.com/paper/2505.16177/) (2025)
- [Generative Latent Coding for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2512.20194/) (2025)
- [DLF: Extreme Image Compression with Dual-generative Latent Fusion](https://mlsanity.com/paper/2503.01428/) (2025)
- [Learned Image Compression with Hierarchical Progressive Context Modeling](https://mlsanity.com/paper/2507.19125/) (2025)
- [Lossless data compression by large models](https://mlsanity.com/paper/2407.07723/) (2024)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [PNVC: Towards Practical INR-based Video Compression](https://mlsanity.com/paper/2409.00953/) (2024)
- [Neural Video Compression with Context Modulation](https://mlsanity.com/paper/2505.14541/) (2025)
- [Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance](https://mlsanity.com/paper/2512.07480/) (2025)
- [Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression](https://mlsanity.com/paper/2405.04274/) (2024)
- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [DiffVC-OSD: One-Step Diffusion-based Perceptual Neural Video Compression Framework](https://mlsanity.com/paper/2508.07682/) (2025)
- [Generative Latent Video Compression](https://mlsanity.com/paper/2510.09987/) (2025)
- [M3-CVC: Controllable Video Compression with Multimodal Generative Models](https://mlsanity.com/paper/2411.15798/) (2024)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates](https://mlsanity.com/paper/2507.07633/) (2025)
- [AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2601.00796/) (2026)
- [Generalized Gaussian Model for Learned Image Compression](https://mlsanity.com/paper/2411.19320/) (2024)
- [Augmented Deep Contexts for Spatially Embedded Video Coding](https://mlsanity.com/paper/2505.05309/) (2025)
- [BiECVC: Gated Diversification of Bidirectional Contexts for Learned Video Compression](https://mlsanity.com/paper/2505.09193/) (2025)
- [GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting](https://mlsanity.com/paper/2501.12060/) (2025)
- [EHVC: Efficient Hierarchical Reference and Quality Structure for Neural Video Coding](https://mlsanity.com/paper/2509.04118/) (2025)
- [Benchmarking Conventional and Learned Video Codecs with a Low-Delay Configuration](https://mlsanity.com/paper/2408.05042/) (2024)
- [3D Gaussian Splatting Data Compression with Mixture of Priors](https://mlsanity.com/paper/2505.03310/) (2025)
- [Ultra-Fast Neural Video Compression](https://mlsanity.com/paper/2606.04410/) (2026)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)

## Abstract (arXiv)

The emerging conditional coding-based neural video codec (NVC) shows superiority over commonly-used residual coding-based codec and the latest NVC already claims to outperform the best traditional codec. However, there still exist critical problems blocking the practicality of NVC. In this paper, we propose a powerful conditional coding-based NVC that solves two critical problems via feature modulation. The first is how to support a wide quality range in a single model. Previous NVC with this capability only supports about 3.8 dB PSNR range on average. To tackle this limitation, we modulate the latent feature of the current frame via the learnable quantization scaler. During the training, we specially design the uniform quantization parameter sampling mechanism to improve the harmonization of encoding and quantization. This results in a better learning of the quantization scaler and helps our NVC support about 11.4 dB PSNR range. The second is how to make NVC still work under a long prediction chain. We expose that the previous SOTA NVC has an obvious quality degradation problem when using a large intra-period setting. To this end, we propose modulating the temporal feature with a periodically refreshing mechanism to boost the quality. %Besides solving the above two problems, we also design a single model that can support both RGB and YUV colorspaces. Notably, under single intra-frame setting, our codec can achieve 29.7% bitrate saving over previous SOTA NVC with 16% MACs reduction. Our codec serves as a notable landmark in the journey of NVC evolution. The codes are at https://github.com/microsoft/DCVC.