# Towards Practical Real-Time Neural Video Compression

arXiv: https://arxiv.org/abs/2502.20762 · HTML: https://mlsanity.com/paper/2502.20762/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow](https://mlsanity.com/paper/2405.14793/) (2024)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting](https://mlsanity.com/paper/2403.08551/) (2024)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [C3: High-performance and low-complexity neural compression from a single image or video](https://mlsanity.com/paper/2312.02753/) (2023)
- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [CANF-VC: Conditional Augmented Normalizing Flows for Video Compression](https://mlsanity.com/paper/2207.05315/) (2022)
- [Conditional Entropy Coding for Efficient Video Compression](https://mlsanity.com/paper/2008.09180/) (2020)
- [Overfitting for Fun and Profit: Instance-Adaptive Data Compression](https://mlsanity.com/paper/2101.08687/) (2021)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [Conditional Coding for Flexible Learned Video Compression](https://mlsanity.com/paper/2104.07930/) (2021)
- [Computationally-Efficient Neural Image Compression with Shallow Decoders](https://mlsanity.com/paper/2304.06244/) (2023)
- [MobileCodec: Neural Inter-frame Video Compression on Mobile Devices](https://mlsanity.com/paper/2207.08338/) (2022)
- [PNVC: Towards Practical INR-based Video Compression](https://mlsanity.com/paper/2409.00953/) (2024)
- [Post-Training Quantization for Cross-Platform Learned Image Compression](https://mlsanity.com/paper/2202.07513/) (2022)

## Built on by (most cited first, 30 of 80)

- [Learned Image Compression with Hierarchical Progressive Context Modeling](https://mlsanity.com/paper/2507.19125/) (2025)
- [Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance](https://mlsanity.com/paper/2512.07480/) (2025)
- [Generative Latent Video Compression](https://mlsanity.com/paper/2510.09987/) (2025)
- [Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model](https://mlsanity.com/paper/2607.24904/) (2026)
- [Ultra-Low Bitrate Perceptual Image Compression with Shallow Encoder](https://mlsanity.com/paper/2512.12229/) (2025)
- [CoD-Lite: Real-Time Diffusion-Based Generative Image Compression](https://mlsanity.com/paper/2604.12525/) (2026)
- [Ultra-Fast Neural Video Compression](https://mlsanity.com/paper/2606.04410/) (2026)
- [Conditional Video Generation for High-Efficiency Video Compression](https://mlsanity.com/paper/2507.15269/) (2025)
- [Neural B-frame Video Compression with Bi-directional Reference Harmonization](https://mlsanity.com/paper/2511.08938/) (2025)
- [DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression](https://mlsanity.com/paper/2603.13162/) (2026)
- [GVCC: Zero-Shot Video Compression via Codebook-Driven Stochastic Rectified Flow](https://mlsanity.com/paper/2603.26571/) (2026)
- [Real-Time Neural Video Compression with Unified Intra and Inter Coding](https://mlsanity.com/paper/2510.14431/) (2025)
- [Low-Bitrate Video Compression through Semantic-Conditioned Diffusion](https://mlsanity.com/paper/2512.00408/) (2025)
- [Generative Video Compression with One-Dimensional Latent Representation](https://mlsanity.com/paper/2603.15302/) (2026)
- [Revisiting MLLM Token Technology through the Lens of Classical Visual Coding](https://mlsanity.com/paper/2508.13460/) (2025)
- [When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs](https://mlsanity.com/paper/2509.24258/) (2025)
- [DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression](https://mlsanity.com/paper/2601.20564/) (2026)
- [ProGIC: Progressive and Lightweight Generative Image Compression with Residual Vector Quantization](https://mlsanity.com/paper/2603.02897/) (2026)
- [Enhancing Neural Video Compression of Static Scenes with Positive-Incentive Noise](https://mlsanity.com/paper/2603.06095/) (2026)
- [What Matters in Practical Learned Image Compression](https://mlsanity.com/paper/2605.05148/) (2026)
- [ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors](https://mlsanity.com/paper/2606.22371/) (2026)
- [Enhanced Neural Video Representation Compression Across Extreme Complexity and Quality Scales](https://mlsanity.com/paper/2606.28163/) (2026)
- [ECoNGS: Efficient Compressive Neural Gaussian Splats for Volume Visualization](https://mlsanity.com/paper/2607.18466/) (2026)
- [Neural Stereo Video Compression with Hybrid Disparity Compensation](https://mlsanity.com/paper/2504.20383/) (2025)
- [SIEDD: Shared-Implicit Encoder with Discrete Decoders](https://mlsanity.com/paper/2506.23382/) (2025)
- [Scaling Learned Image Compression Models up to 1 Billion](https://mlsanity.com/paper/2508.09075/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence](https://mlsanity.com/paper/2601.20769/) (2026)
- [TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Data](https://mlsanity.com/paper/2602.09893/) (2026)
- [Compression as Adaptation: Implicit Visual Representation with Diffusion Foundation Models](https://mlsanity.com/paper/2603.07615/) (2026)

## Abstract (arXiv)

We introduce a practical real-time neural video codec (NVC) designed to deliver high compression ratio, low latency and broad versatility. In practice, the coding speed of NVCs depends on 1) computational costs, and 2) non-computational operational costs, such as memory I/O and the number of function calls. While most efficient NVCs prioritize reducing computational cost, we identify operational cost as the primary bottleneck to achieving higher coding speed. Leveraging this insight, we introduce a set of efficiency-driven design improvements focused on minimizing operational costs. Specifically, we employ implicit temporal modeling to eliminate complex explicit motion modules, and use single low-resolution latent representations rather than progressive downsampling. These innovations significantly accelerate NVC without sacrificing compression quality. Additionally, we implement model integerization for consistent cross-device coding and a module-bank-based rate control scheme to improve practical adaptability. Experiments show our proposed DCVC-RT achieves an impressive average encoding/decoding speed at 125.2/112.8 fps (frames per second) for 1080p video, while saving an average of 21% in bitrate compared to H.266/VTM. The code is available at https://github.com/microsoft/DCVC.