# ELF-VC: Efficient Learned Flexible-Rate Video Coding

arXiv: https://arxiv.org/abs/2104.14335 · HTML: https://mlsanity.com/paper/2104.14335/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Dual Path Networks](https://mlsanity.com/paper/1707.01629/) (2017)
- [Variable Rate Deep Image Compression With a Conditional Autoencoder](https://mlsanity.com/paper/1909.04802/) (2019)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [End-to-end optimization of nonlinear transform codes for perceptual quality](https://mlsanity.com/paper/1607.05006/) (2016)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [A Unified End-to-End Framework for Efficient Deep Image Compression](https://mlsanity.com/paper/2002.03370/) (2020)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [Conditional Entropy Coding for Efficient Video Compression](https://mlsanity.com/paper/2008.09180/) (2020)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [Computationally Efficient Neural Image Compression](https://mlsanity.com/paper/1912.08771/) (2019)
- [Feedback Recurrent Autoencoder for Video Compression](https://mlsanity.com/paper/2004.04342/) (2020)
- [Neural Video Compression using Spatio-Temporal Priors](https://mlsanity.com/paper/1902.07383/) (2019)
- [Deep Predictive Video Compression with Bi-directional Prediction](https://mlsanity.com/paper/1904.02909/) (2019)

## Built on by (most cited first, 30 of 69)

- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [HNeRV: A Hybrid Neural Representation for Videos](https://mlsanity.com/paper/2304.02633/) (2023)
- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [CANF-VC: Conditional Augmented Normalizing Flows for Video Compression](https://mlsanity.com/paper/2207.05315/) (2022)
- [DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos](https://mlsanity.com/paper/2304.06544/) (2023)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [A Residual Diffusion Model for High Perceptual Quality Codec Augmentation](https://mlsanity.com/paper/2301.05489/) (2023)
- [DS-NeRV: Implicit Neural Video Representation with Decomposed Static and Dynamic Codes](https://mlsanity.com/paper/2403.15679/) (2024)
- [Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression](https://mlsanity.com/paper/2401.15864/) (2024)
- [CNeRV: Content-adaptive Neural Representation for Visual Data](https://mlsanity.com/paper/2211.10421/) (2022)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [MobileCodec: Neural Inter-frame Video Compression on Mobile Devices](https://mlsanity.com/paper/2207.08338/) (2022)
- [Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set](https://mlsanity.com/paper/2111.10302/) (2021)
- [B-CANF: Adaptive B-frame Coding with Conditional Augmented Normalizing Flows](https://mlsanity.com/paper/2209.01769/) (2022)
- [Selective compression learning of latent representations for variable-rate image compression](https://mlsanity.com/paper/2211.04104/) (2022)
- [Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression](https://mlsanity.com/paper/2112.13309/) (2021)
- [MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device](https://mlsanity.com/paper/2310.01258/) (2023)
- [Neural Face Video Compression using Multiple Views](https://mlsanity.com/paper/2203.15401/) (2022)
- [Fast Encoding and Decoding for Implicit Video Representation](https://mlsanity.com/paper/2409.19429/) (2024)
- [On the Choice of Perception Loss Function for Learned Video Compression](https://mlsanity.com/paper/2305.19301/) (2023)
- [Towards Real-Time Neural Video Codec for Cross-Platform Application Using Calibration Information](https://mlsanity.com/paper/2309.11276/) (2023)
- [GaussianVideo: Efficient Video Representation and Compression by Gaussian Splatting](https://mlsanity.com/paper/2503.04333/) (2025)
- [Augmented Deep Contexts for Spatially Embedded Video Coding](https://mlsanity.com/paper/2505.05309/) (2025)
- [Versatile Learned Video Compression](https://mlsanity.com/paper/2111.03386/) (2021)
- [Flexible-Rate Learned Hierarchical Bi-Directional Video Compression With Motion Refinement and Frame-Level Bit Allocation](https://mlsanity.com/paper/2206.13613/) (2022)

## Abstract (arXiv)

While learned video codecs have demonstrated great promise, they have yet to achieve sufficient efficiency for practical deployment. In this work, we propose several novel ideas for learned video compression which allow for improved performance for the low-latency mode (I- and P-frames only) along with a considerable increase in computational efficiency. In this setting, for natural videos our approach compares favorably across the entire R-D curve under metrics PSNR, MS-SSIM and VMAF against all mainstream video standards (H.264, H.265, AV1) and all ML codecs. At the same time, our approach runs at least 5x faster and has fewer parameters than all ML codecs which report these figures. Our contributions include a flexible-rate framework allowing a single model to cover a large and dense range of bitrates, at a negligible increase in computation and parameter count; an efficient backbone optimized for ML-based codecs; and a novel in-loop flow prediction scheme which leverages prior information towards more efficient compression. We benchmark our method, which we call ELF-VC (Efficient, Learned and Flexible Video Coding) on popular video test sets UVG and MCL-JCV under metrics PSNR, MS-SSIM and VMAF. For example, on UVG under PSNR, it reduces the BD-rate by 44% against H.264, 26% against H.265, 15% against AV1, and 35% against the current best ML codec. At the same time, on an NVIDIA Titan V GPU our approach encodes/decodes VGA at 49/91 FPS, HD 720 at 19/35 FPS, and HD 1080 at 10/18 FPS.