# MMVC: Learned Multi-Mode Video Compression with Block-based Prediction Mode Selection and Density-Adaptive Entropy Coding

arXiv: https://arxiv.org/abs/2304.02273 · HTML: https://mlsanity.com/paper/2304.02273/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 5 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1812.11677/) (2018)
- [Hierarchical Autoregressive Modeling for Neural Video Compression](https://mlsanity.com/paper/2010.10258/) (2020)
- [ModeNet: Mode Selection Network For Learned Video Coding](https://mlsanity.com/paper/2007.02532/) (2020)
- [Unified Signal Compression Using Generative Adversarial Networks](https://mlsanity.com/paper/1912.03734/) (2019)

## Built on by (most cited first, 9 of 9)

- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [IBVC: Interpolation-driven B-frame Video Compression](https://mlsanity.com/paper/2309.13835/) (2023)
- [VCISR: Blind Single Image Super-Resolution with Video Compression Synthetic Data](https://mlsanity.com/paper/2311.00996/) (2023)
- [Ultra-Fast Neural Video Compression](https://mlsanity.com/paper/2606.04410/) (2026)
- [Neural Stereo Video Compression with Hybrid Disparity Compensation](https://mlsanity.com/paper/2504.20383/) (2025)
- [Accelerating Learned Video Compression via Low-Resolution Representation Learning](https://mlsanity.com/paper/2407.16418/) (2024)
- [Neural Video Compression with In-Loop Contextual Filtering and Out-of-Loop Reconstruction Enhancement](https://mlsanity.com/paper/2509.04051/) (2025)
- [Context Video Semantic Transmission with Variable Length and Rate Coding over MIMO Channels](https://mlsanity.com/paper/2601.06059/) (2025)

## Abstract (arXiv)

Learning-based video compression has been extensively studied over the past years, but it still has limitations in adapting to various motion patterns and entropy models. In this paper, we propose multi-mode video compression (MMVC), a block wise mode ensemble deep video compression framework that selects the optimal mode for feature domain prediction adapting to different motion patterns. Proposed multi-modes include ConvLSTM-based feature domain prediction, optical flow conditioned feature domain prediction, and feature propagation to address a wide range of cases from static scenes without apparent motions to dynamic scenes with a moving camera. We partition the feature space into blocks for temporal prediction in spatial block-based representations. For entropy coding, we consider both dense and sparse post-quantization residual blocks, and apply optional run-length coding to sparse residuals to improve the compression rate. In this sense, our method uses a dual-mode entropy coding scheme guided by a binary density map, which offers significant rate reduction surpassing the extra cost of transmitting the binary selection map. We validate our scheme with some of the most popular benchmarking datasets. Compared with state-of-the-art video compression schemes and standard codecs, our method yields better or competitive results measured with PSNR and MS-SSIM.