# Temporal Context Mining for Learned Video Compression

arXiv: https://arxiv.org/abs/2111.13850 · HTML: https://mlsanity.com/paper/2111.13850/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 27 Nov 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [BVI-DVC: A Training Database for Deep Video Compression](https://mlsanity.com/paper/2003.13552/) (2020)
- [Conditional Entropy Coding for Efficient Video Compression](https://mlsanity.com/paper/2008.09180/) (2020)
- [Perceptual Learned Video Compression with Recurrent Conditional GAN](https://mlsanity.com/paper/2109.03082/) (2021)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [Learned Video Compression via Joint Spatial-Temporal Correlation Exploration](https://mlsanity.com/paper/1912.06348/) (2019)
- [Hierarchical Autoregressive Modeling for Neural Video Compression](https://mlsanity.com/paper/2010.10258/) (2020)
- [Learning Image and Video Compression through Spatial-Temporal Energy Compaction](https://mlsanity.com/paper/1906.09683/) (2019)
- [Learning for Video Compression](https://mlsanity.com/paper/1804.09869/) (2018)

## Built on by (most cited first, 30 of 117)

- [HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression](https://mlsanity.com/paper/2403.14530/) (2024)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Improving the Diffusability of Autoencoders](https://mlsanity.com/paper/2502.14831/) (2025)
- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation](https://mlsanity.com/paper/2306.09818/) (2023)
- [C3: High-performance and low-complexity neural compression from a single image or video](https://mlsanity.com/paper/2312.02753/) (2023)
- [DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos](https://mlsanity.com/paper/2304.06544/) (2023)
- [USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s](https://mlsanity.com/paper/2409.08481/) (2024)
- [How Far Can We Compress Instant-NGP-Based NeRF?](https://mlsanity.com/paper/2406.04101/) (2024)
- [Boosting Neural Representations for Videos with a Conditional Decoder](https://mlsanity.com/paper/2402.18152/) (2024)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video](https://mlsanity.com/paper/2503.18421/) (2025)
- [Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression](https://mlsanity.com/paper/2401.15864/) (2024)
- [Linear Attention Modeling for Learned Image Compression](https://mlsanity.com/paper/2502.05741/) (2025)
- [Generative Latent Coding for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2512.20194/) (2025)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [B-CANF: Adaptive B-frame Coding with Conditional Augmented Normalizing Flows](https://mlsanity.com/paper/2209.01769/) (2022)
- [Offline and Online Optical Flow Enhancement for Deep Video Compression](https://mlsanity.com/paper/2307.05092/) (2023)
- [Neural Video Compression with Context Modulation](https://mlsanity.com/paper/2505.14541/) (2025)
- [Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression](https://mlsanity.com/paper/2112.13309/) (2021)
- [VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision](https://mlsanity.com/paper/2306.10681/) (2023)
- [Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance](https://mlsanity.com/paper/2512.07480/) (2025)
- [Neighbor Correspondence Matching for Flow-based Video Frame Synthesis](https://mlsanity.com/paper/2207.06763/) (2022)
- [IBVC: Interpolation-driven B-frame Video Compression](https://mlsanity.com/paper/2309.13835/) (2023)
- [Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression](https://mlsanity.com/paper/2405.04274/) (2024)
- [Conditional Latent Coding with Learnable Synthesized Reference for Deep Image Compression](https://mlsanity.com/paper/2502.09971/) (2025)
- [DiffVC-OSD: One-Step Diffusion-based Perceptual Neural Video Compression Framework](https://mlsanity.com/paper/2508.07682/) (2025)
- [Bit Allocation using Optimization](https://mlsanity.com/paper/2209.09422/) (2022)

## Abstract (arXiv)

We address end-to-end learned video compression with a special focus on better learning and utilizing temporal contexts. For temporal context mining, we propose to store not only the previously reconstructed frames, but also the propagated features into the generalized decoded picture buffer. From the stored propagated features, we propose to learn multi-scale temporal contexts, and re-fill the learned temporal contexts into the modules of our compression scheme, including the contextual encoder-decoder, the frame generator, and the temporal context encoder. Our scheme discards the parallelization-unfriendly auto-regressive entropy model to pursue a more practical decoding time. We compare our scheme with x264 and x265 (representing industrial software for H.264 and H.265, respectively) as well as the official reference software for H.264, H.265, and H.266 (JM, HM, and VTM, respectively). When intra period is 32 and oriented to PSNR, our scheme outperforms H.265--HM by 14.4% bit rate saving; when oriented to MS-SSIM, our scheme outperforms H.266--VTM by 21.1% bit rate saving.