# Advancing Learned Video Compression with In-loop Frame Prediction

arXiv: https://arxiv.org/abs/2211.07004 · HTML: https://mlsanity.com/paper/2211.07004/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 76 references that are in the map.
- First posted on arXiv 13 Nov 2022.

## Builds on (most cited first, 30 of 49)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Quadratic video interpolation](https://mlsanity.com/paper/1911.00627/) (2019)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)

## Built on by (most cited first, 14 of 14)

- [Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression](https://mlsanity.com/paper/2401.15864/) (2024)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [MMVP: Motion-Matrix-based Video Prediction](https://mlsanity.com/paper/2308.16154/) (2023)
- [IBVC: Interpolation-driven B-frame Video Compression](https://mlsanity.com/paper/2309.13835/) (2023)
- [Uniformly Accelerated Motion Model for Inter Prediction](https://mlsanity.com/paper/2407.11541/) (2024)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [Neural Stereo Video Compression with Hybrid Disparity Compensation](https://mlsanity.com/paper/2504.20383/) (2025)
- [Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding](https://mlsanity.com/paper/2506.07709/) (2025)
- [Higher fidelity perceptual image and video compression with a latent conditioned residual denoising diffusion model](https://mlsanity.com/paper/2505.13152/) (2025)
- [Wireless Video Semantic Communication with Decoupled Diffusion Multi-frame Compensation](https://mlsanity.com/paper/2511.02478/) (2025)
- [Accelerating Learned Video Compression via Low-Resolution Representation Learning](https://mlsanity.com/paper/2407.16418/) (2024)
- [Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment](https://mlsanity.com/paper/2512.10450/) (2025)

## Abstract (arXiv)

Recent years have witnessed an increasing interest in end-to-end learned video compression. Most previous works explore temporal redundancy by detecting and compressing a motion map to warp the reference frame towards the target frame. Yet, it failed to adequately take advantage of the historical priors in the sequential reference frames. In this paper, we propose an Advanced Learned Video Compression (ALVC) approach with the in-loop frame prediction module, which is able to effectively predict the target frame from the previously compressed frames, without consuming any bit-rate. The predicted frame can serve as a better reference than the previously compressed frame, and therefore it benefits the compression performance. The proposed in-loop prediction module is a part of the end-to-end video compression and is jointly optimized in the whole framework. We propose the recurrent and the bi-directional in-loop prediction modules for compressing P-frames and B-frames, respectively. The experiments show the state-of-the-art performance of our ALVC approach in learned video compression. We also outperform the default hierarchical B mode of x265 in terms of PSNR and beat the slowest mode of the SSIM-tuned x265 on MS-SSIM. The project page: https://github.com/RenYang-home/ALVC.