# Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression

arXiv: https://arxiv.org/abs/2401.15864 · HTML: https://mlsanity.com/paper/2401.15864/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 77 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)

## Built on by (most cited first, 23 of 23)

- [USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s](https://mlsanity.com/paper/2409.08481/) (2024)
- [Generative Latent Coding for Ultra-Low Bitrate Image and Video Compression](https://mlsanity.com/paper/2505.16177/) (2025)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [Neural Video Compression with Context Modulation](https://mlsanity.com/paper/2505.14541/) (2025)
- [Augmented Deep Contexts for Spatially Embedded Video Coding](https://mlsanity.com/paper/2505.05309/) (2025)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)
- [Real-Time Neural Video Compression with Unified Intra and Inter Coding](https://mlsanity.com/paper/2510.14431/) (2025)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [Neural Stereo Video Compression with Hybrid Disparity Compensation](https://mlsanity.com/paper/2504.20383/) (2025)
- [Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding](https://mlsanity.com/paper/2506.07709/) (2025)
- [SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization](https://mlsanity.com/paper/2510.21847/) (2025)
- [L-STEC: Learned Video Compression with Long-term Spatio-Temporal Enhanced Context](https://mlsanity.com/paper/2512.12790/) (2025)
- [Recent Advances of End-to-End Video Coding Technologies for AVS Standard Development](https://mlsanity.com/paper/2602.00483/) (2026)
- [A Noise Constrained Diffusion (NC-Diffusion) Framework for High Fidelity Image Compression](https://mlsanity.com/paper/2604.06568/) (2026)
- [Neural Video Compression with Domain Transfer](https://mlsanity.com/paper/2605.13476/) (2026)
- [Multi-Scale Feature Prediction with Auxiliary-Info for Neural Image Compression](https://mlsanity.com/paper/2409.12719/) (2024)
- [Embedding Compression Distortion in Video Coding for Machines](https://mlsanity.com/paper/2503.21469/) (2025)
- [DCVC-MV: Deep Contextual Multiview Video Compression with Efficient Inter-View Prediction](https://mlsanity.com/paper/2509.03922/) (2025)
- [Neural B-Frame Coding: Tackling Domain Shift Issues with Lightweight Online Motion Resolution Adaptation](https://mlsanity.com/paper/2511.18724/) (2025)
- [KD-NVC: A Search-and-Distill Framework to Accelerate Neural Video Coding](https://mlsanity.com/paper/2606.04595/) (2026)
- [Boosting Neural Video Codec via Scale-Driven Online Flow Refinement](https://mlsanity.com/paper/2606.23023/) (2026)
- [Neural Video Compression Based on Deformable Temporal Alignment and Difference-aware Fusion](https://mlsanity.com/paper/2609.03520/) (2026)
- [Video Codec-Guided Training-Free Token Compression for Video Large Language Models](https://mlsanity.com/paper/2610.12959/) (2026)

## Abstract (arXiv)

Video compression performance is closely related to the accuracy of inter prediction. It tends to be difficult to obtain accurate inter prediction for the local video regions with inconsistent motion and occlusion. Traditional video coding standards propose various technologies to handle motion inconsistency and occlusion, such as recursive partitions, geometric partitions, and long-term references. However, existing learned video compression schemes focus on obtaining an overall minimized prediction error averaged over all regions while ignoring the motion inconsistency and occlusion in local regions. In this paper, we propose a spatial decomposition and temporal fusion based inter prediction for learned video compression. To handle motion inconsistency, we propose to decompose the video into structure and detail (SDD) components first. Then we perform SDD-based motion estimation and SDD-based temporal context mining for the structure and detail components to generate short-term temporal contexts. To handle occlusion, we propose to propagate long-term temporal contexts by recurrently accumulating the temporal information of each historical reference feature and fuse them with short-term temporal contexts. With the SDD-based motion model and long short-term temporal contexts fusion, our proposed learned video codec can obtain more accurate inter prediction. Comprehensive experimental results demonstrate that our codec outperforms the reference software of H.266/VVC on all common test datasets for both PSNR and MS-SSIM.