# End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression

arXiv: https://arxiv.org/abs/2112.09529 · HTML: https://mlsanity.com/paper/2112.09529/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [On the Computation of PSNR for a Set of Images or Video](https://mlsanity.com/paper/2104.14868/) (2021)
- [End-to-End Rate-Distortion Optimization for Bi-Directional Learned Video Compression](https://mlsanity.com/paper/2008.05028/) (2020)

## Built on by (most cited first, 28 of 28)

- [Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression](https://mlsanity.com/paper/2401.15864/) (2024)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [B-CANF: Adaptive B-frame Coding with Conditional Augmented Normalizing Flows](https://mlsanity.com/paper/2209.01769/) (2022)
- [Hierarchical B-frame Video Coding Using Two-Layer CANF without Motion Coding](https://mlsanity.com/paper/2304.02690/) (2023)
- [Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression](https://mlsanity.com/paper/2112.13309/) (2021)
- [IBVC: Interpolation-driven B-frame Video Compression](https://mlsanity.com/paper/2309.13835/) (2023)
- [Bit Allocation using Optimization](https://mlsanity.com/paper/2209.09422/) (2022)
- [Flexible-Rate Learned Hierarchical Bi-Directional Video Compression With Motion Refinement and Frame-Level Bit Allocation](https://mlsanity.com/paper/2206.13613/) (2022)
- [Scene Matters: Model-based Deep Video Compression](https://mlsanity.com/paper/2303.04557/) (2023)
- [UCVC: A Unified Contextual Video Compression Framework with Joint P-frame and B-frame Coding](https://mlsanity.com/paper/2402.01289/) (2024)
- [Multi-Scale Deformable Alignment and Content-Adaptive Inference for Flexible-Rate Bi-Directional Video Compression](https://mlsanity.com/paper/2306.16544/) (2023)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)
- [Neural B-frame Video Compression with Bi-directional Reference Harmonization](https://mlsanity.com/paper/2511.08938/) (2025)
- [Exploring Long- and Short-Range Temporal Information for Learned Video Compression](https://mlsanity.com/paper/2208.03754/) (2022)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [L-LBVC: Long-Term Motion Estimation and Prediction for Learned Bi-Directional Video Compression](https://mlsanity.com/paper/2504.02560/) (2025)
- [Reducing The Mismatch Between Marginal and Learned Distributions in Neural Video Compression](https://mlsanity.com/paper/2210.06596/) (2022)
- [Neural Video Compression with Temporal Layer-Adaptive Hierarchical B-frame Coding](https://mlsanity.com/paper/2308.15791/) (2023)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [Fast-OMRA: Fast Online Motion Resolution Adaptation for Neural B-Frame Coding](https://mlsanity.com/paper/2410.21763/) (2024)
- [Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding](https://mlsanity.com/paper/2506.07709/) (2025)
- [Learned Hierarchical B-frame Coding with Adaptive Feature Modulation for YUV 4:2:0 Content](https://mlsanity.com/paper/2212.14187/) (2022)
- [LSCD: A Large-Scale Screen Content Dataset for Video Compression](https://mlsanity.com/paper/2308.09332/) (2023)
- [Conditional Neural Video Coding with Spatial-Temporal Super-Resolution](https://mlsanity.com/paper/2401.13959/) (2024)
- [Neural-based Video Compression on Solar Dynamics Observatory Images](https://mlsanity.com/paper/2407.15730/) (2024)
- [Exploiting Latent Properties to Optimize Neural Codecs](https://mlsanity.com/paper/2501.01231/) (2025)
- [High Dynamic Range Video Compression: A Large-Scale Benchmark Dataset and A Learned Bit-depth Scalable Compression Algorithm](https://mlsanity.com/paper/2503.00410/) (2025)
- [Generative Video Compression Based on Hierarchical Referencing](https://mlsanity.com/paper/2608.11618/) (2026)

## Abstract (arXiv)

Conventional video compression (VC) methods are based on motion compensated transform coding, and the steps of motion estimation, mode and quantization parameter selection, and entropy coding are optimized individually due to the combinatorial nature of the end-to-end optimization problem. Learned VC allows end-to-end rate-distortion (R-D) optimized training of nonlinear transform, motion and entropy model simultaneously. Most works on learned VC consider end-to-end optimization of a sequential video codec based on R-D loss averaged over pairs of successive frames. It is well-known in conventional VC that hierarchical, bi-directional coding outperforms sequential compression because of its ability to use both past and future reference frames. This paper proposes a learned hierarchical bi-directional video codec (LHBDC) that combines the benefits of hierarchical motion-compensated prediction and end-to-end optimization. Experimental results show that we achieve the best R-D results that are reported for learned VC schemes to date in both PSNR and MS-SSIM. Compared to conventional video codecs, the R-D performance of our end-to-end optimized codec outperforms those of both x265 and SVT-HEVC encoders ("veryslow" preset) in PSNR and MS-SSIM as well as HM 16.23 reference software in MS-SSIM. We present ablation studies showing performance gains due to proposed novel tools such as learned masking, flow-field subsampling, and temporal flow vector prediction. The models and instructions to reproduce our results can be found in https://github.com/makinyilmaz/LHBDC/