# Neural B-frame Video Compression with Bi-directional Reference Harmonization

arXiv: https://arxiv.org/abs/2511.08938 · HTML: https://mlsanity.com/paper/2511.08938/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 55 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Builds on (most cited first, 30 of 37)

- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s](https://mlsanity.com/paper/2409.08481/) (2024)
- [End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression](https://mlsanity.com/paper/2112.09529/) (2021)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [Object Segmentation-Assisted Inter Prediction for Versatile Video Coding](https://mlsanity.com/paper/2403.11694/) (2024)
- [Bi-Directional Deep Contextual Video Compression](https://mlsanity.com/paper/2408.08604/) (2024)
- [ECVC: Exploiting Non-Local Correlations in Multiple Frames for Contextual Video Compression](https://mlsanity.com/paper/2410.09706/) (2024)
- [B-CANF: Adaptive B-frame Coding with Conditional Augmented Normalizing Flows](https://mlsanity.com/paper/2209.01769/) (2022)
- [Offline and Online Optical Flow Enhancement for Deep Video Compression](https://mlsanity.com/paper/2307.05092/) (2023)
- [Neural Video Compression with Context Modulation](https://mlsanity.com/paper/2505.14541/) (2025)
- [Advances In Video Compression System Using Deep Neural Network: A Review And Case Studies](https://mlsanity.com/paper/2101.06341/) (2021)

## Built on by (most cited first, 5 of 5)

- [DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression](https://mlsanity.com/paper/2601.20564/) (2026)
- [Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compression](https://mlsanity.com/paper/2603.05756/) (2026)
- [Generative Video Compression Based on Hierarchical Referencing](https://mlsanity.com/paper/2608.11618/) (2026)
- [DiffVC-ONE: Diffusion-based Generative Video Compression with One-Step Video Diffusion Transformer](https://mlsanity.com/paper/2608.20515/) (2026)
- [Neural video codecs quality assessment dataset and benchmark](https://mlsanity.com/paper/2608.29331/) (2026)

## Abstract (arXiv)

Neural video compression (NVC) has made significant progress in recent years, while neural B-frame video compression (NBVC) remains underexplored compared to P-frame compression. NBVC can adopt bi-directional reference frames for better compression performance. However, NBVC's hierarchical coding may complicate continuous temporal prediction, especially at some hierarchical levels with a large frame span, which could cause the contribution of the two reference frames to be unbalanced. To optimize reference information utilization, we propose a novel NBVC method, termed Bi-directional Reference Harmonization Video Compression (BRHVC), with the proposed Bi-directional Motion Converge (BMC) and Bi-directional Contextual Fusion (BCF). BMC converges multiple optical flows in motion compression, leading to more accurate motion compensation on a larger scale. Then BCF explicitly models the weights of reference contexts under the guidance of motion compensation accuracy. With more efficient motions and contexts, BRHVC can effectively harmonize bi-directional references. Experimental results indicate that our BRHVC outperforms previous state-of-the-art NVC methods, even surpassing the traditional coding, VTM-RA (under random access configuration), on the HEVC datasets. The source code is released at https://github.com/kwai/NVC.