# Joint Reference Frame Synthesis and Post Filter Enhancement for Versatile Video Coding

arXiv: https://arxiv.org/abs/2404.18058 · HTML: https://mlsanity.com/paper/2404.18058/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 28 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation](https://mlsanity.com/paper/1907.01361/) (2019)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation](https://mlsanity.com/paper/2205.14620/) (2022)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [BVI-DVC: A Training Database for Deep Video Compression](https://mlsanity.com/paper/2003.13552/) (2020)
- [On the Generalization of BasicVSR++ to Video Deblurring and Denoising](https://mlsanity.com/paper/2204.05308/) (2022)
- [MFRNet: A New CNN Architecture for Post-Processing and In-loop Filtering](https://mlsanity.com/paper/2007.07099/) (2020)
- [Deep Learning-Based Video Coding: A Review and A Case Study](https://mlsanity.com/paper/1904.12462/) (2019)
- [Convolutional Neural Network-Based Block Up-sampling for Intra Frame Coding](https://mlsanity.com/paper/1702.06728/) (2017)
- [Optical Flow Reusing for High-Efficiency Space-Time Video Super Resolution](https://mlsanity.com/paper/2110.06786/) (2021)
- [Iterative training of neural networks for intra prediction](https://mlsanity.com/paper/2003.06812/) (2020)
- [Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis](https://mlsanity.com/paper/2105.05961/) (2021)
- [Designs and Implementations in Neural Network-based Video Coding](https://mlsanity.com/paper/2309.05846/) (2023)
- [Enhanced Intra Prediction for Video Coding by Using Multiple Neural Networks](https://mlsanity.com/paper/2005.02973/) (2020)

## Built on by (most cited first, 1 of 1)

- [Compressed Video Quality Enhancement: Classifying and Benchmarking over Standards](https://mlsanity.com/paper/2509.10407/) (2025)

## Abstract (arXiv)

This paper presents the joint reference frame synthesis (RFS) and post-processing filter enhancement (PFE) for Versatile Video Coding (VVC), aiming to explore the combination of different neural network-based video coding (NNVC) tools to better utilize the hierarchical bi-directional coding structure of VVC. Both RFS and PFE utilize the Space-Time Enhancement Network (STENet), which receives two input frames with artifacts and produces two enhanced frames with suppressed artifacts, along with an intermediate synthesized frame. STENet comprises two pipelines, the synthesis pipeline and the enhancement pipeline, tailored for different purposes. During RFS, two reconstructed frames are sent into STENet's synthesis pipeline to synthesize a virtual reference frame, similar to the current to-be-coded frame. The synthesized frame serves as an additional reference frame inserted into the reference picture list (RPL). During PFE, two reconstructed frames are fed into STENet's enhancement pipeline to alleviate their artifacts and distortions, resulting in enhanced frames with reduced artifacts and distortions. To reduce inference complexity, we propose joint inference of RFS and PFE (JISE), achieved through a single execution of STENet. Integrated into the VVC reference software VTM-15.0, RFS, PFE, and JISE are coordinated within a novel Space-Time Enhancement Window (STEW) under Random Access (RA) configuration. The proposed method could achieve -7.34%/-17.21%/-16.65% PSNR-based BD-rate on average for three components under RA configuration.