# Learning-Based Video Coding with Joint Deep Compression and Enhancement

arXiv: https://arxiv.org/abs/2111.14474 · HTML: https://mlsanity.com/paper/2111.14474/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 31)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation](https://mlsanity.com/paper/2103.00860/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks](https://mlsanity.com/paper/1703.10114/) (2017)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video](https://mlsanity.com/paper/1902.09707/) (2019)
- [Space-Time-Aware Multi-Resolution Video Enhancement](https://mlsanity.com/paper/2003.13170/) (2020)
- [Multi-Frame Quality Enhancement for Compressed Video](https://mlsanity.com/paper/1803.04680/) (2018)
- [Slimmable Compressive Autoencoders for Practical Neural Image Compression](https://mlsanity.com/paper/2103.15726/) (2021)
- [Towards Unsupervised Deep Image Enhancement with Generative Adversarial Network](https://mlsanity.com/paper/2012.15020/) (2020)
- [URIE: Universal Image Enhancement for Visual Recognition in the Wild](https://mlsanity.com/paper/2007.08979/) (2020)
- [ProxIQA: A Proxy Approach to Perceptual Optimization of Learned Image Compression](https://mlsanity.com/paper/1910.08845/) (2019)

## Built on by (most cited first, 4 of 4)

- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Frequency Regularization: Restricting Information Redundancy of Convolutional Neural Networks](https://mlsanity.com/paper/2304.07973/) (2023)
- [GAN-Based Multi-View Video Coding with Spatio-Temporal EPI Reconstruction](https://mlsanity.com/paper/2205.03599/) (2022)

## Abstract (arXiv)

The end-to-end learning-based video compression has attracted substantial attentions by paving another way to compress video signals as stacked visual features. This paper proposes an efficient end-to-end deep video codec with jointly optimized compression and enhancement modules (JCEVC). First, we propose a dual-path generative adversarial network (DPEG) to reconstruct video details after compression. An α-path facilitates the structure information reconstruction with a large receptive field and multi-frame references, while a β-path facilitates the reconstruction of local textures. Both paths are fused and co-trained within a generative-adversarial process. Second, we reuse the DPEG network in both motion compensation and quality enhancement modules, which are further combined with other necessary modules to formulate our JCEVC framework. Third, we employ a joint training of deep video compression and enhancement that further improves the rate-distortion (RD) performance of compression. Compared with x265 LDP very fast mode, our JCEVC reduces the average bit-per-pixel (bpp) by 39.39%/54.92% at the same PSNR/MS-SSIM, which outperforms the state-of-the-art deep video codecs by a considerable margin.