# SuperTran: Reference Based Video Transformer for Enhancing Low Bitrate Streams in Real Time

arXiv: https://arxiv.org/abs/2211.12604 · HTML: https://mlsanity.com/paper/2211.12604/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation](https://mlsanity.com/paper/1611.05250/) (2016)
- [Image Super-Resolution by Neural Texture Transfer](https://mlsanity.com/paper/1903.00834/) (2019)
- [MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement](https://mlsanity.com/paper/1810.08768/) (2018)
- [CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping](https://mlsanity.com/paper/1807.10547/) (2018)
- [A Deep Journey into Super-resolution: A survey](https://mlsanity.com/paper/1904.07523/) (2019)
- [Multi-Frame Quality Enhancement for Compressed Video](https://mlsanity.com/paper/1803.04680/) (2018)
- [Generative Adversarial Networks and Perceptual Losses for Video Super-Resolution](https://mlsanity.com/paper/1806.05764/) (2018)
- [Perceptual Video Super Resolution with Enhanced Temporal Consistency](https://mlsanity.com/paper/1807.07930/) (2018)

## Abstract (arXiv)

This work focuses on low bitrate video streaming scenarios (e.g. 50 - 200Kbps) where the video quality is severely compromised. We present a family of novel deep generative models for enhancing perceptual video quality of such streams by performing super-resolution while also removing compression artifacts. Our model, which we call SuperTran, consumes as input a single high-quality, high-resolution reference images in addition to the low-quality, low-resolution video stream. The model thus learns how to borrow or copy visual elements like textures from the reference image and fill in the remaining details from the low resolution stream in order to produce perceptually enhanced output video. The reference frame can be sent once at the start of the video session or be retrieved from a gallery. Importantly, the resulting output has substantially better detail than what has been otherwise possible with methods that only use a low resolution input such as the SuperVEGAN method. SuperTran works in real-time (up to 30 frames/sec) on the cloud alongside standard pipelines.