# Real-Time Neural Video Recovery and Enhancement on Mobile Devices

arXiv: https://arxiv.org/abs/2307.12152 · HTML: https://mlsanity.com/paper/2307.12152/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 55 references that are in the map.
- First posted on arXiv 22 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Pixel Difference Networks for Efficient Edge Detection](https://mlsanity.com/paper/2108.07009/) (2021)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution](https://mlsanity.com/paper/2007.11803/) (2020)
- [Efficient Video Super-Resolution through Recurrent Latent Space Propagation](https://mlsanity.com/paper/1909.08080/) (2019)
- [BOLA: Near-Optimal Bitrate Adaptation for Online Videos](https://mlsanity.com/paper/1601.06748/) (2016)
- [Learning for Video Super-Resolution through HR Optical Flow Estimation](https://mlsanity.com/paper/1809.08573/) (2018)
- [Recurrent Flow-Guided Semantic Forecasting](https://mlsanity.com/paper/1809.08318/) (2018)
- [Online Video Super-Resolution with Convolutional Kernel Bypass Graft](https://mlsanity.com/paper/2208.02470/) (2022)

## Abstract (arXiv)

As mobile devices become increasingly popular for video streaming, it's crucial to optimize the streaming experience for these devices. Although deep learning-based video enhancement techniques are gaining attention, most of them cannot support real-time enhancement on mobile devices. Additionally, many of these techniques are focused solely on super-resolution and cannot handle partial or complete loss or corruption of video frames, which is common on the Internet and wireless networks. To overcome these challenges, we present a novel approach in this paper. Our approach consists of (i) a novel video frame recovery scheme, (ii) a new super-resolution algorithm, and (iii) a receiver enhancement-aware video bit rate adaptation algorithm. We have implemented our approach on an iPhone 12, and it can support 30 frames per second (FPS). We have evaluated our approach in various networks such as WiFi, 3G, 4G, and 5G networks. Our evaluation shows that our approach enables real-time enhancement and results in a significant increase in video QoE (Quality of Experience) of 24% - 82% in our video streaming system.