# Deep Multi-modality Soft-decoding of Very Low Bit-rate Face Videos

arXiv: https://arxiv.org/abs/2008.01652 · HTML: https://mlsanity.com/paper/2008.01652/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2020.

## Built on by (most cited first, 10 of 10)

- [Multi-modality Deep Restoration of Extremely Compressed Face Videos](https://mlsanity.com/paper/2107.05548/) (2021)
- [Learning Degradation-Independent Representations for Camera ISP Pipelines](https://mlsanity.com/paper/2307.00761/) (2023)
- [Functional Neural Networks for Parametric Image Restoration Problems](https://mlsanity.com/paper/2111.00361/) (2021)
- [Perceptual Quality Improvement in Videoconferencing using Keyframes-based GAN](https://mlsanity.com/paper/2311.04263/) (2023)
- [Fast Point Cloud Geometry Compression with Context-based Residual Coding and INR-based Refinement](https://mlsanity.com/paper/2408.02966/) (2024)
- [Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion](https://mlsanity.com/paper/2204.12756/) (2022)
- [Dual-layer Image Compression via Adaptive Downsampling and Spatially Varying Upconversion](https://mlsanity.com/paper/2302.06096/) (2023)
- [Self-Supervised Face Image Restoration with a One-Shot Reference](https://mlsanity.com/paper/2203.03005/) (2022)
- [Revealing Microscopic Objects in Fluorescence Live Imaging by Video-to-video Translation Based on A Spatial-temporal Generative Adversarial Network](https://mlsanity.com/paper/2502.16342/) (2025)
- [Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning](https://mlsanity.com/paper/2508.04161/) (2025)

## Abstract (arXiv)

We propose a novel deep multi-modality neural network for restoring very low bit rate videos of talking heads. Such video contents are very common in social media, teleconferencing, distance education, tele-medicine, etc., and often need to be transmitted with limited bandwidth. The proposed CNN method exploits the correlations among three modalities, video, audio and emotion state of the speaker, to remove the video compression artifacts caused by spatial down sampling and quantization. The deep learning approach turns out to be ideally suited for the video restoration task, as the complex non-linear cross-modality correlations are very difficult to model analytically and explicitly. The new method is a video post processor that can significantly boost the perceptual quality of aggressively compressed talking head videos, while being fully compatible with all existing video compression standards.