# GRACE: Loss-Resilient Real-Time Video through Neural Codecs

arXiv: https://arxiv.org/abs/2305.12333 · HTML: https://mlsanity.com/paper/2305.12333/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 120 references that are in the map.
- First posted on arXiv 21 May 2023.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [YouTube UGC Dataset for Video Compression Research](https://mlsanity.com/paper/1904.06457/) (2019)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting](https://mlsanity.com/paper/2109.02974/) (2021)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Sample Efficient Reinforcement Learning with REINFORCE](https://mlsanity.com/paper/2010.11364/) (2020)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Flow-Guided Transformer for Video Inpainting](https://mlsanity.com/paper/2208.06768/) (2022)
- [Machine Learning in the Air](https://mlsanity.com/paper/1904.12385/) (2019)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [Error Compensation Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2207.10391/) (2022)
- [Reinforcement Learning for Channel Coding: Learned Bit-Flipping Decoding](https://mlsanity.com/paper/1906.04448/) (2019)
- [Measuring the Performance and Network Utilization of Popular Video Conferencing Applications](https://mlsanity.com/paper/2105.13478/) (2021)
- [Gemino: Practical and Robust Neural Compression for Video Conferencing](https://mlsanity.com/paper/2209.10507/) (2022)
- [SENSEI: Aligning Video Streaming Quality with Dynamic User Sensitivity](https://mlsanity.com/paper/2008.04687/) (2020)
- [Do Not Deceive Your Employer with a Virtual Background: A Video Conferencing Manipulation-Detection System](https://mlsanity.com/paper/2106.15130/) (2021)
- [DeepRS: Deep-learning Based Network-Adaptive FEC for Real-Time Video Communications](https://mlsanity.com/paper/2001.07852/) (2020)
- [SQP: Congestion Control for Low-Latency Interactive Video Streaming](https://mlsanity.com/paper/2207.11857/) (2022)

## Built on by (most cited first, 9 of 9)

- [Reparo: Loss-Resilient Generative Codec for Video Conferencing](https://mlsanity.com/paper/2305.14135/) (2023)
- [Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AI](https://mlsanity.com/paper/2507.10510/) (2025)
- [Token Encoding for Semantic Recovery](https://mlsanity.com/paper/2604.12931/) (2026)
- [AquaVLM: Improving Underwater Situation Awareness with Mobile Vision Language Models](https://mlsanity.com/paper/2510.21722/) (2025)
- [Artic: AI-oriented Real-time Communication for MLLM Video Assistant](https://mlsanity.com/paper/2602.12641/) (2026)
- [Proteus: A Truncation-Robust Entropy Model for Progressive LiDAR Compression](https://mlsanity.com/paper/2608.00687/) (2026)
- [Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors](https://mlsanity.com/paper/2608.21837/) (2026)
- [TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment](https://mlsanity.com/paper/2608.30857/) (2026)
- [Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission](https://mlsanity.com/paper/2609.16279/) (2026)

## Abstract (arXiv)

In real-time video communication, retransmitting lost packets over high-latency networks is not viable due to strict latency requirements. To counter packet losses without retransmission, two primary strategies are employed -- encoder-based forward error correction (FEC) and decoder-based error concealment. The former encodes data with redundancy before transmission, yet determining the optimal redundancy level in advance proves challenging. The latter reconstructs video from partially received frames, but dividing a frame into independently coded partitions inherently compromises compression efficiency, and the lost information cannot be effectively recovered by the decoder without adapting the encoder. We present a loss-resilient real-time video system called GRACE, which preserves the user's quality of experience (QoE) across a wide range of packet losses through a new neural video codec. Central to GRACE's enhanced loss resilience is its joint training of the neural encoder and decoder under a spectrum of simulated packet losses. In lossless scenarios, GRACE achieves video quality on par with conventional codecs (e.g., H.265). As the loss rate escalates, GRACE exhibits a more graceful, less pronounced decline in quality, consistently outperforming other loss-resilient schemes. Through extensive evaluation on various videos and real network traces, we demonstrate that GRACE reduces undecodable frames by 95% and stall duration by 90% compared with FEC, while markedly boosting video quality over error concealment methods. In a user study with 240 crowdsourced participants and 960 subjective ratings, GRACE registers a 38% higher mean opinion score (MOS) than other baselines.