# Compressing Video Calls using Synthetic Talking Heads

arXiv: https://arxiv.org/abs/2210.03692 · HTML: https://mlsanity.com/paper/2210.03692/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [GAN Prior Embedded Network for Blind Face Restoration in the Wild](https://mlsanity.com/paper/2105.06070/) (2021)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [You said that?](https://mlsanity.com/paper/1705.02966/) (2017)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [ReenactGAN: Learning to Reenact Faces via Boundary Transfer](https://mlsanity.com/paper/1807.11079/) (2018)
- [Low Bandwidth Video-Chat Compression using Deep Generative Models](https://mlsanity.com/paper/2012.00328/) (2020)
- [Audio-Visual Face Reenactment](https://mlsanity.com/paper/2210.02755/) (2022)

## Built on by (most cited first, 10 of 10)

- [Generative Face Video Coding Techniques and Standardization Efforts: A Review](https://mlsanity.com/paper/2311.02649/) (2023)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [THInImg: Cross-modal Steganography for Presenting Talking Heads in Images](https://mlsanity.com/paper/2311.17177/) (2023)
- [OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions](https://mlsanity.com/paper/2309.16148/) (2023)
- [Pleno-Generation: A Scalable Generative Face Video Compression Framework with Bandwidth Intelligence](https://mlsanity.com/paper/2502.17085/) (2025)
- [Generative Models at the Frontier of Compression: A Survey on Generative Face Video Coding](https://mlsanity.com/paper/2506.07369/) (2025)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)
- [GaussianHeadTalk: Wobble-Free 3D Talking Heads with Audio Driven Gaussian Splatting](https://mlsanity.com/paper/2512.10939/) (2025)
- [Revising Second Order Terms in Deep Animation Video Coding](https://mlsanity.com/paper/2510.23561/) (2025)

## Abstract (arXiv)

We leverage the modern advancements in talking head generation to propose an end-to-end system for talking head video compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them. We use a state-of-the-art face reenactment network to detect key points in the non-pivot frames and transmit them to the receiver. A dense flow is then calculated to warp a pivot frame to reconstruct the non-pivot ones. Transmitting key points instead of full frames leads to significant compression. We propose a novel algorithm to adaptively select the best-suited pivot frames at regular intervals to provide a smooth experience. We also propose a frame-interpolater at the receiver's end to improve the compression levels further. Finally, a face enhancement network improves reconstruction quality, significantly improving several aspects like the sharpness of the generations. We evaluate our method both qualitatively and quantitatively on benchmark datasets and compare it with multiple compression techniques. We release a demo video and additional information at https://cvit.iiit.ac.in/research/projects/cvit-projects/talking-video-compression.