# Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text

arXiv: https://arxiv.org/abs/2106.14014 · HTML: https://mlsanity.com/paper/2106.14014/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars](https://mlsanity.com/paper/2008.10174/) (2020)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [Semantic Communication Systems for Speech Transmission](https://mlsanity.com/paper/2102.12605/) (2021)
- [Towards Automatic Face-to-Face Translation](https://mlsanity.com/paper/2003.00418/) (2020)
- [A Lite Distributed Semantic Communication System for Internet of Things](https://mlsanity.com/paper/2007.11095/) (2020)
- [VisemeNet: Audio-Driven Animator-Centric Speech Animation](https://mlsanity.com/paper/1805.09488/) (2018)
- [X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers](https://mlsanity.com/paper/2009.11278/) (2020)
- [Speech Driven Talking Face Generation from a Single Image and an Emotion Condition](https://mlsanity.com/paper/2008.03592/) (2020)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)
- [TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator](https://mlsanity.com/paper/2009.02018/) (2020)
- [3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head](https://mlsanity.com/paper/2104.12051/) (2021)
- [The Deepfake Detection Dilemma: A Multistakeholder Exploration of Adversarial Dynamics in Synthetic Media](https://mlsanity.com/paper/2102.06109/) (2021)
- [Reducing latency and bandwidth for video streaming using keypoint extraction and digital puppetry](https://mlsanity.com/paper/2011.03800/) (2020)

## Built on by (most cited first, 9 of 9)

- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [Joint Source-Channel Coding: Fundamentals and Recent Progress in Practical Designs](https://mlsanity.com/paper/2409.17557/) (2024)
- [Audio-Visual Face Reenactment](https://mlsanity.com/paper/2210.02755/) (2022)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Optimization of Image Transmission in a Cooperative Semantic Communication Networks](https://mlsanity.com/paper/2301.00433/) (2023)
- [Synchronous Multi-modal Semantic Communication System with Packet-level Coding](https://mlsanity.com/paper/2408.04535/) (2024)
- [THInImg: Cross-modal Steganography for Presenting Talking Heads in Images](https://mlsanity.com/paper/2311.17177/) (2023)
- [Large Generative Model-assisted Talking-face Semantic Communication System](https://mlsanity.com/paper/2411.03876/) (2024)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)

## Abstract (arXiv)

Video represents the majority of internet traffic today, driving a continual race between the generation of higher quality content, transmission of larger file sizes, and the development of network infrastructure. In addition, the recent COVID-19 pandemic fueled a surge in the use of video conferencing tools. Since videos take up considerable bandwidth (~100 Kbps to a few Mbps), improved video compression can have a substantial impact on network performance for live and pre-recorded content, providing broader access to multimedia content worldwide. We present a novel video compression pipeline, called Txt2Vid, which dramatically reduces data transmission rates by compressing webcam videos ("talking-head videos") to a text transcript. The text is transmitted and decoded into a realistic reconstruction of the original video using recent advances in deep learning based voice cloning and lip syncing models. Our generative pipeline achieves two to three orders of magnitude reduction in the bitrate as compared to the standard audio-video codecs (encoders-decoders), while maintaining equivalent Quality-of-Experience based on a subjective evaluation by users (n = 242) in an online study. The Txt2Vid framework opens up the potential for creating novel applications such as enabling audio-video communication during poor internet connectivity, or in remote terrains with limited bandwidth. The code for this work is available at https://github.com/tpulkit/txt2vid.git.