# Audio-Visual Driven Compression for Low-Bitrate Talking Head Videos

arXiv: https://arxiv.org/abs/2506.13419 · HTML: https://mlsanity.com/paper/2506.13419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [Ultra-low bitrate video conferencing using deep image animation](https://mlsanity.com/paper/2012.00346/) (2020)
- [Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation](https://mlsanity.com/paper/2405.04327/) (2024)
- [Predictive Coding For Animation-Based Video Compression](https://mlsanity.com/paper/2307.04187/) (2023)
- [A Hybrid Deep Animation Codec for Low-bitrate Video Conferencing](https://mlsanity.com/paper/2207.13530/) (2022)
- [Bidirectional Learned Facial Animation Codec for Low Bitrate Talking Head Videos](https://mlsanity.com/paper/2503.09787/) (2025)

## Abstract (arXiv)

Talking head video compression has advanced with neural rendering and keypoint-based methods, but challenges remain, especially at low bit rates, including handling large head movements, suboptimal lip synchronization, and distorted facial reconstructions. To address these problems, we propose a novel audio-visual driven video codec that integrates compact 3D motion features and audio signals. This approach robustly models significant head rotations and aligns lip movements with speech, improving both compression efficiency and reconstruction quality. Experiments on the CelebV-HQ dataset show that our method reduces bitrate by 22% compared to VVC and by 8.5% over state-of-the-art learning-based codec. Furthermore, it provides superior lip-sync accuracy and visual fidelity at comparable bitrates, highlighting its effectiveness in bandwidth-constrained scenarios.