# REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation

arXiv: https://arxiv.org/abs/2512.11229 · HTML: https://mlsanity.com/paper/2512.11229/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 36 of its 44 references that are in the map.
- First posted on arXiv 12 Dec 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 36)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Flow Matching Guide and Code](https://mlsanity.com/paper/2412.06264/) (2024)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation](https://mlsanity.com/paper/2406.08801/) (2024)
- [EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions](https://mlsanity.com/paper/2407.08136/) (2024)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [Flow Matching in Latent Space](https://mlsanity.com/paper/2307.08698/) (2023)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)
- [FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis](https://mlsanity.com/paper/2504.04842/) (2025)
- [Sonic: Shifting Focus to Global Audio Perception in Portrait Animation](https://mlsanity.com/paper/2411.16331/) (2024)
- [AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding](https://mlsanity.com/paper/2405.03121/) (2024)
- [What comprises a good talking-head video generation?: A Survey and Benchmark](https://mlsanity.com/paper/2005.03201/) (2020)
- [Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis](https://mlsanity.com/paper/2411.19509/) (2024)
- [ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model](https://mlsanity.com/paper/2502.20323/) (2025)

## Built on by (most cited first, 4 of 4)

- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation](https://mlsanity.com/paper/2602.13669/) (2026)
- [LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation](https://mlsanity.com/paper/2608.00079/) (2026)
- [Decoupled Self-Forcing Distillation for Streaming Talking Head Generation](https://mlsanity.com/paper/2609.10317/) (2026)

## Abstract (arXiv)

Diffusion models have significantly advanced the field of talking head generation (THG). However, slow inference speeds and prevalent non-autoregressive paradigms severely constrain the application of diffusion-based THG models. In this study, we propose REST, a pioneering diffusion-based, real-time, end-to-end streaming audio-driven talking head generation framework. To support real-time end-to-end generation, a compact video latent space is first learned through a spatiotemporal variational autoencoder with a high compression ratio. Additionally, to enable semi-autoregressive streaming within the compact video latent space, we introduce an ID-Context Cache mechanism, which integrates ID-Sink and Context-Cache principles into key-value caching for maintaining identity consistency and temporal coherence during long-term streaming generation. Furthermore, an Asynchronous Streaming Distillation (ASD) strategy is proposed to mitigate error accumulation and enhance temporal consistency in streaming generation, leveraging a non-streaming teacher with an asynchronous noise schedule to supervise the streaming student. REST bridges the gap between autoregressive and diffusion-based approaches, achieving a breakthrough in efficiency for applications requiring real-time THG. Experimental results demonstrate that REST outperforms state-of-the-art methods in both generation speed and overall performance.