mlsanityData to 7 Oct 2026

Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs

arXiv 2606.01620 ↗First posted 1 Jun 2026cs.CVcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Video diffusion models have significantly advanced portrait video generation, yet their high computational demands limit their use in interactive applications. This work presents a framework for streamable talking portrait video generation conditioned on speech audio and reference images. Designed meticulously for streaming scenarios, it features a causal video VAE for deep latent compression and an autoregressive latent denoising model. Our causal VAE integrates a variable number of reference images as guidance, allowing the network to focus on dynamic information rather than static appearance, thereby enhancing compression efficacy and reconstruction quality. Additionally, we extend the residual auto-encoding paradigm to improve spatial-temporal causality handling in our VAE. The generator is based on a Rectified Flow Transformer architecture and produces video latents in a blockwise auto-regressive manner. Our method enables the real-time generation of high-quality talking portrait videos, achieving speeds significantly faster than baseline models. Furthermore, comprehensive experiments demonstrate that it is on par with or even outperforms these large models in realism, vividness, and video quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 65 of its 75 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

65 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025EchoShot: Multi-Shot Portrait Video Generation16 Jun
  2. 2025TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models3 Jun
  3. 2025ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model27 Mar
  4. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer1 Dec
  5. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation10 Oct
  6. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  7. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan
  8. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors7 Dec

Other references · 57

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
38 more
  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  4. 2016Pixel Recurrent Neural Networkscited by 814
  5. 2023Adversarial Diffusion Distillationcited by 741
  6. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  10. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  11. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  12. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  13. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  14. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  15. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  16. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  17. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  18. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 276
  19. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  20. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  21. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  22. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  23. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  24. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  25. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  26. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  27. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  28. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  29. 2018Lip Movements Generation at a Glancecited by 154
  30. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  31. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  32. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 142
  33. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  34. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  35. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  36. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  37. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  38. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
  2. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
  3. 2024Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation
  4. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  5. 2026AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising
  6. 2025SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers
  7. 2025LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models
  8. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation