# LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation

arXiv: https://arxiv.org/abs/2512.23576 · HTML: https://mlsanity.com/paper/2512.23576/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 36)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [Rolling Forcing: Autoregressive Long Video Diffusion in Real Time](https://mlsanity.com/paper/2509.25161/) (2025)
- [Towards Robust Blind Face Restoration with Codebook Lookup Transformer](https://mlsanity.com/paper/2206.11253/) (2022)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation](https://mlsanity.com/paper/2303.04248/) (2023)
- [Elucidating the Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2308.15321/) (2023)
- [BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping](https://mlsanity.com/paper/2306.05544/) (2023)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation](https://mlsanity.com/paper/2505.04512/) (2025)
- [OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation](https://mlsanity.com/paper/2506.18866/) (2025)
- [FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis](https://mlsanity.com/paper/2504.04842/) (2025)

## Built on by (most cited first, 9 of 9)

- [Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models](https://mlsanity.com/paper/2606.25041/) (2026)
- [RealCam: Real-Time Novel-View Video Generation with Interactive Camera Control](https://mlsanity.com/paper/2605.06051/) (2026)
- [CollectionLoRA: Collecting 50 Effects in 1 LoRA via Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2605.25378/) (2026)
- [On-Policy Adversarial Flow Distillation for Autoregressive Video Generation](https://mlsanity.com/paper/2605.26105/) (2026)
- [UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars](https://mlsanity.com/paper/2605.14731/) (2026)
- [Where and When to Force: Routed Forcing for Streaming Avatars](https://mlsanity.com/paper/2609.30963/) (2026)
- [EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation](https://mlsanity.com/paper/2602.13669/) (2026)
- [ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation](https://mlsanity.com/paper/2603.17427/) (2026)
- [LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation](https://mlsanity.com/paper/2608.00079/) (2026)

## Abstract (arXiv)

Real-time video generation via diffusion is essential for building general-purpose multimodal interactive AI systems. However, the simultaneous denoising of all video frames with bidirectional attention via an iterative process in diffusion models prevents real-time interaction. While existing distillation methods can make the model autoregressive and reduce sampling steps to mitigate this, they focus primarily on text-to-video generation, leaving the human-AI interaction unnatural and less efficient. This paper targets real-time interactive video diffusion conditioned on a multimodal context, including text, image, and audio, to bridge the gap. Given the observation that the leading on-policy distillation approach Self Forcing encounters challenges (visual artifacts like flickering, black frames, and quality degradation) with multimodal conditioning, we investigate an improved distillation recipe with emphasis on the quality of condition inputs as well as the initialization and schedule for the on-policy optimization. On benchmarks for multimodal-conditioned (audio, image, and text) avatar video generation including HDTF, AVSpeech, and CelebV-HQ, our distilled model matches the visual quality of the full-step, bidirectional baselines of similar or larger size with 20x less inference cost and latency. Further, we integrate our model with audio language models and long-form video inference technique Anchor-Heavy Identity Sinks to build LiveTalk, a real-time multimodal interactive avatar system. System-level evaluation on our curated multi-turn interaction benchmark shows LiveTalk outperforms state-of-the-art models (Sora2, Veo3) in multi-turn video coherence and content quality, while reducing response latency from 1 to 2 minutes to real-time generation, enabling seamless human-AI multimodal interaction.