Omni-LiveAvatar: Minute-Level Real-Time Streaming Joint Audio-Video Avatar Generation
Not on the map. Why?
Abstract (arXiv)
Joint audio-video generative models serve as foundation for immersive and interactive digital-human generation. Nevertheless, most existing models rely on bidirectional attention and multi-step denoising and can generate only short clips, making them unsuitable for real-time interaction over extended durations. We present Omni-LiveAvatar, the first framework for minute-level, real-time streaming joint audio-video avatar generation. Specifically, we propose (1) a progressive autoregressive distillation pipeline that transfers a large bidirectional joint audio-video diffusion model into a few-step autoregressive generator without auxiliary stabilization mechanisms; (2) a synchronized audio-video long-short-term memory that preserves global consistency under a bounded memory budget; and (3) a hierarchical rolling prompt planning strategy that enables coherent semantic evolution and seamless prompt transitions. Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time. In terms of speed, it achieves a 33× generation speedup over its teacher, LTX-2, on a single NVIDIA H200 GPU; in terms of generation quality, it outperforms accelerated baselines across visual quality, audio quality, cross-modal synchronization, and human fidelity. Our code is available at https://github.com/Aoko955/Omni-LiveAvatar.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
30 of 36 references.
References · 30
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
- 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
- 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
- 2025Improving Video Generation with Human Feedbackcited by 284
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
- 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
- 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
- 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
18 more
- 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
- 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
- 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 122
- 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
- 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
- 2025Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generationcited by 83
- 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
- 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
- 2025SenseFlow: Scaling Distribution Matching for Flow-based Text-to-Image Distillationcited by 32
- 2026Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Modelcited by 25
- 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
- 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
- 2026Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generationcited by 13
- 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
- 2025LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generationcited by 11
- 2025RemedyGS: Defend 3D Gaussian Splatting against Computation Cost Attackscited by 7
- 2026Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generationcited by 7
- 2026Real-Time Human Frontal View Synthesis from a Single Imagecited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
- 2025Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
- 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
- 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
- 2025TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models
- 2025LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models
- 2025SoulX-FlashTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation
- 2026EchoAvatar: Real-time Generative Avatar Animation from Audio Streams