# InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing

arXiv: https://arxiv.org/abs/2508.14033 · HTML: https://mlsanity.com/paper/2508.14033/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 30 of 42)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](https://mlsanity.com/paper/2305.10474/) (2023)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation](https://mlsanity.com/paper/2406.08801/) (2024)
- [EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions](https://mlsanity.com/paper/2407.08136/) (2024)
- [AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation](https://mlsanity.com/paper/2403.17694/) (2024)
- [Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance](https://mlsanity.com/paper/2306.00943/) (2023)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer](https://mlsanity.com/paper/2412.00733/) (2024)
- [Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency](https://mlsanity.com/paper/2409.02634/) (2024)
- [OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation](https://mlsanity.com/paper/2506.18866/) (2025)

## Built on by (most cited first, 30 of 41)

- [AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement](https://mlsanity.com/paper/2511.23475/) (2025)
- [OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators](https://mlsanity.com/paper/2607.08766/) (2026)
- [Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation](https://mlsanity.com/paper/2512.21734/) (2025)
- [SoulX-FlashTalk: Real-Time Infinite Streaming of Audio-Driven Avatars via Self-Correcting Bidirectional Distillation](https://mlsanity.com/paper/2512.23379/) (2025)
- [JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion](https://mlsanity.com/paper/2601.22143/) (2026)
- [LongCat-Video-Avatar 1.5 Technical Report](https://mlsanity.com/paper/2605.26486/) (2026)
- [Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms](https://mlsanity.com/paper/2603.28489/) (2026)
- [MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation](https://mlsanity.com/paper/2512.03034/) (2025)
- [SoulX-FlashHead: Oracle-guided Generation of Infinite Real-time Streaming Talking Heads](https://mlsanity.com/paper/2602.07449/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [JoyStreamer-Flash: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion](https://mlsanity.com/paper/2512.11423/) (2025)
- [Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation](https://mlsanity.com/paper/2512.13495/) (2025)
- [Active Intelligence in Video Avatars via Closed-loop World Modeling](https://mlsanity.com/paper/2512.20615/) (2025)
- [Gloria: Consistent Character Video Generation via Content Anchors](https://mlsanity.com/paper/2603.29931/) (2026)
- [OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation](https://mlsanity.com/paper/2604.18326/) (2026)
- [CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation](https://mlsanity.com/paper/2604.19636/) (2026)
- [Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length](https://mlsanity.com/paper/2512.04677/) (2025)
- [SyncAnyone: Implicit Disentanglement via Progressive Self-Correction for Lip-Syncing in the wild](https://mlsanity.com/paper/2512.21736/) (2025)
- [EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers](https://mlsanity.com/paper/2601.22127/) (2026)
- [SoulX-LiveAct: Towards Hour-Scale Real-Time Human Animation with Neighbor Forcing and ConvKV Memory](https://mlsanity.com/paper/2603.11746/) (2026)
- [Diff-VF: Training-free High-quality Long Video Generation via Diffusion Model](https://mlsanity.com/paper/2608.05976/) (2026)
- [JoyStreamer: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning](https://mlsanity.com/paper/2602.00702/) (2026)
- [A2-LLM: An End-to-end Conversational Audio Avatar Large Language Model](https://mlsanity.com/paper/2602.04913/) (2026)
- [FlexAM: Flexible Appearance-Motion Decomposition for Versatile Video Generation Control](https://mlsanity.com/paper/2602.13185/) (2026)
- [InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance](https://mlsanity.com/paper/2603.23132/) (2026)
- [TurboTalk: Progressive Distillation for One-Step Audio-Driven Talking Avatar Generation](https://mlsanity.com/paper/2604.14580/) (2026)
- [DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation](https://mlsanity.com/paper/2606.26058/) (2026)
- [AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars](https://mlsanity.com/paper/2607.24013/) (2026)
- [Avatar-Forever: Decoupled Parallel Training for High-Quality Real-Time Infinite Avatars](https://mlsanity.com/paper/2608.12107/) (2026)
- [YingVideo-MV: Music-Driven Multi-Stage Video Generation](https://mlsanity.com/paper/2512.02492/) (2025)

## Abstract (arXiv)

Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.