# MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation

arXiv: https://arxiv.org/abs/2508.19320 · HTML: https://mlsanity.com/paper/2508.19320/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 41 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Builds on (most cited first, 30 of 39)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [MAGI-1: Autoregressive Video Generation at Scale](https://mlsanity.com/paper/2505.13211/) (2025)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions](https://mlsanity.com/paper/2407.08136/) (2024)
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://mlsanity.com/paper/2404.10667/) (2024)
- [Autoregressive Video Generation without Vector Quantization](https://mlsanity.com/paper/2412.14169/) (2024)
- [MegaPortraits: One-shot Megapixel Neural Head Avatars](https://mlsanity.com/paper/2207.07621/) (2022)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [Long-Context Autoregressive Video Modeling with Next-Frame Prediction](https://mlsanity.com/paper/2503.19325/) (2025)
- [Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer](https://mlsanity.com/paper/2412.00733/) (2024)
- [Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency](https://mlsanity.com/paper/2409.02634/) (2024)
- [AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding](https://mlsanity.com/paper/2405.03121/) (2024)
- [Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation](https://mlsanity.com/paper/2505.22647/) (2025)

## Built on by (most cited first, 19 of 19)

- [StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars](https://mlsanity.com/paper/2512.22065/) (2025)
- [Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis](https://mlsanity.com/paper/2509.09595/) (2025)
- [AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement](https://mlsanity.com/paper/2511.23475/) (2025)
- [X-Streamer: Unified Human World Modeling with Audiovisual Interaction](https://mlsanity.com/paper/2509.21574/) (2025)
- [KlingAvatar 2.0 Technical Report](https://mlsanity.com/paper/2512.13313/) (2025)
- [GLaD: Geometric Latent Distillation for Vision-Language-Action Models](https://mlsanity.com/paper/2512.09619/) (2025)
- [Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation](https://mlsanity.com/paper/2512.21734/) (2025)
- [FlowAct-R1: Towards Interactive Humanoid Video Generation](https://mlsanity.com/paper/2601.10103/) (2026)
- [ReWorld: An Interactive World Model with Long-Horizon Memory](https://mlsanity.com/paper/2608.23565/) (2026)
- [AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner](https://mlsanity.com/paper/2512.10571/) (2025)
- [Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models](https://mlsanity.com/paper/2606.25041/) (2026)
- [SemanticGen: Video Generation in Semantic Space](https://mlsanity.com/paper/2512.20619/) (2025)
- [AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising](https://mlsanity.com/paper/2603.14331/) (2026)
- [SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation](https://mlsanity.com/paper/2605.10079/) (2026)
- [Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation](https://mlsanity.com/paper/2512.13495/) (2025)
- [Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length](https://mlsanity.com/paper/2512.04677/) (2025)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)
- [MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations](https://mlsanity.com/paper/2606.27779/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)

## Abstract (arXiv)

Recently, interactive digital human video generation has attracted widespread attention and achieved remarkable progress. However, building such a practical system that can interact with diverse input signals in real time remains challenging to existing methods, which often struggle with heavy computational cost and limited controllability. In this work, we introduce an autoregressive video generation framework that enables interactive multimodal control and low-latency extrapolation in a streaming manner. With minimal modifications to a standard large language model (LLM), our framework accepts multimodal condition encodings including audio, pose, and text, and outputs spatially and semantically coherent representations to guide the denoising process of a diffusion head. To support this, we construct a large-scale dialogue dataset of approximately 20,000 hours from multiple sources, providing rich conversational scenarios for training. We further introduce a deep compression autoencoder with up to 64× reduction ratio, which effectively alleviates the long-horizon inference burden of the autoregressive model. Extensive experiments on duplex conversation, multilingual human synthesis, and interactive world model highlight the advantages of our approach in low latency, high efficiency, and fine-grained multimodal controllability.