DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Audio-driven talking-head generation has advanced rapidly with diffusion-based generative models, yet producing temporally coherent videos with fine-grained motion control remains challenging. We propose DEMO, a flow-matching generative framework for audio-driven talking-portrait video synthesis that delivers disentangled, high-fidelity control of lip motion, head pose, and eye gaze. The core contribution is a motion auto-encoder that builds a structured latent space in which motion factors are independently represented and approximately orthogonalized. On this disentangled motion space, we apply optimal-transport-based flow matching with a transformer predictor to generate temporally smooth motion trajectories conditioned on audio. Extensive experiments across multiple benchmarks show that DEMO outperforms prior methods in video realism, lip-audio synchronization, and motion fidelity. These results demonstrate that combining fine-grained motion disentanglement with flow-based generative modeling provides a powerful new paradigm for controllable talking-head video synthesis.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 12 Oct 2025.
Builds on
22 of 28 references.
Closest ancestors · 2
Same direction, newest first
- 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait2 Dec
- 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
Other references · 20
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
- 2020First Order Motion Model for Image Animationcited by 704
- 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
- 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
- 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
- 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
8 more
- 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
- 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
- 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
- 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
- 2020Neural Head Reenactment with Latent Pose Descriptorscited by 106
- 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 80
- 2018Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learningcited by 28
- 2020LandmarkGAN: Synthesizing Faces from Landmarkscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
- 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis
- 2025ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search
- 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
- 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
- 2024PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation
- 2025IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfer
- 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation