mlsanityData to 7 Oct 2026

FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

arXiv 2412.01064 ↗First posted 2 Dec 2024cs.CV cs.AI cs.LGcited by 40

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 84 of its 105 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

84 of 105 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency4 Sep
  2. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  3. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models30 Sep
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan

Other references · 80

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2016Layer Normalizationcited by 5,579
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2018Neural Ordinary Differential Equationscited by 3,733
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  13. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  14. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  15. 2015Variational Inference with Normalizing Flowscited by 2,161
  16. 2016Density estimation using Real NVPcited by 2,029
  17. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  18. 2023Consistency Modelscited by 1,778
  19. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  20. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  21. 2022Human Motion Diffusion Modelcited by 1,062
  22. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  23. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  24. 2020First Order Motion Model for Image Animationcited by 698
  25. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  26. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  27. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  28. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  29. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  30. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  31. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  32. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  33. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  34. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  35. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 347
  36. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  37. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  38. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 24.

40 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
  3. 2025Celeb-DF++: A Large-scale Challenging Video DeepFake Benchmark for Generalizable Forensicscited by 18
  4. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  5. 2025Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentationscited by 11
  6. 2025FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generationcited by 10
  7. 2025OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimickingcited by 9
  8. 2025Controllable and Expressive One-Shot Video Head Swappingcited by 2
  9. 2025Learning Robot Manipulation from Audio World Modelscited by 2
  10. 2025DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representationscited by 2
  11. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modelingcited by 2
  12. 2026CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planningcited by 2
28 more
  1. 2025Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Fieldcited by 1
  2. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 1
  3. 2025IMTalker: Efficient Audio-driven Talking Face Generation with Implicit Motion Transfercited by 1
  4. 2025A Sanity Check for Multi-In-Domain Face Forgery Detection in the Real Worldcited by 1
  5. 2026IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animationcited by 1
  6. 2026Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Videocited by 1
  7. 2025DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesiscited by 0
  8. 2025STARCaster: Spatio-Temporal AutoRegressive Video Diffusion for Identity- and View-Aware Talking Portraitscited by 0
  9. 2025FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANscited by 0
  10. 2026Rethinking Refinement: Correcting Generative Bias without Noise Injectioncited by 0
  11. 2026Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalizationcited by 0
  12. 2026Native Extrapolation Awareness in Flow-Based Conditional Generationcited by 0
  13. 2026EmbedTalk: Talking Head Synthesis using Gaussian Embeddingscited by 0
  14. 2026ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generationcited by 0
  15. 2026GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detectioncited by 0
  16. 2026PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactmentcited by 0
  17. 2026MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generationcited by 0
  18. 2026Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generationcited by 0
  19. 2026Temporally-Aligned Evaluation for Audio-Driven Talking Head Generationcited by 0
  20. 2026Soul Computing: A Theoretical Framework and Technical Architecture for Intelligent Agents with Independent Consciousnesscited by 0
  21. 2026Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detectioncited by 0
  22. 2026Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animationcited by 0
  23. 2026Can Tainted Pixels Expose Deepfake Videos?cited by 0
  24. 2026Decoupled Self-Forcing Distillation for Streaming Talking Head Generationcited by 0
  25. 2026VibeAvatar: Aligning Phonetic Kinematics and Human Aesthetics for High-Fidelity Talking Avatar Synthesiscited by 0
  26. 2026Complementary rPPG-Derived and Lip-Region Frequency Cues for Talking-Face Deepfake Detectioncited by 0
  27. 2026AVTR-1: Open Stack for Real-Time Interactive Avatarscited by 0
  28. 2026GLARE: Generating Listening Heads with Appropriate Reactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
  2. 2024GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
  3. 2025X-Actor: Emotional and Expressive Long-Range Portrait Acting from Audio
  4. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
  5. 2026Real-Time Generation of Streamable Talking Portrait Video with Reference-Guided Deep Compression VAEs
  6. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation
  7. 2024Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
  8. 2026FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation