mlsanityData to 7 Oct 2026

MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations

arXiv 2606.27779 ↗First posted 26 Jun 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Generating lifelike facial animation for dyadic conversations requires reconciling high-level cognitive intent with precise low-level motor reflexes, yet existing methods fall short in the semantic understanding of dialogue context and in precise dynamic control. In this paper, we propose MindFlow, a dual-pathway generative framework inspired by the Ventral-Dorsal pathway model in neuroscience, which decouples generation into two collaborative streams, thereby harmonizing deep semantic reasoning with fine-grained control. In the Ventral module, we transform the conventional Sentence-Action approach into a novel Chunk-State approach that models raw acoustic streams as a context-aware, evolving emotional state chain, capturing subtle paralinguistic nuances and mid-utterance emotional shifts missed by sentence-level modeling. The Dorsal module features a conditional autoregressive flow matching network for high-fidelity facial motion, driven by high-frequency acoustic cues and modulated by emotion states, plus a Selective Acoustic Injector for adaptive audio gating to ensure robustness in talking-and-listening dynamics without interference. Extensive experiments demonstrate that MindFlow achieves superior semantic appropriateness and motion naturalness compared to state-of-the-art baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

41 of 54 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  6. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  7. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  8. 2025Qwen3-Omni Technical Reportcited by 451
  9. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  10. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  11. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  12. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
29 more
  1. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
  2. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  3. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  4. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  5. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  6. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  7. 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
  8. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
  9. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  10. 2025Ready-to-React: Online Reaction Policy for Two-Character Interaction Generationcited by 25
  11. 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
  12. 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
  13. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  14. 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
  15. 2025Fun-Audio-Chat Technical Reportcited by 17
  16. 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
  17. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  18. 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
  19. 2023Interactive Conversational Head Generationcited by 14
  20. 2024RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Networkcited by 14
  21. 2025Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Controlcited by 12
  22. 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusioncited by 10
  23. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  24. 2024You Think, You ACT: The New Task of Arbitrary Text to Motion Generationcited by 8
  25. 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
  26. 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
  27. 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 4
  28. 2025UniTalker: Conversational Speech-Visual Synthesiscited by 4
  29. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Modelcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026STEER: Steerable Dyadic Head Avatars
  2. 2026GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization
  3. 2025Think-Before-Draw: Decomposing Emotion Semantics & Fine-Grained Controllable Expressive Talking Head Generation
  4. 2026InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance
  5. 2026Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction
  6. 2026EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation
  7. 2026SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching
  8. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body