MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations
Not on the map. Why?
Abstract (arXiv)
Generating lifelike facial animation for dyadic conversations requires reconciling high-level cognitive intent with precise low-level motor reflexes, yet existing methods fall short in the semantic understanding of dialogue context and in precise dynamic control. In this paper, we propose MindFlow, a dual-pathway generative framework inspired by the Ventral-Dorsal pathway model in neuroscience, which decouples generation into two collaborative streams, thereby harmonizing deep semantic reasoning with fine-grained control. In the Ventral module, we transform the conventional Sentence-Action approach into a novel Chunk-State approach that models raw acoustic streams as a context-aware, evolving emotional state chain, capturing subtle paralinguistic nuances and mid-utterance emotional shifts missed by sentence-level modeling. The Dorsal module features a conditional autoregressive flow matching network for high-fidelity facial motion, driven by high-frequency acoustic cues and modulated by emotion states, plus a Selective Acoustic Injector for adaptive audio gating to ensure robustness in talking-and-listening dynamics without interference. Extensive experiments demonstrate that MindFlow achieves superior semantic appropriateness and motion naturalness compared to state-of-the-art baselines.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 26 Jun 2026.
Builds on
41 of 54 references.
References · 41
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
- 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
- 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
- 2025Qwen3-Omni Technical Reportcited by 451
- 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
- 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
- 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
- 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
29 more
- 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversationscited by 82
- 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
- 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
- 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
- 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
- 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
- 2025DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversationscited by 31
- 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 29
- 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
- 2025Ready-to-React: Online Reaction Policy for Two-Character Interaction Generationcited by 25
- 2025StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatarscited by 24
- 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
- 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
- 2025MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generationcited by 18
- 2025Fun-Audio-Chat Technical Reportcited by 17
- 2026Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversationcited by 17
- 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
- 2025X-Streamer: Unified Human World Modeling with Audiovisual Interactioncited by 16
- 2023Interactive Conversational Head Generationcited by 14
- 2024RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Networkcited by 14
- 2025Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Controlcited by 12
- 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusioncited by 10
- 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
- 2024You Think, You ACT: The New Task of Arbitrary Text to Motion Generationcited by 8
- 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
- 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
- 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 4
- 2025UniTalker: Conversational Speech-Visual Synthesiscited by 4
- 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Modelcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026STEER: Steerable Dyadic Head Avatars
- 2026GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization
- 2025Think-Before-Draw: Decomposing Emotion Semantics & Fine-Grained Controllable Expressive Talking Head Generation
- 2026InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance
- 2026Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction
- 2026EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation
- 2026SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching
- 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body