# MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations

arXiv: https://arxiv.org/abs/2606.27779 · HTML: https://mlsanity.com/paper/2606.27779/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2112.13548/) (2021)
- [LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision](https://mlsanity.com/paper/2412.09262/) (2024)
- [Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation](https://mlsanity.com/paper/2406.07867/) (2024)
- [INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations](https://mlsanity.com/paper/2412.04037/) (2024)
- [Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model](https://mlsanity.com/paper/2404.01862/) (2024)
- [DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations](https://mlsanity.com/paper/2505.18096/) (2025)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling](https://mlsanity.com/paper/2410.10122/) (2024)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars](https://mlsanity.com/paper/2512.22065/) (2025)
- [CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation](https://mlsanity.com/paper/2403.00274/) (2024)
- [AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement](https://mlsanity.com/paper/2511.23475/) (2025)
- [MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation](https://mlsanity.com/paper/2508.19320/) (2025)
- [Fun-Audio-Chat Technical Report](https://mlsanity.com/paper/2512.20156/) (2025)
- [Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation](https://mlsanity.com/paper/2601.00664/) (2026)
- [X-Streamer: Unified Human World Modeling with Audiovisual Interaction](https://mlsanity.com/paper/2509.21574/) (2025)
- [ARIG: Autoregressive Interactive Head Generation for Real-time Conversations](https://mlsanity.com/paper/2507.00472/) (2025)

## Abstract (arXiv)

Generating lifelike facial animation for dyadic conversations requires reconciling high-level cognitive intent with precise low-level motor reflexes, yet existing methods fall short in the semantic understanding of dialogue context and in precise dynamic control. In this paper, we propose MindFlow, a dual-pathway generative framework inspired by the Ventral-Dorsal pathway model in neuroscience, which decouples generation into two collaborative streams, thereby harmonizing deep semantic reasoning with fine-grained control. In the Ventral module, we transform the conventional Sentence-Action approach into a novel Chunk-State approach that models raw acoustic streams as a context-aware, evolving emotional state chain, capturing subtle paralinguistic nuances and mid-utterance emotional shifts missed by sentence-level modeling. The Dorsal module features a conditional autoregressive flow matching network for high-fidelity facial motion, driven by high-frequency acoustic cues and modulated by emotion states, plus a Selective Acoustic Injector for adaptive audio gating to ensure robustness in talking-and-listening dynamics without interference. Extensive experiments demonstrate that MindFlow achieves superior semantic appropriateness and motion naturalness compared to state-of-the-art baselines.