# DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations

arXiv: https://arxiv.org/abs/2505.18096 · HTML: https://mlsanity.com/paper/2505.18096/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 23 May 2025.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Imitator: Personalized Speech-driven 3D Facial Animation](https://mlsanity.com/paper/2301.00023/) (2022)
- [One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field](https://mlsanity.com/paper/2304.05097/) (2023)
- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2112.13548/) (2021)
- [VisemeNet: Audio-Driven Animator-Centric Speech Animation](https://mlsanity.com/paper/1805.09488/) (2018)
- [NOFA: NeRF-based One-shot Facial Avatar Reconstruction](https://mlsanity.com/paper/2307.03441/) (2023)
- [Can Language Models Learn to Listen?](https://mlsanity.com/paper/2308.10897/) (2023)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors](https://mlsanity.com/paper/2211.15064/) (2022)
- [LaughTalk: Expressive 3D Talking Head Generation with Laughter](https://mlsanity.com/paper/2311.00994/) (2023)
- [ScanTalk: 3D Talking Heads from Unregistered Scans](https://mlsanity.com/paper/2403.10942/) (2024)
- [CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation](https://mlsanity.com/paper/2403.00274/) (2024)
- [DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer](https://mlsanity.com/paper/2402.05712/) (2024)
- [ReactFace: Online Multiple Appropriate Facial Reaction Generation in Dyadic Interactions](https://mlsanity.com/paper/2305.15748/) (2023)
- [MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Model](https://mlsanity.com/paper/2308.16635/) (2023)

## Built on by (most cited first, 30 of 32)

- [OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers](https://mlsanity.com/paper/2505.21448/) (2025)
- [UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking](https://mlsanity.com/paper/2512.09327/) (2025)
- [Towards Interactive Intelligence for Digital Humans](https://mlsanity.com/paper/2512.13674/) (2025)
- [SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splatting](https://mlsanity.com/paper/2506.14742/) (2025)
- [ShoulderShot: Generating Over-the-Shoulder Dialogue Videos](https://mlsanity.com/paper/2508.07597/) (2025)
- [ActAvatar: Temporally-Aware Precise Action Control for Talking Avatars](https://mlsanity.com/paper/2512.19546/) (2025)
- [GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation](https://mlsanity.com/paper/2506.21513/) (2025)
- [Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics](https://mlsanity.com/paper/2512.15340/) (2025)
- [OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation](https://mlsanity.com/paper/2604.18326/) (2026)
- [RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation](https://mlsanity.com/paper/2601.10606/) (2026)
- [Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models](https://mlsanity.com/paper/2602.07106/) (2026)
- [DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction](https://mlsanity.com/paper/2606.03874/) (2026)
- [MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancement](https://mlsanity.com/paper/2601.01749/) (2026)
- [EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents](https://mlsanity.com/paper/2604.17211/) (2026)
- [VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents](https://mlsanity.com/paper/2605.30256/) (2026)
- [Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation](https://mlsanity.com/paper/2606.31088/) (2026)
- [Warm Chat: Diffuse Emotion-aware Interactive Talking Head Avatar with Tree-Structured Guidance](https://mlsanity.com/paper/2508.18337/) (2025)
- [3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars](https://mlsanity.com/paper/2602.10516/) (2026)
- [Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction](https://mlsanity.com/paper/2603.07093/) (2026)
- [Talking Together: Synthesizing Co-Located 3D Conversations from Audio](https://mlsanity.com/paper/2603.08674/) (2026)
- [ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation](https://mlsanity.com/paper/2603.17427/) (2026)
- [GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization](https://mlsanity.com/paper/2603.25020/) (2026)
- [Generate Your Talking Avatar from Video Reference](https://mlsanity.com/paper/2604.27918/) (2026)
- [MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations](https://mlsanity.com/paper/2606.27779/) (2026)
- [FacePlex: Toward Natural Full-Duplex Conversational Avatars](https://mlsanity.com/paper/2606.30145/) (2026)
- [Conversational Human Audio-visual Talking Dialogue Generation](https://mlsanity.com/paper/2607.02799/) (2026)
- [STEER: Steerable Dyadic Head Avatars](https://mlsanity.com/paper/2607.23840/) (2026)
- [InteractGesture: Progressive Chunk Guidance for Continuous Streaming Co-Speech Gesture Control](https://mlsanity.com/paper/2608.25734/) (2026)
- [InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond](https://mlsanity.com/paper/2609.04903/) (2026)
- [Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation](https://mlsanity.com/paper/2609.34419/) (2026)

## Abstract (arXiv)

In face-to-face conversations, individuals need to switch between speaking and listening roles seamlessly. Existing 3D talking head generation models focus solely on speaking or listening, neglecting the natural dynamics of interactive conversation, which leads to unnatural interactions and awkward transitions. To address this issue, we propose a new task -- multi-round dual-speaker interaction for 3D talking head generation -- which requires models to handle and generate both speaking and listening behaviors in continuous conversation. To solve this task, we introduce DualTalk, a novel unified framework that integrates the dynamic behaviors of speakers and listeners to simulate realistic and coherent dialogue interactions. This framework not only synthesizes lifelike talking heads when speaking but also generates continuous and vivid non-verbal feedback when listening, effectively capturing the interplay between the roles. We also create a new dataset featuring 50 hours of multi-round conversations with over 1,000 characters, where participants continuously switch between speaking and listening roles. Extensive experiments demonstrate that our method significantly enhances the naturalness and expressiveness of 3D talking heads in dual-speaker conversations. We recommend watching the supplementary video: https://ziqiaopeng.github.io/dualtalk.