TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
Not on the map. Why?
Abstract (arXiv)
The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 63 of its 84 references that are in the map. First posted on arXiv 23 Dec 2025.
Builds on
63 of 84 references.
References · 63
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
38 more
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
- 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
- 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
- 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
- 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
- 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
- 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
- 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
- 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
- 2022Generative Spoken Dialogue Language Modelingcited by 125
- 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
- 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
- 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
- 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 95
- 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
- 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
- 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
- 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
- 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
- 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
- 2023Can Language Models Learn to Listen?cited by 40
- 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
- 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
- 2024Dyadic Interaction Modeling for Social Behavior Generationcited by 32
- 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
- 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
- 2023FT2TF: First-Person Statement Text-To-Talking Face Generationcited by 26
- 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
- 2024CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generationcited by 22
- 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Textcited by 21
- 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
- 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
- 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.