mlsanityData to 14 Oct 2026

Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation

arXiv 2406.07867 ↗First posted 12 Jun 2024cs.CV cs.AI cs.HCcited by 41

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/IVLLab/MultiDialog, respectively.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 49 of its 81 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

49 of 81 references.

References · 49

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  12. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
37 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  2. 2018CoQA: A Conversational Question Answering Challengecited by 683
  3. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  4. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  5. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  10. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  11. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  12. 2021Towards Real-World Blind Face Restoration with Generative Facial Priorcited by 349
  13. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  14. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  15. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  16. 2018Deep Audio-Visual Speech Recognitioncited by 322
  17. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  18. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  19. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  20. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  21. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  22. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  23. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  24. 2018A Dataset for Document Grounded Conversationscited by 104
  25. 2023Textually Pretrained Speech Language Modelscited by 101
  26. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  27. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 89
  28. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  29. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  30. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  31. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  32. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 69
  33. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  34. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  35. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  36. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  37. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 18.

41 papers.

Sort:

By area:

  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  3. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  4. 2025ARIG: Autoregressive Interactive Head Generation for Real-time Conversationscited by 16
  5. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 15
  6. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  7. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  8. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  9. 2025MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Modelscited by 9
  10. 2025DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Datasetcited by 8
  11. 2025Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Routercited by 6
  12. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
29 more
  1. 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 5
  2. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  3. 2025DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Modelcited by 5
  4. 2025Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialoguescited by 4
  5. 2024AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cuescited by 3
  6. 2025Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactionscited by 3
  7. 2024J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modelingcited by 2
  8. 2024When Words Smile: Generating Diverse Emotional Facial Expressions from Textcited by 2
  9. 2026Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systemscited by 2
  10. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  11. 2026Integrating Facial Generation into Full-Duplex Spoken Dialogue Systemscited by 2
  12. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 1
  13. 2026Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interactioncited by 1
  14. 2026ReactMotion: Generating Reactive Listener Motions from Speaker Utterancecited by 1
  15. 2026DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolutioncited by 1
  16. 2026InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidancecited by 1
  17. 2026When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platformscited by 1
  18. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  19. 2026AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesiscited by 1
  20. 2026StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speechcited by 1
  21. 2025Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditionscited by 0
  22. 2025Towards Human-like Multimodal Conversational Agent by Generating Engaging Speechcited by 0
  23. 2025Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignmentcited by 0
  24. 2026Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoningcited by 0
  25. 2026Titans-as-a-Layer: Test-Time Memory for Conversational Speech Emotion Recognitioncited by 0
  26. 2026MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversationscited by 0
  27. 2026Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesiscited by 0
  28. 2026EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialoguecited by 0
  29. 2026COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoningcited by 0