mlsanityData to 14 Oct 2026

Generative Spoken Dialogue Language Modeling

arXiv 2203.16502 ↗First posted 30 Mar 2022cs.CL cs.LG cs.SDcited by 126

Not on the map. Why?

Abstract (arXiv)

We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 35 of its 69 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

35 of 69 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
23 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2020Recipes for building an open-domain chatbotcited by 593
  3. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  6. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  9. 2021Internet-Augmented Dialogue Generationcited by 217
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  11. 2015A Neural Conversational Modelcited by 196
  12. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
  13. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  14. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  15. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  16. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  17. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  18. 2020TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialogcited by 56
  19. 2021Variable-rate discrete representation learningcited by 24
  20. 2018Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMscited by 16
  21. 2021Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systemscited by 15
  22. 2016Generative Deep Neural Networks for Dialogue: A Short Reviewcited by 13
  23. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 21; 2024: 23; 2025: 37; 2026: 40.

126 papers.

Sort:

By area:

  1. 2022High Fidelity Neural Audio Compressioncited by 687
  2. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  4. 2022AudioGen: Textually Guided Audio Generationcited by 295
  5. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  6. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  7. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  8. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  9. 2023Textually Pretrained Speech Language Modelscited by 101
  10. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  11. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  12. 2024Recent Advances in Speech Language Models: A Surveycited by 87
38 more
  1. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  2. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  3. 2024Language Model Can Listen While Speakingcited by 63
  4. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  5. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  6. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 47
  7. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 45
  8. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  9. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  10. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  11. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  12. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  13. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  14. 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 23
  15. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  16. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  17. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  18. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 19
  19. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  20. 2022Augmentation Invariant Discrete Representation for Generative Spoken Language Modelingcited by 17
  21. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  22. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  23. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 15
  24. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  25. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  26. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  27. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 14
  28. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  29. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  30. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  31. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  32. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  33. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  34. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  35. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  36. 2025From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Modelscited by 10
  37. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  38. 2024LAST: Language Model Aware Speech Tokenizationcited by 9

50 most cited of 126 All 126 (JSON)