mlsanityData to 14 Oct 2026

SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation

arXiv 2501.00805 ↗First posted 1 Jan 2025eess.AS cs.CL cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 1 Jan 2025.

Builds on

19 of 36 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
7 more
  1. 2022Generative Spoken Dialogue Language Modelingcited by 126
  2. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  3. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  4. 2023TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorchcited by 27
  5. 2016Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machinecited by 23
  6. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  7. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  2. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  3. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  4. 2025The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approachcited by 1
  5. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  6. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0