# SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation

arXiv: https://arxiv.org/abs/2501.00805 · HTML: https://mlsanity.com/paper/2501.00805/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 1 Jan 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study](https://mlsanity.com/paper/2309.15800/) (2023)
- [TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch](https://mlsanity.com/paper/2310.17864/) (2023)
- [Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine](https://mlsanity.com/paper/1608.06378/) (2016)
- [Towards human-like spoken dialogue generation between AI agents from written dialogue](https://mlsanity.com/paper/2310.01088/) (2023)
- [CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations](https://mlsanity.com/paper/2404.06690/) (2024)

## Built on by (most cited first, 6 of 6)

- [MoonCast: High-Quality Zero-Shot Podcast Generation](https://mlsanity.com/paper/2503.14345/) (2025)
- [CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching](https://mlsanity.com/paper/2506.00885/) (2025)
- [ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching](https://mlsanity.com/paper/2507.09318/) (2025)
- [The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach](https://mlsanity.com/paper/2510.09424/) (2025)
- [TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation](https://mlsanity.com/paper/2512.20296/) (2025)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)

## Abstract (arXiv)

Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.