mlsanityData to 7 Oct 2026

PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems

arXiv 2406.12428 ↗First posted 18 Jun 2024cs.CL cs.AI cs.LGcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal language models that process both text and speech have a potential for applications in spoken dialogue systems. However, current models face two major challenges in response generation latency: (1) generating a spoken response requires the prior generation of a written response, and (2) speech sequences are significantly longer than text sequences. This study addresses these issues by extending the input and output sequences of the language model to support the parallel generation of text and speech. Our experiments on spoken question answering tasks demonstrate that our approach improves latency while maintaining the quality of response content. Additionally, we show that latency can be further reduced by generating speech in multiple sequences. Demo samples are available at https://rinnakk.github.io/research/publications/PSLM.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 34 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

21 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  7. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  8. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  9. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  10. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  11. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  12. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
8 more
  1. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  2. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  3. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 36
  4. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  5. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  6. 2024Release of Pre-Trained Models for the Japanese Languagecited by 22
  7. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  8. 2022Streaming non-autoregressive model for any-to-many voice conversioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  4. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16
  5. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  6. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  7. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 8
  8. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interactioncited by 7
  9. 2025OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactionscited by 4
  10. 2025MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidancecited by 3
  11. 2026Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMscited by 3
  12. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
1 more
  1. 2026Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation
  2. 2024Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
  3. 2023Speak While You Think: Streaming Speech Synthesis During Text Generation
  4. 2022Long-term Control for Dialogue Generation: Methods and Evaluation
  5. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
  6. 2025From What to Respond to When to Respond: Timely Response Generation for Open-domain Dialogue Agents
  7. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
  8. 2025Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech