mlsanityData to 14 Oct 2026

ESPnet-SpeechLM: An Open Speech Language Model Toolkit

arXiv 2502.15218 ↗First posted 21 Feb 2025cs.CL cs.SD eess.AScited by 9

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks. The toolkit and its recipes are fully transparent and reproducible at: https://github.com/espnet/espnet/tree/speechlm.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 48 of its 65 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

48 of 65 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
36 more
  1. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  10. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  13. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  14. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  15. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  16. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  17. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  18. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  19. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  20. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  21. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  22. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  23. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  24. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  25. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  26. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 60
  27. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  28. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  29. 2024Scaling Properties of Speech Language Modelscited by 32
  30. 2023Amphion: An Open-Source Audio, Music and Speech Generation Toolkitcited by 27
  31. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 26
  32. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  33. 2024UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learnercited by 22
  34. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  35. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  36. 2024Muskits-ESPnet: A Comprehensive Toolkit for Singing Voice Synthesis in New Paradigmcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  3. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  4. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  5. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 5
  6. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  7. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  8. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0
  9. 2026LACE: Layer-Wise Compression for Dynamic Frame Rate Codecscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
  2. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
  3. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  4. 2026Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech ↗
  5. 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing
  6. 2026KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
  7. 2023Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings
  8. 2026Raon-Speech Technical Report