mlsanityData to 14 Oct 2026

OpusLM: A Family of Open Unified Speech Language Models

arXiv 2506.17611 ↗First posted 21 Jun 2025cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 21 Jun 2025.

Builds on

43 of 53 references.

References · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  12. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
31 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2024Qwen2-Audio Technical Reportcited by 560
  5. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  6. 2023Simple and Controllable Music Generationcited by 488
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  8. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  10. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  12. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  13. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
  14. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  15. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  16. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  17. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  18. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
  19. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  20. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  21. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  22. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  23. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  24. 2024A Survey on Speech Large Language Models for Understandingcited by 71
  25. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  26. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 60
  27. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
  28. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  29. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
  30. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  31. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  2. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  3. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  4. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  5. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  6. 2026Bagpiper-TTS: Natural Language Guided Universal Speech Synthesiscited by 1
  7. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0
  8. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generationcited by 0
  9. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0