mlsanityData to 14 Oct 2026

From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM

arXiv 2503.10620 ↗First posted 13 Mar 2025cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 55 of its 74 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

55 of 74 references.

References · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2023BloombergGPT: A Large Language Model for Financecited by 939
  10. 2020Aligning AI With Shared Human Valuescited by 723
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  13. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  14. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  15. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  16. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  17. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  18. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  19. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  20. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  21. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  22. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  23. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  24. 2019Episodic Memory in Lifelong Language Learningcited by 188
  25. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  26. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  27. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  28. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  29. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  30. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  31. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  32. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  33. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  34. 2023LLaSM: Large Language and Speech Modelcited by 58
  35. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  36. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  37. 2023Toward Joint Language Modeling for Speech Units and Textcited by 28
  38. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025NAVER LABS Europe Submission to the Instruction-following Trackcited by 5
  2. 2026SpeechMapper: Speech-to-text Embedding Projector for LLMscited by 4
  3. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 2
  4. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translationcited by 1
  5. 2026Why We Need Speech to Evaluate Speech Translationcited by 1
  6. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learningcited by 0
  7. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
  8. 2026Two-Point Local Optimality in k-Means via Boundary-Point Screeningcited by 0