mlsanityData to 7 Oct 2026

Zero-resource Speech Translation and Recognition with LLMs

arXiv 2412.18566 ↗First posted 24 Dec 2024cs.CL eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Despite recent advancements in speech processing, zero-resource speech translation (ST) and automatic speech recognition (ASR) remain challenging problems. In this work, we propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data. We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM. We perform several experiments both in ST and ASR to understand how to best train the model and what data has the most impact on performance in previously unseen languages. In ST, our best model is capable to achieve BLEU scores over 23 in CoVoST2 for two previously unseen languages, while in ASR, we achieve WERs of up to 28.2%. We finally show that the performance of our system is bounded by the ability of the LLM to output text in the desired language.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

25 of 32 references.

References · 25

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
13 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2023Listen, Think, and Understandcited by 206
  3. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  4. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  5. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  6. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  7. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  8. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  9. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
  10. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  11. 2021Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASRcited by 11
  12. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  13. 2024Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASRcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languagescited by 7
  2. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmarkcited by 6
  3. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  4. 2026Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundariescited by 2
  5. 2026MEUSLI: a Multilingual Projector for LLM-based ASR and Beyondcited by 1
  6. 2025LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Datacited by 0