mlsanityData to 14 Oct 2026

Contrastive Learning for Task-Independent SpeechLLM-Pretraining

arXiv 2412.15712 ↗First posted 20 Dec 2024cs.CL cs.HCcited by 8

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

35 of 52 references.

References · 35

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  12. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
23 more
  1. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  2. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  3. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  4. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  5. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  6. 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
  7. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  8. 2020On Unbalanced Optimal Transport: An Analysis of Sinkhorn Algorithmcited by 61
  9. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  10. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  11. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  12. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  13. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  14. 2022End-to-end Spoken Conversational Question Answering: Task, Dataset and Modelcited by 37
  15. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  16. 2022Contrastive Audio-Language Learning for Musiccited by 33
  17. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  18. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  19. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  20. 2020Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translationcited by 12
  21. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  22. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  23. 2022CALM: Contrastive Aligned Audio-Language Multirate and Multimodal Representationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 5
  2. 2025KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025cited by 4
  3. 2025NUTSHELL: A Dataset for Abstract Generation from Scientific Talkscited by 3
  4. 2026Why We Need Speech to Evaluate Speech Translationcited by 1
  5. 2026Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026cited by 1
  6. 2026MEUSLI: a Multilingual Projector for LLM-based ASR and Beyondcited by 1
  7. 2026Towards High-Level Semantic Intelligencecited by 0
  8. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0