mlsanityData to 7 Oct 2026

SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation

arXiv 2310.09424 ↗First posted 13 Oct 2023cs.CL cs.HC cs.SDcited by 75

Not on the map. Why?

Abstract (arXiv)

We present a novel Speech Augmented Language Model (SALM) with em multitask and em in-context learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, em speech supervised in-context training is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

34 of 40 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  11. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
22 more
  1. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  2. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  3. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  4. 2023Listen, Think, and Understandcited by 206
  5. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  6. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  7. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  8. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  9. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  10. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  13. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  14. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  15. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  16. 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
  17. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  18. 2018On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translationcited by 17
  19. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  20. 2020Accelerating RNN Transducer Inference via One-Step Constrained Beam Searchcited by 13
  21. 2023Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognitioncited by 11
  22. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 27; 2026: 24.

75 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  4. 2024A Survey on LoRA of Large Language Modelscited by 67
  5. 2024Language Model Can Listen While Speakingcited by 62
  6. 2025Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and ASTcited by 51
  7. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  8. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  9. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 25
  10. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  11. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  12. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
38 more
  1. 2024Comparing Discrete and Continuous Space LLMs for Speech Recognitioncited by 15
  2. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 15
  3. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  4. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  5. 2024Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbationcited by 11
  6. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  7. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  8. 2024Instruction Data Generation and Unsupervised Adaptation for Speech Language Modelscited by 9
  9. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9
  10. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  11. 2024Audio Dialogues: Dialogues dataset for audio and music understandingcited by 7
  12. 2024Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spottercited by 7
  13. 2025In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varietiescited by 7
  14. 2025NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decodingcited by 7
  15. 2024Using Large Language Model for End-to-End Chinese ASR and NERcited by 6
  16. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  17. 2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
  18. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  19. 2024EMMeTT: Efficient Multimodal Machine Translation Trainingcited by 4
  20. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  21. 2025TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Treecited by 4
  22. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
  23. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  24. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  25. 2025LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriorscited by 3
  26. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  27. 2025Locate-and-Focus: Enhancing Terminology Translation in Speech Language Modelscited by 3
  28. 2025SpeechLLMs for Large-scale Contextualized Zero-shot Slot Fillingcited by 3
  29. 2025Speech Translation Refinement using Large Language Modelscited by 2
  30. 2025Synthetic data enables context-aware bioacoustic sound event detectioncited by 2
  31. 2025CMU's IWSLT 2025 Simultaneous Speech Translation Systemcited by 2
  32. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  33. 2026SE-DiCoW: Self-Enrolled Diarization-Conditioned Whispercited by 2
  34. 2026MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audiocited by 2
  35. 2025SparQLe: Speech Queries to Text Translation Through LLMscited by 1
  36. 2025Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishingcited by 1
  37. 2025Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Datacited by 1
  38. 2025SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settingscited by 1

50 most cited of 75 All 75 (JSON)