mlsanityData to 14 Oct 2026

SALMONN: Towards Generic Hearing Abilities for Large Language Models

arXiv 2310.13289 ↗First posted 20 Oct 2023cs.SD cs.CL eess.AScited by 506

Not on the map. Why?

Cited by 85 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning etc. The presence of cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. The source code, model checkpoints and data are available at https://github.com/bytedance/SALMONN.

Key facts. Cited by 506 papers in the map (data through 14 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
34 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  5. 2023Instruction Tuning with GPT-4cited by 686
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  7. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  8. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  9. 2019Clotho: An Audio Captioning Datasetcited by 358
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  11. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  12. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  13. 2023Listen, Think, and Understandcited by 209
  14. 2022Learning Video Representations from Large Language Modelscited by 205
  15. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  16. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  17. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  18. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  19. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  20. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  21. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  22. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  23. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  24. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  25. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  26. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  27. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  28. 2016Learning Features of Music from Scratchcited by 58
  29. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  30. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  31. 2020Emotion recognition by fusing time synchronous and time asynchronous representationscited by 26
  32. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  33. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  34. 2022Joint Speech Recognition and Audio Captioningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 121; 2025: 199; 2026: 183.

506 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 736
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  4. 2025Kimi-Audio Technical Reportcited by 222
  5. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  6. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  7. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  8. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  9. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  10. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  11. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  12. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
38 more
  1. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  2. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  3. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  4. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  5. 2024Large Language Models for Time Series: A Surveycited by 114
  6. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  7. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  8. 2025Step-Audio 2 Technical Reportcited by 105
  9. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  10. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  11. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  12. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  13. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  14. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  15. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  16. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  17. 2024Best-of-N Jailbreakingcited by 76
  18. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  19. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  20. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  21. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 68
  22. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 65
  23. 2024Language Model Can Listen While Speakingcited by 63
  24. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 62
  25. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  26. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  27. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  28. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  29. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  30. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  31. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 47
  32. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  33. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  34. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  35. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  36. 2024Baichuan-Omni Technical Reportcited by 42
  37. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  38. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39

50 most cited of 506 All 506 (JSON)