mlsanityData to 7 Oct 2026

Joint Audio and Speech Understanding

arXiv 2309.14405 ↗First posted 25 Sep 2023cs.SD cs.AI eess.AScited by 99

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

34 of 48 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
22 more
  1. 2021AST: Audio Spectrogram Transformercited by 634
  2. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  3. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  4. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  5. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  6. 2023Listen, Think, and Understandcited by 206
  7. 2016FMA: A Dataset For Music Analysiscited by 205
  8. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  9. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  10. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  11. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  12. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  13. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  14. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  15. 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
  16. 2021VoxCeleb Enrichment for Age and Gender Recognitioncited by 20
  17. 2023PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Modelscited by 17
  18. 2019Bottom-up Broadcast Neural Network For Music Genre Classificationcited by 10
  19. 2019Crowdsourcing a Dataset of Audio Captionscited by 8
  20. 2022Joint Speech Recognition and Audio Captioningcited by 5
  21. 2023Unified Keyword Spotting and Audio Tagging on Mobile Devices with Transformerscited by 3
  22. 2023Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddingscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 35; 2025: 45; 2026: 18.

99 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  3. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  5. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  6. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  7. 2025Step-Audio 2 Technical Reportcited by 101
  8. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  9. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  10. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
  11. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  12. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 55
38 more
  1. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  2. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  4. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  5. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  6. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  7. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  8. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  9. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  10. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  11. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  12. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  13. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  14. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  15. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 20
  16. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  17. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  18. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  19. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  20. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  21. 2024Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Modelscited by 15
  22. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  23. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  24. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  25. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  26. 2025BLAB: Brutally Long Audio Benchcited by 11
  27. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  28. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  29. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  30. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  31. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  32. 2026MOSS-Audio Technical Reportcited by 9
  33. 2024Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimizationcited by 8
  34. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  35. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  36. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  37. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  38. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning
  2. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
  3. 2024What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models
  4. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
  5. 2026Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models
  6. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
  7. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
  8. 2026AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoning