mlsanityData to 14 Oct 2026

Listen, Think, and Understand

arXiv 2305.10790 ↗First posted 18 May 2023eess.AS cs.SDcited by 209

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general categories, but also to listen to the finer details of the sounds, explain the reason for the predictions, think about what the sound infers, and understand the scene and what action needs to be taken, if any. Such capabilities beyond perception are not yet present in existing audio models. On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities. Therefore, we ask the question: can we build a model that has both audio perception and a reasoning ability? In this paper, we propose a new audio foundation model, called LTU (Listen, Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse (audio, question, answer) tuples, and have used an autoregressive training framework with a perception-to-understanding curriculum. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning. More importantly, it exhibits emerging audio reasoning and comprehension abilities that are absent in existing audio models. To the best of our knowledge, LTU is one of the first multimodal large language models that focus on general audio (rather than just speech) understanding.

Key facts. Cited by 209 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 May 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 71; 2025: 64; 2026: 50.

209 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  5. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  6. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  7. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  9. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  10. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  11. 2024Large Language Models for Time Series: A Surveycited by 114
  12. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
38 more
  1. 2025Step-Audio 2 Technical Reportcited by 105
  2. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  3. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  4. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  5. 2023Joint Audio and Speech Understandingcited by 100
  6. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
  7. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  8. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  9. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  10. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  11. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  12. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  13. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  14. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  15. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  16. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  17. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  18. 2024Language Model Can Listen While Speakingcited by 63
  19. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  20. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  21. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  22. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  23. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  24. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
  25. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  26. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  27. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  28. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  29. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 38
  30. 2024Foundation Models for Music: A Surveycited by 37
  31. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  32. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  33. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  34. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  35. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  36. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  37. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  38. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31

50 most cited of 209 All 209 (JSON)