How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 60 of its 65 references that are in the map. First posted on arXiv 19 Mar 2026.
Builds on
60 of 65 references.
References · 60
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2024GPT-4o System Cardcited by 4,588
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2025OpenAI GPT-5 System Cardcited by 989
38 more
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2024Phi-4 Technical Reportcited by 711
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
- 2024Qwen2-Audio Technical Reportcited by 560
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
- 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
- 2025Qwen3-Omni Technical Reportcited by 497
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
- 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
- 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
- 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
- 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
- 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
- 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
- 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
- 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
- 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
- 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
- 2023Joint Audio and Speech Understandingcited by 100
- 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 95
- 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
- 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
- 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
- 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
- 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
- 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 42
- 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
- 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
- 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
- 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 30
- 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
- 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
- 2024Building a Taiwanese Mandarin Spoken Language Model: A First Attemptcited by 23
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.