mlsanityData to 7 Oct 2026

GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities

arXiv 2406.11768 ↗First posted 17 Jun 2024cs.SD cs.AI cs.CLcited by 151

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We build GAMA by integrating an LLM with multiple types of audio representations, including features from a custom Audio Q-Former, a multi-layer aggregator that aggregates features from multiple layers of an audio encoder. We fine-tune GAMA on a large-scale audio-language dataset, which augments it with audio understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for Complex Audio Reasoning), a synthetically generated instruction-tuning (IT) dataset with instructions that require the model to perform complex reasoning on the input audio. We instruction-tune GAMA with CompA-R to endow it with complex reasoning abilities, where we further add a soft prompt as input with high-level semantic evidence by leveraging event tags of the input audio. Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for evaluating the capabilities of LALMs on open-ended audio question-answering that requires complex reasoning. Through automated and expert human evaluations, we show that GAMA outperforms all other LALMs in literature on diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on CompA-R proves to be superior in its complex reasoning and instruction following capabilities.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 52 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

28 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models12 Oct

Other references · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
15 more
  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  2. 2023MusicLM: Generating Music From Textcited by 442
  3. 2019Clotho: An Audio Captioning Datasetcited by 354
  4. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  5. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  6. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  7. 2023The False Promise of Imitating Proprietary LLMscited by 254
  8. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  9. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  10. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  11. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  12. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  13. 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
  14. 2023Ada-Instruct: Adapting Instruction Generators for Complex Reasoningcited by 11
  15. 2022Enemy Spotted: in-game gun sound dataset for gunshot classification and localizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 77; 2026: 57.

151 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2025Kimi-Audio Technical Reportcited by 210
  3. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  4. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  7. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  8. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  9. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  10. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  11. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  12. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
38 more
  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  2. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  3. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
  4. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
  5. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  6. 2024Taming Data and Transformers for Audio Generationcited by 28
  7. 2025Mellow: a small audio language model for reasoningcited by 27
  8. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  9. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  10. 2026Large Language Model Reasoning Failurescited by 20
  11. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  12. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  13. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  14. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  15. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  16. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  17. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  18. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  19. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalitiescited by 13
  20. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  21. 2025BLAB: Brutally Long Audio Benchcited by 11
  22. 2025Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoningcited by 10
  23. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  24. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Modelscited by 9
  25. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Modelscited by 9
  26. 2026MOSS-Audio Technical Reportcited by 9
  27. 2025CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoningcited by 8
  28. 2025Attacker's Noise Can Manipulate Your Audio-based LLM in the Real Worldcited by 8
  29. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  30. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  31. 2025Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Promptingcited by 7
  32. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  33. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  34. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 7
  35. 2025BRACE: A Benchmark for Robust Audio Caption Quality Evaluationcited by 7
  36. 2025`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMscited by 6
  37. 2025Soundwave: Less is More for Speech-Text Alignment in LLMscited by 6
  38. 2025FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusioncited by 6

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AudioRouter: Data Efficient Audio Understanding via RL based Dual Reasoning
  2. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
  3. 2026Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
  4. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
  5. 2026Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning
  6. 2026Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models
  7. 2025Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning
  8. 2026Continuous Audio Thinking for Large Audio Language Models