mlsanityData to 14 Oct 2026

AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions

arXiv 2509.01787 ↗First posted 1 Sep 2025eess.AS cs.AI cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

Although current large audio language models (LALMs) extend text large language models (LLMs) with generic acoustic understanding abilities, they usually suffer from prompt sensitivity, where different instructions of the same intention can yield drastically different outcomes. In this work, we propose AHAMask, where we simply mask some of the attention heads in the decoder-only LLM backbone of LALMs, to trigger specific acoustic task functionalities without instructions. These masks are efficiently obtained by training on an LALM, with the number of trainable parameters equal to the attention head count in its LLM backbone. We show by experiments that applying such selective attention head masks achieves comparable or even better performance than using instructions, either on single or composite tasks. Besides achieving reliable acoustic task specification for LALMs, this also reveals that LALMs exhibit certain "functional pathways" in their attention heads.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 1 Sep 2025.

Builds on

27 of 48 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  10. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  11. 2025Kimi-Audio Technical Reportcited by 222
  12. 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
15 more
  1. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  2. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  3. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  4. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  6. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  7. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  8. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Modelscited by 34
  9. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  10. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  11. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  12. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  13. 2020How Does Selective Mechanism Improve Self-Attention Networks?cited by 19
  14. 2025UltraIF: Advancing Instruction Following from the Wildcited by 18
  15. 2025OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academiacited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Modelscited by 3
  2. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMscited by 1
  3. 2026Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different setcited by 0