# ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models

arXiv: https://arxiv.org/abs/2510.23558 · HTML: https://mlsanity.com/paper/2510.23558/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://mlsanity.com/paper/2401.03601/) (2024)
- [WavLLM: Towards Robust and Adaptive Speech Large Language Model](https://mlsanity.com/paper/2404.00656/) (2024)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)
- [A Survey on Speech Large Language Models for Understanding](https://mlsanity.com/paper/2410.18908/) (2024)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [On the Worst Prompt Performance of Large Language Models](https://mlsanity.com/paper/2406.10248/) (2024)
- [DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment](https://mlsanity.com/paper/2406.18871/) (2024)
- [Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models](https://mlsanity.com/paper/2505.19037/) (2025)
- [UltraIF: Advancing Instruction Following from the Wild](https://mlsanity.com/paper/2502.04153/) (2025)
- [CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following](https://mlsanity.com/paper/2506.12285/) (2025)
- [IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Models](https://mlsanity.com/paper/2505.16774/) (2025)
- [AHAMask: Reliable Task Specification for Large Audio Language Models without Instructions](https://mlsanity.com/paper/2509.01787/) (2025)

## Built on by (most cited first, 3 of 3)

- [SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models](https://mlsanity.com/paper/2601.07331/) (2026)
- [When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal](https://mlsanity.com/paper/2603.20433/) (2026)
- [EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation](https://mlsanity.com/paper/2605.23954/) (2026)

## Abstract (arXiv)

Large Audio Language Models (LALMs), which couple acoustic perception with large language models (LLMs) to extract and understand diverse information from audio, have attracted intense interest from both academic and industrial communities. However, existing LALMs are highly sensitive to how instructions are phrased, affecting both (i) instruction-following rates and (ii) task performance. Yet, no existing benchmarks offer a systematic and comprehensive evaluation of this sensitivity. We introduce ISA-Bench, a dynamic benchmark evaluating instruction sensitivity for LALMs along three axes: instruction description, output format, and task composition. We assess recent open-source and proprietary LALMs using ISA-Bench, profiling both compliance and accuracy under controlled instruction variations. Experimental results reveal that even state-of-the-art LALMs suffer significant instruction sensitivity, leading to degraded performance on fundamental audio understanding tasks. To mitigate this issue, we fine-tune Qwen2-Audio on a specifically constructed complex instruction-variant dataset, achieving a marked improvement in instruction-following performance. However, this also induces nontrivial catastrophic forgetting: the model loses some previously mastered task capabilities when exposed to new instruction styles. Our benchmark provides a standardized basis for assessing and improving instruction sensitivity in LALMs, underscoring the need for instruction-robust audio understanding in real-world pipelines.