mlsanityData to 7 Oct 2026

Hearing the Order: Investigating Position Bias in Large Audio-Language Models

arXiv 2510.00628 ↗First posted 1 Oct 2025cs.SD cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a form of position bias and undermine their reliability. In this paper, we identify and analyze this problem in LALMs. We demonstrate that no model is immune to this bias through extensive experiments on six LALMs across three widely used benchmarks and their spoken counterparts. Shuffling the order of answer options can cause performance fluctuations of up to 24% and even change model rankings, raising concerns about the reliability of current evaluation practices. We also study permutation-based strategies and show that they can mitigate bias in most cases. Our work represents the first systematic investigation of this issue in LALMs, and we hope it raises awareness and motivates further research in this direction.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 43 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

41 of 43 references.

References · 41

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2025Qwen2.5-Omni Technical Reportcited by 704
  11. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
29 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  4. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  5. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  6. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  7. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  8. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  9. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  10. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  11. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 66
  12. 2025MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligencecited by 59
  13. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  14. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  15. 2025Voxtralcited by 41
  16. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  17. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31
  18. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
  19. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  20. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
  21. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  22. 2024Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Modelscited by 20
  23. 2024Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questionscited by 15
  24. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Modelscited by 14
  25. 2024Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectorscited by 13
  26. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  27. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  28. 2024Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Modelscited by 11
  29. 2024Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Modelscited by 2
  2. 2026HumMusQA: A Human-written Music Understanding QA Benchmark Datasetcited by 1
  3. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  4. 2026Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputscited by 0