mlsanityData to 14 Oct 2026

Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning

arXiv 2505.13115 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 10

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

The popular success of text-based large language models (LLM) has streamlined the attention of the multimodal community to combine other modalities like vision and audio along with text to achieve similar multimodal capabilities. In this quest, large audio language models (LALMs) have to be evaluated on reasoning related tasks which are different from traditional classification or generation tasks. Towards this goal, we propose a novel dataset called temporal reasoning evaluation of audio (TREA). We benchmark open-source LALMs and observe that they are consistently behind human capabilities on the tasks in the TREA dataset. While evaluating LALMs, we also propose an uncertainty metric, which computes the invariance of the model to semantically identical perturbations of the input. Our analysis shows that the accuracy and uncertainty metrics are not necessarily correlated and thus, points to a need for wholesome evaluation of LALMs for high-stakes applications.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 43 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

38 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark24 Oct
  2. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning21 Oct
  3. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities17 Jun
  4. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models12 Oct

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Mistral 7Bcited by 3,164
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
22 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  5. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  8. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  9. 2023Pengi: An Audio Language Model for Audio Taskscited by 220
  10. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  11. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  12. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  13. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  14. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  15. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  16. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  17. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  18. 2023Can Whisper perform speech-based in-context learning?cited by 35
  19. 2024TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluationscited by 13
  20. 2020Classification Confidence Estimation with Test-Time Data-Augmentationcited by 12
  21. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  22. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agentscited by 6
  2. 2025FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMscited by 5
  3. 2025Robustness assessment of large audio language models in multiple-choice evaluationcited by 5
  4. 2026Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoningcited by 3
  5. 2025ORCA: Open-ended Response Correctness Assessment for Audio Question Answeringcited by 2
  6. 2026A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Modelscited by 2
  7. 2026TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Modelscited by 1
  8. 2026Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognitioncited by 0
  9. 2026BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMscited by 0
  10. 2026On Temporal Binding in Large Audio Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Reliable Large Audio Language Model
  2. 2024Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models
  3. 2025AHELM: A Holistic Evaluation of Audio-Language Models
  4. 2025Mellow: a small audio language model for reasoning
  5. 2025Not in Sync: Unveiling Temporal Bias in Audio Chat Models
  6. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
  7. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models
  8. 2026A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models