mlsanityData to 7 Oct 2026

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

arXiv 2602.23649 ↗First posted 27 Feb 2026cs.SD cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: textitaccuracy (semantic correctness), textitcompleteness (coverage of reference content), and textithallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 23 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

15 of 23 references.

References · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2021AST: Audio Spectrogram Transformercited by 634
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  7. 2023MusicLM: Generating Music From Textcited by 442
  8. 2019Clotho: An Audio Captioning Datasetcited by 354
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  11. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  12. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
3 more
  1. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  2. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
  3. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructionscited by 1
  2. 2026PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
  2. 2024CLAIR-A: Leveraging Large Language Models to Judge Audio Captions
  3. 2024ETTA: Elucidating the Design Space of Text-to-Audio Models
  4. 2026What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
  5. 2026Rethinking Music Captioning with Music Metadata LLMs
  6. 2025AHELM: A Holistic Evaluation of Audio-Language Models
  7. 2026A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models
  8. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence