# An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

arXiv: https://arxiv.org/abs/2607.21424 · HTML: https://mlsanity.com/paper/2607.21424/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence](https://mlsanity.com/paper/2508.13992/) (2025)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [AudioTime: A Temporally-aligned Audio-text Benchmark Dataset](https://mlsanity.com/paper/2407.02857/) (2024)
- [MACE: Leveraging Audio for Evaluating Audio Captioning Systems](https://mlsanity.com/paper/2411.00321/) (2024)
- [The language of sound search: Examining User Queries in Audio Search Engines](https://mlsanity.com/paper/2410.08324/) (2024)
- [TAC: Timestamped Audio Captioning](https://mlsanity.com/paper/2602.15766/) (2026)
- [Audiocards: Structured Metadata Improves Audio Language Models For Sound Design](https://mlsanity.com/paper/2602.13835/) (2026)
- [ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding](https://mlsanity.com/paper/2603.24038/) (2026)
- [CapTune: Adapting Non-Speech Captions With Anchored Generative Models](https://mlsanity.com/paper/2508.19971/) (2025)

## Abstract (arXiv)

Recent advances in automated audio captioning (AAC) are driving a shift from monolithic sentences toward structured formats that disentangle acoustic and semantic properties, such as timestamped captions for different sound events. Such representations can support faceted sound search for creators and richer access to auditory information for Deaf and Hard of Hearing people. Yet, it remains unclear how to meaningfully evaluate these hybrid, structured captions. We propose an evaluation framework for structured audio descriptions, spanning five complementary axes: tag sets, descriptions, reasoning, numeric measurements, and spectral profiles. The framework combines large language model (LLM) judges for semantic fields with deterministic metrics for temporal and acoustic attributes. To validate these metrics, we introduce controlled perturbations that apply typed, graded changes to ground-truth annotations. Results show that the proposed metrics remain robust to meaning-preserving paraphrases while responding to genuine semantic and acoustic corruptions, enabling more reliable evaluation of structured captions.