# Can Audio Captions Be Evaluated with Image Caption Metrics?

arXiv: https://arxiv.org/abs/2110.04684 · HTML: https://mlsanity.com/paper/2110.04684/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [TIGEr: Text-to-Image Grounding for Image Caption Evaluation](https://mlsanity.com/paper/1909.02050/) (2019)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning](https://mlsanity.com/paper/2102.11457/) (2021)
- [An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning](https://mlsanity.com/paper/2108.02752/) (2021)
- [Audio Caption: Listen and Tell](https://mlsanity.com/paper/1902.09254/) (2019)
- [Enriching Ontology with Temporal Commonsense for Low-Resource Audio Tagging](https://mlsanity.com/paper/2110.01009/) (2021)

## Built on by (most cited first, 30 of 33)

- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Learning Spatially-Aware Language and Audio Embeddings](https://mlsanity.com/paper/2409.11369/) (2024)
- [Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding](https://mlsanity.com/paper/2406.13275/) (2024)
- [DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning](https://mlsanity.com/paper/2410.09472/) (2024)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering](https://mlsanity.com/paper/2601.14728/) (2026)
- [BRACE: A Benchmark for Robust Audio Caption Quality Evaluation](https://mlsanity.com/paper/2512.10403/) (2025)
- [MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks](https://mlsanity.com/paper/2507.23511/) (2025)
- [Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models](https://mlsanity.com/paper/2409.06223/) (2024)
- [CLAIR-A: Leveraging Large Language Models to Judge Audio Captions](https://mlsanity.com/paper/2409.12962/) (2024)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?](https://mlsanity.com/paper/2308.15090/) (2023)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech](https://mlsanity.com/paper/2602.23649/) (2026)
- [Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity](https://mlsanity.com/paper/2210.06354/) (2022)
- [Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates](https://mlsanity.com/paper/2211.08983/) (2022)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)
- [EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation](https://mlsanity.com/paper/2410.12028/) (2024)
- [Aligning Audio Captions with Human Preferences](https://mlsanity.com/paper/2509.14659/) (2025)
- [LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2601.04658/) (2026)
- [UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning](https://mlsanity.com/paper/2606.04939/) (2026)
- [Improving Audio Caption Fluency with Automatic Error Correction](https://mlsanity.com/paper/2306.10090/) (2023)
- [S2Cap: A Benchmark and a Baseline for Singing Style Captioning](https://mlsanity.com/paper/2409.09866/) (2024)
- [Make Some Noise: Towards LLM audio reasoning and generation using sound tokens](https://mlsanity.com/paper/2503.22275/) (2025)
- [AURA Score: A Metric For Holistic Audio Question Answering Evaluation](https://mlsanity.com/paper/2510.04934/) (2025)
- [Semantic-Aware Confidence Calibration for Automated Audio Captioning](https://mlsanity.com/paper/2512.10170/) (2025)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)

## Abstract (arXiv)

Automated audio captioning aims at generating textual descriptions for an audio clip. To evaluate the quality of generated audio captions, previous works directly adopt image captioning metrics like SPICE and CIDEr, without justifying their suitability in this new domain, which may mislead the development of advanced models. This problem is still unstudied due to the lack of human judgment datasets on caption quality. Therefore, we firstly construct two evaluation benchmarks, AudioCaps-Eval and Clotho-Eval. They are established with pairwise comparison instead of absolute rating to achieve better inter-annotator agreement. Current metrics are found in poor correlation with human annotations on these datasets. To overcome their limitations, we propose a metric named FENSE, where we combine the strength of Sentence-BERT in capturing similarity, and a novel Error Detector to penalize erroneous sentences for robustness. On the newly established benchmarks, FENSE outperforms current metrics by 14-25% accuracy. Code, data and web demo available at: https://github.com/blmoistawinde/fense