# MACE: Leveraging Audio for Evaluating Audio Captioning Systems

arXiv: https://arxiv.org/abs/2411.00321 · HTML: https://mlsanity.com/paper/2411.00321/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2024.

## Built on by (most cited first, 5 of 5)

- [Mellow: a small audio language model for reasoning](https://mlsanity.com/paper/2503.08540/) (2025)
- [Aligning Audio Captions with Human Preferences](https://mlsanity.com/paper/2509.14659/) (2025)
- [AURA Score: A Metric For Holistic Audio Question Answering Evaluation](https://mlsanity.com/paper/2510.04934/) (2025)
- [Semantic-Aware Confidence Calibration for Automated Audio Captioning](https://mlsanity.com/paper/2512.10170/) (2025)
- [An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations](https://mlsanity.com/paper/2607.21424/) (2026)

## Abstract (arXiv)

The Automated Audio Captioning (AAC) task aims to describe an audio signal using natural language. To evaluate machine-generated captions, the metrics should take into account audio events, acoustic scenes, paralinguistics, signal characteristics, and other audio information. Traditional AAC evaluation relies on natural language generation metrics like ROUGE and BLEU, image captioning metrics such as SPICE and CIDEr, or Sentence-BERT embedding similarity. However, these metrics only compare generated captions to human references, overlooking the audio signal itself. In this work, we propose MACE (Multimodal Audio-Caption Evaluation), a novel metric that integrates both audio and reference captions for comprehensive audio caption evaluation. MACE incorporates audio information from audio as well as predicted and reference captions and weights it with a fluency penalty. Our experiments demonstrate MACE's superior performance in predicting human quality judgments compared to traditional metrics. Specifically, MACE achieves a 3.28% and 4.36% relative accuracy improvement over the FENSE metric on the AudioCaps-Eval and Clotho-Eval datasets respectively. Moreover, it significantly outperforms all the previous metrics on the audio captioning evaluation task. The metric is opensourced at https://github.com/satvik-dixit/mace