# Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer

arXiv: https://arxiv.org/abs/2305.01482 · HTML: https://mlsanity.com/paper/2305.01482/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2023.

## Built on by (most cited first, 3 of 3)

- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)

## Abstract (arXiv)

In this work, we propose to study the performance of a model trained with a sentence embedding regression loss component for the Automated Audio Captioning task. This task aims to build systems that can describe audio content with a single sentence written in natural language. Most systems are trained with the standard Cross-Entropy loss, which does not take into account the semantic closeness of the sentence. We found that adding a sentence embedding loss term reduces overfitting, but also increased SPIDEr from 0.397 to 0.418 in our first setting on the AudioCaps corpus. When we increased the weight decay value, we found our model to be much closer to the current state-of-the-art methods, with a SPIDEr score up to 0.444 compared to a 0.475 score. Moreover, this model uses eight times less trainable parameters. In this training setting, the sentence embedding loss has no more impact on the model performance.