# Automated Audio Captioning and Language-Based Audio Retrieval

arXiv: https://arxiv.org/abs/2207.04156 · HTML: https://mlsanity.com/paper/2207.04156/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 12 of 12)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events](https://mlsanity.com/paper/2102.11474/) (2021)
- [Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases](https://mlsanity.com/paper/2110.02939/) (2021)
- [Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning](https://mlsanity.com/paper/2110.07410/) (2021)
- [Audio Captioning Using Sound Event Detection](https://mlsanity.com/paper/2110.01210/) (2021)

## Built on by (most cited first, 2 of 2)

- [Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement](https://mlsanity.com/paper/2308.11923/) (2023)
- [ATIR: Towards Audio-Text Interleaved Contextual Retrieval](https://mlsanity.com/paper/2604.20267/) (2026)

## Abstract (arXiv)

This project involved participation in the DCASE 2022 Competition (Task 6) which had two subtasks: (1) Automated Audio Captioning and (2) Language-Based Audio Retrieval. The first subtask involved the generation of a textual description for audio samples, while the goal of the second was to find audio samples within a fixed dataset that match a given description. For both subtasks, the Clotho dataset was used. The models were evaluated on BLEU1, BLEU2, BLEU3, ROUGEL, METEOR, CIDEr, SPICE, and SPIDEr scores for audio captioning and R1, R5, R10 and mARP10 scores for audio retrieval. We have conducted a handful of experiments that modify the baseline models for these tasks. Our final architecture for Automated Audio Captioning is close to the baseline performance, while our model for Language-Based Audio Retrieval has surpassed its counterpart.