# Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases

arXiv: https://arxiv.org/abs/2110.02939 · HTML: https://mlsanity.com/paper/2110.02939/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2021.

## Built on by (most cited first, 8 of 8)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [Dissecting Temporal Understanding in Text-to-Audio Retrieval](https://mlsanity.com/paper/2409.00851/) (2024)
- [Automated Audio Captioning and Language-Based Audio Retrieval](https://mlsanity.com/paper/2207.04156/) (2022)
- [Improving Natural-Language-based Audio Retrieval with Transfer Learning and Audio & Text Augmentations](https://mlsanity.com/paper/2208.11460/) (2022)
- [Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval](https://mlsanity.com/paper/2408.11641/) (2024)
- [Towards Diverse and Efficient Audio Captioning via Diffusion Models](https://mlsanity.com/paper/2409.09401/) (2024)
- [Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss](https://mlsanity.com/paper/2206.14659/) (2022)

## Abstract (arXiv)

We investigate unsupervised learning of correspondences between sound events and textual phrases through aligning audio clips with textual captions describing the content of a whole audio clip. We align originally unaligned and unannotated audio clips and their captions by scoring the similarities between audio frames and words, as encoded by modality-specific encoders and using a ranking-loss criterion to optimize the model. After training, we obtain clip-caption similarity by averaging frame-word similarities and estimate event-phrase correspondences by calculating frame-phrase similarities. We evaluate the method with two cross-modal tasks: audio-caption retrieval, and phrase-based sound event detection (SED). Experimental results show that the proposed method can globally associate audio clips with captions as well as locally learn correspondences between individual sound events and textual phrases in an unsupervised manner.