# Data leakage in cross-modal retrieval training: A case study

arXiv: https://arxiv.org/abs/2302.12258 · HTML: https://mlsanity.com/paper/2302.12258/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 11 of 11)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use](https://mlsanity.com/paper/1306.1461/) (2013)
- [Language-based Audio Retrieval Task in DCASE 2022 Challenge](https://mlsanity.com/paper/2209.09967/) (2022)
- [Matching Text and Audio Embeddings: Exploring Transfer-learning Strategies for Language-based Audio Retrieval](https://mlsanity.com/paper/2210.02833/) (2022)

## Built on by (most cited first, 1 of 1)

- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)

## Abstract (arXiv)

The recent progress in text-based audio retrieval was largely propelled by the release of suitable datasets. Since the manual creation of such datasets is a laborious task, obtaining data from online resources can be a cheap solution to create large-scale datasets. We study the recently proposed SoundDesc benchmark dataset, which was automatically sourced from the BBC Sound Effects web page. In our analysis, we find that SoundDesc contains several duplicates that cause leakage of training data to the evaluation data. This data leakage ultimately leads to overly optimistic retrieval performance estimates in previous benchmarks. We propose new training, validation, and testing splits for the dataset that we make available online. To avoid weak contamination of the test data, we pool audio files that share similar recording setups. In our experiments, we find that the new splits serve as a more challenging benchmark.