Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrieval
Not on the map. Why?
Abstract (arXiv)
Matching raw audio signals with textual descriptions requires understanding the audio's content and the description's semantics and then drawing connections between the two modalities. This paper investigates a hybrid retrieval system that utilizes audio metadata as an additional clue to understand the content of audio signals before matching them with textual queries. We experimented with metadata often attached to audio recordings, such as keywords and natural-language descriptions, and we investigated late and mid-level fusion strategies to merge audio and metadata. Our hybrid approach with keyword metadata and late fusion improved the retrieval performance over a content-based baseline by 2.36 and 3.69 pp. mAP@10 on the ClothoV2 and AudioCaps benchmarks, respectively.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 22 Jun 2024.
Builds on
24 of 26 references.
References · 24
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Searching for MobileNetV3cited by 1,994
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2020Multi-modal Transformer for Video Retrievalcited by 395
- 2019Clotho: An Audio Captioning Datasetcited by 358
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
- 2021Efficient Training of Audio Transformers with Patchoutcited by 226
12 more
- 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
- 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
- 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
- 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
- 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 27
- 2023Audio-Text Models Do Not Yet Leverage Natural Languagecited by 26
- 2024Cacophony: An Improved Contrastive Audio-Text Modelcited by 18
- 2023Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Losscited by 12
- 2022Audio-text Retrieval in Contextcited by 10
- 2023Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Setscited by 6
- 2023Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?cited by 3
- 2022Improving Natural-Language-based Audio Retrieval with Transfer Learning and Audio & Text Augmentationscited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.