# Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss

arXiv: https://arxiv.org/abs/2303.05681 · HTML: https://mlsanity.com/paper/2303.05681/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2023.

## Built on by (most cited first, 12 of 12)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?](https://mlsanity.com/paper/2308.15090/) (2023)
- [Dissecting Temporal Understanding in Text-to-Audio Retrieval](https://mlsanity.com/paper/2409.00851/) (2024)
- [Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrieval](https://mlsanity.com/paper/2406.15897/) (2024)
- [Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval](https://mlsanity.com/paper/2408.11641/) (2024)
- [A Language-based solution to enable Metaverse Retrieval](https://mlsanity.com/paper/2312.14630/) (2023)
- [Distance Sampling-based Paraphraser Leveraging ChatGPT for Text Data Manipulation](https://mlsanity.com/paper/2405.00367/) (2024)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)

## Abstract (arXiv)

In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.