# Dissecting Temporal Understanding in Text-to-Audio Retrieval

arXiv: https://arxiv.org/abs/2409.00851 · HTML: https://mlsanity.com/paper/2409.00851/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Uncovering Hidden Challenges in Query-Based Video Moment Retrieval](https://mlsanity.com/paper/2009.00325/) (2020)
- [A CLIP-Hitchhiker's Guide to Long Video Retrieval](https://mlsanity.com/paper/2205.08508/) (2022)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [Audio-Text Models Do Not Yet Leverage Natural Language](https://mlsanity.com/paper/2303.10667/) (2023)
- [Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events](https://mlsanity.com/paper/2102.11474/) (2021)
- [Improving Text-Audio Retrieval by Text-aware Attention Pooling and Prior Matrix Revised Loss](https://mlsanity.com/paper/2303.05681/) (2023)
- [Training Audio Captioning Models without Audio](https://mlsanity.com/paper/2309.07372/) (2023)
- [Towards Weakly Supervised Text-to-Audio Grounding](https://mlsanity.com/paper/2401.02584/) (2024)
- [Unsupervised Audio-Caption Aligning Learns Correspondences between Individual Sound Events and Textual Phrases](https://mlsanity.com/paper/2110.02939/) (2021)
- [FLAP: Fast Language-Audio Pre-training](https://mlsanity.com/paper/2311.01615/) (2023)
- [A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval](https://mlsanity.com/paper/2402.19106/) (2024)
- [A Detailed Audio-Text Data Simulation Pipeline using Single-Event Sounds](https://mlsanity.com/paper/2403.04594/) (2024)

## Built on by (most cited first, 3 of 3)

- [ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval](https://mlsanity.com/paper/2605.03361/) (2026)
- [TAIL: Text-Audio Incremental Learning](https://mlsanity.com/paper/2503.04258/) (2025)
- [Forgive or forget: Understanding the context of hate in audio retrieval systems](https://mlsanity.com/paper/2606.05857/) (2026)

## Abstract (arXiv)

Recent advancements in machine learning have fueled research on multimodal tasks, such as for instance text-to-video and text-to-audio retrieval. These tasks require models to understand the semantic content of video and audio data, including objects, and characters. The models also need to learn spatial arrangements and temporal relationships. In this work, we analyse the temporal ordering of sounds, which is an understudied problem in the context of text-to-audio retrieval. In particular, we dissect the temporal understanding capabilities of a state-of-the-art model for text-to-audio retrieval on the AudioCaps and Clotho datasets. Additionally, we introduce a synthetic text-audio dataset that provides a controlled setting for evaluating temporal capabilities of recent models. Lastly, we present a loss function that encourages text-audio models to focus on the temporal ordering of events. Code and data are available at https://www.robots.ox.ac.uk/~vgg/research/audio-retrieval/dtu/.