# Language-based Audio Moment Retrieval

arXiv: https://arxiv.org/abs/2409.15672 · HTML: https://mlsanity.com/paper/2409.15672/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset](https://mlsanity.com/paper/2305.18500/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [End-to-End Neural Speaker Diarization with Permutation-Free Objectives](https://mlsanity.com/paper/1909.05952/) (2019)
- [Audio Retrieval with Natural Language Queries: A Benchmark Study](https://mlsanity.com/paper/2112.09418/) (2021)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [Sound Event Detection: A Tutorial](https://mlsanity.com/paper/2107.05463/) (2021)
- [Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection](https://mlsanity.com/paper/1702.06286/) (2017)
- [Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline](https://mlsanity.com/paper/2303.12930/) (2023)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation](https://mlsanity.com/paper/2406.02032/) (2024)
- [Fine-tune the pretrained ATST model for sound event detection](https://mlsanity.com/paper/2309.08153/) (2023)
- [AST-SED: An Effective Sound Event Detection Method Based on Audio Spectrogram Transformer](https://mlsanity.com/paper/2303.03689/) (2023)
- [Target Sound Extraction with Variable Cross-modality Clues](https://mlsanity.com/paper/2303.08372/) (2023)
- [Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Sets](https://mlsanity.com/paper/2308.04258/) (2023)
- [Language Modelling for Sound Event Detection with Teacher Forcing and Scheduled Sampling](https://mlsanity.com/paper/1907.08506/) (2019)
- [Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2408.02901/) (2024)
- [Sound Event Detection by Multitask Learning of Sound Events and Scenes with Soft Scene Labels](https://mlsanity.com/paper/2002.05848/) (2020)
- [Acoustic Event Detection with Classifier Chains](https://mlsanity.com/paper/2202.08470/) (2022)

## Built on by (most cited first, 4 of 4)

- [CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries](https://mlsanity.com/paper/2511.15131/) (2025)
- [Multimodal Data Curation Through Ranked Retrieval](https://mlsanity.com/paper/2605.01163/) (2026)
- [GigaChat Audio: Time-aware Large Audio Language Model](https://mlsanity.com/paper/2607.10387/) (2026)
- [Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception](https://mlsanity.com/paper/2609.15215/) (2026)

## Abstract (arXiv)

In this paper, we propose and design a new task called audio moment retrieval (AMR). Unlike conventional language-based audio retrieval tasks that search for short audio clips from an audio database, AMR aims to predict relevant moments in untrimmed long audio based on a text query. Given the lack of prior work in AMR, we first build a dedicated dataset, Clotho-Moment, consisting of large-scale simulated audio recordings with moment annotations. We then propose a DETR-based model, named Audio Moment DETR (AM-DETR), as a fundamental framework for AMR tasks. This model captures temporal dependencies within audio features, inspired by similar video moment retrieval tasks, thus surpassing conventional clip-level audio retrieval methods. Additionally, we provide manually annotated datasets to properly measure the effectiveness and robustness of our methods on real data. Experimental results show that AM-DETR, trained with Clotho-Moment, outperforms a baseline model that applies a clip-level audio retrieval method with a sliding window on all metrics, particularly improving Recall1@0.7 by 9.00 points. Our datasets and code are publicly available in https://h-munakata.github.io/Language-based-Audio-Moment-Retrieval.