# CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries

arXiv: https://arxiv.org/abs/2511.15131 · HTML: https://mlsanity.com/paper/2511.15131/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [The Benefit Of Temporally-Strong Labels In Audio Event Classification](https://mlsanity.com/paper/2105.07031/) (2021)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline](https://mlsanity.com/paper/2303.12930/) (2023)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [Frequency Dynamic Convolution: Frequency-Adaptive Pattern Recognition for Sound Event Detection](https://mlsanity.com/paper/2203.15296/) (2022)
- [AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Models](https://mlsanity.com/paper/2411.18953/) (2024)
- [M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation](https://mlsanity.com/paper/2406.02032/) (2024)
- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [Forward-Backward Convolutional Recurrent Neural Networks and Tag-Conditioned Convolutional Neural Networks for Weakly Labeled Semi-supervised Sound Event Detection](https://mlsanity.com/paper/2103.06581/) (2021)
- [DCASE 2024 Task 4: Sound Event Detection with Heterogeneous Data and Missing Labels](https://mlsanity.com/paper/2406.08056/) (2024)
- [Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2408.02901/) (2024)
- [Language-based Audio Moment Retrieval](https://mlsanity.com/paper/2409.15672/) (2024)
- [Improving Audio Spectrogram Transformers for Sound Event Detection Through Multi-Stage Training](https://mlsanity.com/paper/2408.00791/) (2024)
- [MomentMix Augmentation with Length-Aware DETR for Temporally Robust Moment Retrieval](https://mlsanity.com/paper/2412.20816/) (2024)
- [Sound event detection with audio-text models and heterogeneous temporal annotations](https://mlsanity.com/paper/2508.20703/) (2025)

## Built on by (most cited first, 3 of 3)

- [Event-Grounded Question Answering over Long Audio via Structured Retrieval](https://mlsanity.com/paper/2602.14612/) (2026)
- [Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding](https://mlsanity.com/paper/2607.04383/) (2026)
- [GigaChat Audio: Time-aware Large Audio Language Model](https://mlsanity.com/paper/2607.10387/) (2026)

## Abstract (arXiv)

We introduce CASTELLA, a human-annotated audio benchmark for the task of audio moment retrieval (AMR). Although AMR has various useful potential applications, there is still no established benchmark with real-world data. The initial study of AMR trained the models solely on synthetic datasets. Moreover, the evaluation is based on an annotated dataset of fewer than 100 samples. This resulted in less reliable reported performance. To ensure performance for applications in real-world environments, we present CASTELLA, a large-scale manually annotated AMR dataset. CASTELLA consists of 1009, 213, and 640 audio recordings for training, validation, and test splits, respectively, which is 24 times larger than the previous dataset. We also establish a baseline model for AMR using CASTELLA. Our experiments demonstrate that a model fine-tuned on CASTELLA after pre-training on the synthetic data outperformed a model trained solely on the synthetic data by 10.4 points in Recall1@0.7. CASTELLA is publicly available in https://h-munakata.github.io/CASTELLA-demo/.