# Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding

arXiv: https://arxiv.org/abs/2607.04383 · HTML: https://mlsanity.com/paper/2607.04383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 5 Jul 2026.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions](https://mlsanity.com/paper/2005.14623/) (2020)
- [A Framework for the Robust Evaluation of Sound Event Detection](https://mlsanity.com/paper/1910.08440/) (2019)
- [Time-R1: Towards Comprehensive Temporal Reasoning in LLMs](https://mlsanity.com/paper/2505.13508/) (2025)
- [AudioTime: A Temporally-aligned Audio-text Benchmark Dataset](https://mlsanity.com/paper/2407.02857/) (2024)
- [Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music](https://mlsanity.com/paper/2604.10905/) (2026)
- [Text-to-Audio Grounding: Building Correspondence Between Captions and Sound Events](https://mlsanity.com/paper/2102.11474/) (2021)
- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments](https://mlsanity.com/paper/1807.10501/) (2018)

## Abstract (arXiv)

Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection (SED) attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. Progress is bottlenecked by data scarcity: no large-scale resource provides open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry placement-exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields large temporal-localization gains on AEGBench, an independent difficulty-stratified benchmark whose annotations we release, and these gains generalize to various held-out SED and audio grounding benchmarks. Our results show that automatically constructed data, coupled with interval-aware reward design, provides an effective data-side route to expanding the temporal localization capability of LALMs.