# FlexSED: Towards Open-Vocabulary Sound Event Detection

arXiv: https://arxiv.org/abs/2509.18606 · HTML: https://mlsanity.com/paper/2509.18606/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [The Benefit Of Temporally-Strong Labels In Audio Event Classification](https://mlsanity.com/paper/2105.07031/) (2021)
- [TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization](https://mlsanity.com/paper/2412.21037/) (2024)
- [Sound Event Detection: A Tutorial](https://mlsanity.com/paper/2107.05463/) (2021)
- [Scaling up masked audio encoder learning for general audio classification](https://mlsanity.com/paper/2406.06992/) (2024)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer](https://mlsanity.com/paper/2409.10819/) (2024)
- [A Framework for the Robust Evaluation of Sound Event Detection](https://mlsanity.com/paper/1910.08440/) (2019)
- [Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks](https://mlsanity.com/paper/2306.04186/) (2023)
- [Frequency Dynamic Convolution: Frequency-Adaptive Pattern Recognition for Sound Event Detection](https://mlsanity.com/paper/2203.15296/) (2022)
- [Fine-tune the pretrained ATST model for sound event detection](https://mlsanity.com/paper/2309.08153/) (2023)
- [Effective Pre-Training of Audio Transformers for Sound Event Detection](https://mlsanity.com/paper/2409.09546/) (2024)
- [SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer](https://mlsanity.com/paper/2409.08425/) (2024)
- [Threshold Independent Evaluation of Sound Event Detection Scores](https://mlsanity.com/paper/2201.13148/) (2022)
- [FilterAugment: An Acoustic Environmental Data Augmentation Method](https://mlsanity.com/paper/2110.03282/) (2021)
- [Exploring Text-Queried Sound Event Detection with Audio Source Separation](https://mlsanity.com/paper/2409.13292/) (2024)

## Built on by (most cited first, 4 of 4)

- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [Talking with Verifiers: Automatic Specification Generation for Neural Network Verification](https://mlsanity.com/paper/2603.02235/) (2026)
- [Towards Open World Sound Event Detection](https://mlsanity.com/paper/2605.03934/) (2026)
- [VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement](https://mlsanity.com/paper/2609.03153/) (2026)

## Abstract (arXiv)

Despite recent progress in large-scale sound event detection (SED) systems capable of handling hundreds of sound classes, existing multi-class classification frameworks remain fundamentally limited. They cannot process free-text sound queries, which enable more flexible and user-friendly interaction, and they lack zero-shot capabilities and offer poor few-shot adaptability. Although text-query-based separation methods have been explored, they primarily focus on source separation and are ill-suited for SED tasks that require precise temporal localization and efficient detection across large and diverse sound vocabularies. In this paper, we propose FlexSED, an open-vocabulary sound event detection system. FlexSED builds on a pretrained audio SSL model and the CLAP text encoder, introducing an encoder-decoder composition and an adaptive fusion strategy to enable effective continuous training from pretrained weights. To ensure robust supervision, it also employs large language models (LLMs) to assist in event query selection during training, addressing challenges related to missing labels. As a result, FlexSED achieves superior performance compared to vanilla SED models on AudioSet-Strong, while demonstrating strong zero-shot and few-shot capabilities. We release the code and pretrained models to support future research and applications based on FlexSED.