# Improving Audio Spectrogram Transformers for Sound Event Detection Through Multi-Stage Training

arXiv: https://arxiv.org/abs/2408.00791 · HTML: https://mlsanity.com/paper/2408.00791/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2024.

## Built on by (most cited first, 3 of 3)

- [Prototype based Masked Audio Model for Self-Supervised Learning of Sound Event Detection](https://mlsanity.com/paper/2409.17656/) (2024)
- [CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries](https://mlsanity.com/paper/2511.15131/) (2025)
- [Spectrogram features for audio and speech analysis](https://mlsanity.com/paper/2603.14917/) (2026)

## Abstract (arXiv)

This technical report describes the CP-JKU team's submission for Task 4 Sound Event Detection with Heterogeneous Training Datasets and Potentially Missing Labels of the DCASE 24 Challenge. We fine-tune three large Audio Spectrogram Transformers, PaSST, BEATs, and ATST, on the joint DESED and MAESTRO datasets in a two-stage training procedure. The first stage closely matches the baseline system setup and trains a CRNN model while keeping the large pre-trained transformer model frozen. In the second stage, both CRNN and transformer are fine-tuned using heavily weighted self-supervised losses. After the second stage, we compute strong pseudo-labels for all audio clips in the training set using an ensemble of all three fine-tuned transformers. Then, in a second iteration, we repeat the two-stage training process and include a distillation loss based on the pseudo-labels, boosting single-model performance substantially. Additionally, we pre-train PaSST and ATST on the subset of AudioSet that comes with strong temporal labels, before fine-tuning them on the Task 4 datasets.