# MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection

arXiv: https://arxiv.org/abs/2408.08673 · HTML: https://mlsanity.com/paper/2408.08673/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection](https://mlsanity.com/paper/1702.06286/) (2017)
- [A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling](https://mlsanity.com/paper/1810.09050/) (2018)
- [A Framework for the Robust Evaluation of Sound Event Detection](https://mlsanity.com/paper/1910.08440/) (2019)
- [Fine-tune the pretrained ATST model for sound event detection](https://mlsanity.com/paper/2309.08153/) (2023)
- [FilterAugment: An Acoustic Environmental Data Augmentation Method](https://mlsanity.com/paper/2110.03282/) (2021)
- [Semi-supervsied Learning-based Sound Event Detection using Freuqency Dynamic Convolution with Large Kernel Attention for DCASE Challenge 2023 Task 4](https://mlsanity.com/paper/2306.06461/) (2023)

## Built on by (most cited first, 7 of 7)

- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [Prototype based Masked Audio Model for Self-Supervised Learning of Sound Event Detection](https://mlsanity.com/paper/2409.17656/) (2024)
- [RealDESED: A Real-World Domestic Sound Event Detection Benchmark](https://mlsanity.com/paper/2607.16736/) (2026)
- [Exploring Performance-Complexity Trade-Offs in Sound Event Detection Models](https://mlsanity.com/paper/2503.11373/) (2025)
- [Revisiting SSL for sound event detection: complementary fusion and adaptive post-processing](https://mlsanity.com/paper/2505.11889/) (2025)
- [DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams](https://mlsanity.com/paper/2511.17693/) (2025)
- [Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Loss](https://mlsanity.com/paper/2606.29901/) (2026)

## Abstract (arXiv)

Sound event detection (SED) methods that leverage a large pre-trained Transformer encoder network have shown promising performance in recent DCASE challenges. However, they still rely on an RNN-based context network to model temporal dependencies, largely due to the scarcity of labeled data. In this work, we propose a pure Transformer-based SED model with masked-reconstruction based pre-training, termed MAT-SED. Specifically, a Transformer with relative positional encoding is first designed as the context network, pre-trained by the masked-reconstruction task on all available target data in a self-supervised way. Both the encoder and the context network are jointly fine-tuned in a semi-supervised manner. Furthermore, a global-local feature fusion strategy is proposed to enhance the localization capability. Evaluation of MAT-SED on DCASE2023 task4 surpasses state-of-the-art performance, achieving 0.587/0.896 PSDS1/PSDS2 respectively.