# Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers

arXiv: https://arxiv.org/abs/2208.11402 · HTML: https://mlsanity.com/paper/2208.11402/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Zero-Shot Audio Classification via Semantic Embeddings](https://mlsanity.com/paper/2011.12133/) (2020)
- [Zero-shot Learning for Audio-based Music Classification and Tagging](https://mlsanity.com/paper/1907.02670/) (2019)
- [Zero-Shot Audio Classification Based on Class Label Embeddings](https://mlsanity.com/paper/1905.01926/) (2019)
- [Zero-Shot Audio Classification with Factored Linear and Nonlinear Acoustic-Semantic Projections](https://mlsanity.com/paper/2011.12657/) (2020)

## Built on by (most cited first, 2 of 2)

- [Multi-label Zero-Shot Audio Classification with Temporal Attention](https://mlsanity.com/paper/2409.00408/) (2024)
- [Generalized zero-shot audio-to-intent classification](https://mlsanity.com/paper/2311.02482/) (2023)

## Abstract (arXiv)

Standard machine learning models for tagging and classifying acoustic signals cannot handle classes that were not seen during training. Zero-Shot (ZS) learning overcomes this restriction by predicting classes based on adaptable class descriptions. This study sets out to investigate the effectiveness of self-attention-based audio embedding architectures for ZS learning. To this end, we compare the very recent patchout spectrogram transformer with two classic convolutional architectures. We evaluate these three architectures on three tasks and on three different benchmark datasets: general-purpose tagging on AudioSet, environmental sound classification on ESC-50, and instrument tagging on OpenMIC. Our results show that the self-attention-based embedding methods outperform both compared convolutional architectures in all of these settings. By designing training and test data accordingly, we observe that prediction performance suffers significantly when the `semantic distance' between training and new test classes is large, an effect that will deserve more detailed investigations.