# MAST: Multiscale Audio Spectrogram Transformers

arXiv: https://arxiv.org/abs/2211.01515 · HTML: https://mlsanity.com/paper/2211.01515/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [Towards Multi-Scale Style Control for Expressive Speech Synthesis](https://mlsanity.com/paper/2104.03521/) (2021)
- [Learning Multiscale Features Directly From Waveforms](https://mlsanity.com/paper/1603.09509/) (2016)
- [DECAR: Deep Clustering for learning general-purpose Audio Representations](https://mlsanity.com/paper/2110.08895/) (2021)
- [Y-Vector: Multiscale Waveform Encoder for Speaker Embedding](https://mlsanity.com/paper/2010.12951/) (2020)

## Built on by (most cited first, 3 of 3)

- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [On the Effect of Data-Augmentation on Local Embedding Properties in the Contrastive Learning of Music Audio Representations](https://mlsanity.com/paper/2401.08889/) (2024)
- [AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformers](https://mlsanity.com/paper/2512.03637/) (2025)

## Abstract (arXiv)

We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram, we first patchify and project it into an initial temporal resolution and embedding dimension, post which the multiple stages in MAST progressively expand the embedding dimension while reducing the temporal resolution of the input. We use a pyramid structure that allows early layers of MAST operating at a high temporal resolution but low embedding space to model simple low-level acoustic information and deeper temporally coarse layers to model high-level acoustic information with high-dimensional embeddings. We also extend our approach to present a new Self-Supervised Learning (SSL) method called SS-MAST, which calculates a symmetric contrastive loss between latent representations from a student and a teacher encoder, leveraging patch-drop, a novel audio augmentation approach that we introduce. In practice, MAST significantly outperforms AST by an average accuracy of 3.4% across 8 speech and non-speech tasks from the LAPE Benchmark, achieving state-of-the-art results on keyword spotting in Speech Commands. Additionally, our proposed SS-MAST achieves an absolute average improvement of 2.6% over the previously proposed SSAST.