# Audio Transformers

arXiv: https://arxiv.org/abs/2105.00335 · HTML: https://mlsanity.com/paper/2105.00335/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [Language Through a Prism: A Spectral Approach for Multiscale Language Representations](https://mlsanity.com/paper/2011.04823/) (2020)
- [Neural Style Transfer for Audio Spectograms](https://mlsanity.com/paper/1801.01589/) (2018)
- [Audio-Linguistic Embeddings for Spoken Sentences](https://mlsanity.com/paper/1902.07817/) (2019)
- [A Framework for Generative and Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2010.11459/) (2020)
- [Conditional End-to-End Audio Transforms](https://mlsanity.com/paper/1804.00047/) (2018)
- [Neuralogram: A Deep Neural Network Based Representation for Audio Signals](https://mlsanity.com/paper/1904.05073/) (2019)

## Built on by (most cited first, 30 of 41)

- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Localizing Moments in Long Video Via Multimodal Guidance](https://mlsanity.com/paper/2302.13372/) (2023)
- [A Generative Model for Raw Audio Using Transformer Architectures](https://mlsanity.com/paper/2106.16036/) (2021)
- [It's Time for Artistic Correspondence in Music and Video](https://mlsanity.com/paper/2206.07148/) (2022)
- [The iNaturalist Sounds Dataset](https://mlsanity.com/paper/2506.00343/) (2025)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)
- [ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers](https://mlsanity.com/paper/2307.03493/) (2023)
- [SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction](https://mlsanity.com/paper/2309.02320/) (2023)
- [Towards Signal Processing In Large Language Models](https://mlsanity.com/paper/2406.10254/) (2024)
- [Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks](https://mlsanity.com/paper/2107.00623/) (2021)
- [ATGNN: Audio Tagging Graph Neural Network](https://mlsanity.com/paper/2311.01526/) (2023)
- [Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification](https://mlsanity.com/paper/2401.04023/) (2024)
- [LEAN: Light and Efficient Audio Classification Network](https://mlsanity.com/paper/2305.12712/) (2023)
- [Generating Coherent Drum Accompaniment With Fills And Improvisations](https://mlsanity.com/paper/2209.00291/) (2022)
- [A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization](https://mlsanity.com/paper/2310.15177/) (2023)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music](https://mlsanity.com/paper/2412.11449/) (2024)
- [Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....](https://mlsanity.com/paper/2110.03183/) (2021)
- [Study of positional encoding approaches for Audio Spectrogram Transformers](https://mlsanity.com/paper/2110.06999/) (2021)
- [Video Pre-trained Transformer: A Multimodal Mixture of Pre-trained Experts](https://mlsanity.com/paper/2304.10505/) (2023)
- [Conformer LLMs -- Convolution Augmented Large Language Models](https://mlsanity.com/paper/2307.00461/) (2023)
- [Compositional Structures in Neural Embedding and Interaction Decompositions](https://mlsanity.com/paper/2407.08934/) (2024)
- [Wavelet GPT: Wavelet Inspired Large Language Models](https://mlsanity.com/paper/2409.12924/) (2024)
- [Large Language Models Implicitly Learn to See and Hear Just By Reading](https://mlsanity.com/paper/2505.17091/) (2025)
- [Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems](https://mlsanity.com/paper/2509.15448/) (2025)
- [From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification](https://mlsanity.com/paper/2509.21341/) (2025)
- [Enhancing Audio Perception of Music By AI Picked Room Acoustics](https://mlsanity.com/paper/2208.07994/) (2022)
- [H4VDM: H.264 Video Device Matching](https://mlsanity.com/paper/2210.11549/) (2022)
- [One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall](https://mlsanity.com/paper/2210.15750/) (2022)

## Abstract (arXiv)

Over the past two decades, CNN architectures have produced compelling models of sound perception and cognition, learning hierarchical organizations of features. Analogous to successes in computer vision, audio feature classification can be optimized for a particular task of interest, over a wide variety of datasets and labels. In fact similar architectures designed for image understanding have proven effective for acoustic scene analysis. Here we propose applying Transformer based architectures without convolutional layers to raw audio signals. On a standard dataset of Free Sound 50K,comprising of 200 categories, our model outperforms convolutional models to produce state of the art results. This is significant as unlike in natural language processing and computer vision, we do not perform unsupervised pre-training for outperforming convolutional architectures. On the same training set, with respect mean aver-age precision benchmarks, we show a significant improvement. We further improve the performance of Transformer architectures by using techniques such as pooling inspired from convolutional net-work designed in the past few years. In addition, we also show how multi-rate signal processing ideas inspired from wavelets, can be applied to the Transformer embeddings to improve the results. We also show how our models learns a non-linear non constant band-width filter-bank, which shows an adaptable time frequency front end representation for the task of audio understanding, different from other tasks e.g. pitch estimation.