# ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions

arXiv: https://arxiv.org/abs/2407.08691 · HTML: https://mlsanity.com/paper/2407.08691/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 33 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Epic-Sounds: A Large-scale Dataset of Actions That Sound](https://mlsanity.com/paper/2302.00646/) (2023)
- [Vision Transformers with Mixed-Resolution Tokenization](https://mlsanity.com/paper/2304.00287/) (2023)
- [Simple Pooling Front-ends For Efficient Audio Classification](https://mlsanity.com/paper/2210.00943/) (2022)
- [FlexiAST: Flexibility is What AST Needs](https://mlsanity.com/paper/2307.09286/) (2023)

## Built on by (most cited first, 2 of 2)

- [SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training](https://mlsanity.com/paper/2601.12594/) (2026)
- [The Hitchhiker's Guide to Agentic AI: From Foundations to Systems](https://mlsanity.com/paper/2606.24937/) (2026)

## Abstract (arXiv)

Transformers have rapidly overtaken CNN-based architectures as the new standard in audio classification. Transformer-based models, such as the Audio Spectrogram Transformers (AST), also inherit the fixed-size input paradigm from CNNs. However, this leads to performance degradation for ASTs in the inference when input lengths vary from the training. This paper introduces an approach that enables the use of variable-length audio inputs with AST models during both training and inference. By employing sequence packing, our method ElasticAST, accommodates any audio length during training, thereby offering flexibility across all lengths and resolutions at the inference. This flexibility allows ElasticAST to maintain evaluation capabilities at various lengths or resolutions and achieve similar performance to standard ASTs trained at specific lengths or resolutions. Moreover, experiments demonstrate ElasticAST's better performance when trained and evaluated on native-length audio datasets.