# FlexiAST: Flexibility is What AST Needs

arXiv: https://arxiv.org/abs/2307.09286 · HTML: https://mlsanity.com/paper/2307.09286/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [Learning from Between-class Examples for Deep Sound Recognition](https://mlsanity.com/paper/1711.10282/) (2017)
- [Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2204.12260/) (2022)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Multiscale Audio Spectrogram Transformer for Efficient Audio Classification](https://mlsanity.com/paper/2303.10757/) (2023)

## Built on by (most cited first, 4 of 4)

- [ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions](https://mlsanity.com/paper/2407.08691/) (2024)
- [From Coarse to Fine: Efficient Training for Audio Spectrogram Transformers](https://mlsanity.com/paper/2401.08415/) (2024)
- [Mixer is more than just a model](https://mlsanity.com/paper/2402.18007/) (2024)
- [Geometry-Conditioned Fixed-Scaffold Encoders for Time-Warp Robust Sequence Retrieval](https://mlsanity.com/paper/2609.36809/) (2026)

## Abstract (arXiv)

The objective of this work is to give patch-size flexibility to Audio Spectrogram Transformers (AST). Recent advancements in ASTs have shown superior performance in various audio-based tasks. However, the performance of standard ASTs degrades drastically when evaluated using different patch sizes from that used during training. As a result, AST models are typically re-trained to accommodate changes in patch sizes. To overcome this limitation, this paper proposes a training procedure to provide flexibility to standard AST models without architectural changes, allowing them to work with various patch sizes at the inference stage - FlexiAST. This proposed training approach simply utilizes random patch size selection and resizing of patch and positional embedding weights. Our experiments show that FlexiAST gives similar performance to standard AST models while maintaining its evaluation ability at various patch sizes on different datasets for audio classification tasks.