mlsanityData to 7 Oct 2026

FlexiAST: Flexibility is What AST Needs

arXiv 2307.09286 ↗First posted 18 Jul 2023cs.SD cs.LG eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The objective of this work is to give patch-size flexibility to Audio Spectrogram Transformers (AST). Recent advancements in ASTs have shown superior performance in various audio-based tasks. However, the performance of standard ASTs degrades drastically when evaluated using different patch sizes from that used during training. As a result, AST models are typically re-trained to accommodate changes in patch sizes. To overcome this limitation, this paper proposes a training procedure to provide flexibility to standard AST models without architectural changes, allowing them to work with various patch sizes at the inference stage - FlexiAST. This proposed training approach simply utilizes random patch size selection and resizing of patch and positional embedding weights. Our experiments show that FlexiAST gives similar performance to standard AST models while maintaining its evaluation ability at various patch sizes on different datasets for audio classification tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

28 of 33 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2021Multiscale Vision Transformerscited by 896
16 more
  1. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  2. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  3. 2021AST: Audio Spectrogram Transformercited by 634
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  6. 2018Video Action Transformer Networkcited by 316
  7. 2022Masked Autoencoders that Listencited by 271
  8. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  9. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  10. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  11. 2022FlexiViT: One Model for All Patch Sizescited by 151
  12. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  13. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 53
  14. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  15. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  16. 2023Multiscale Audio Spectrogram Transformer for Efficient Audio Classificationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutionscited by 2
  2. 2024From Coarse to Fine: Efficient Training for Audio Spectrogram Transformerscited by 1
  3. 2024Mixer is more than just a modelcited by 0
  4. 2026Geometry-Conditioned Fixed-Scaffold Encoders for Time-Warp Robust Sequence Retrievalcited by 0