mlsanityData to 14 Oct 2026

Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners

arXiv 2306.00561 ↗First posted 1 Jun 2023cs.SD cs.AI eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy. Code for feature extraction and downstream experiments along with pre-trained models will be released publically.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

57 of 74 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2021Multiscale Vision Transformerscited by 897
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  12. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  13. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  14. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  15. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 448
  16. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  17. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  18. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  19. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  20. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  21. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  22. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  23. 2022Masked Autoencoders that Listencited by 273
  24. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  25. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  26. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  27. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  28. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  29. 2022Masked World Models for Visual Controlcited by 170
  30. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  31. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  32. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  33. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  34. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  35. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  36. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  37. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  38. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 1.

11 papers.

  1. 2024Foundation Models for Music: A Surveycited by 37
  2. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  3. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  4. 2024How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matchingcited by 5
  5. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  6. 2025Structured-Noise Masked Modeling for Video, Audio and Beyondcited by 1
  7. 2025Text-Queried Audio Source Separation via Hierarchical Modelingcited by 0
  8. 2025GRAM: Spatial general-purpose audio representation models for real-world applicationscited by 0
  9. 2025AudioMAE++: learning better masked audio representations with SwiGLU FFNscited by 0
  10. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  11. 2026Neonatal Hypoxic-ischaemic Encephalopathy Classification from the EEG and HRV Signals Using a Conformer based Masked Autoencodercited by 0