mlsanityData to 7 Oct 2026

Masked Autoencoders that Listen

arXiv 2207.06405 ↗First posted 13 Jul 2022cs.SD cs.AI cs.LGcited by 271

Not on the map. Why?

Abstract (arXiv)

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Builds on 53 of its 83 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

53 of 83 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  12. 2019When Does Label Smoothing Help?cited by 985
  13. 2016Deep Networks with Stochastic Depthcited by 973
  14. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  15. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  16. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  17. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  18. 2021AST: Audio Spectrogram Transformercited by 634
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  20. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  22. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  23. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  24. 2017Objects that Soundcited by 254
  25. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  26. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  27. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  28. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  29. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  30. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  31. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  32. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  33. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  34. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  35. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  36. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  37. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  38. 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 59; 2024: 72; 2025: 76; 2026: 44.

271 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  6. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  7. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  8. 2022AudioGen: Textually Guided Audio Generationcited by 292
  9. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  10. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  11. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  12. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
38 more
  1. 20234M: Massively Multimodal Masked Modelingcited by 118
  2. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  3. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  4. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  5. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  6. 2022MAEEG: Masked Auto-encoder for EEG Representation Learningcited by 61
  7. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  8. 2024Scaling Wearable Foundation Modelscited by 57
  9. 2022MAViL: Masked Audio-Video Learnerscited by 56
  10. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generationcited by 54
  11. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  12. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  13. 2024Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biologycited by 52
  14. 2023Rethinking Tokenizer and Decoder in Masked Graph Modeling for Moleculescited by 49
  15. 2022Audiovisual Masked Autoencoderscited by 47
  16. 2025Scaling Laws for Native Multimodal Modelscited by 47
  17. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  18. 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 42
  19. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  20. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  21. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  22. 2023TabMT: Generating tabular data with masked transformerscited by 36
  23. 2024BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacousticscited by 36
  24. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 36
  25. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  26. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  27. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 34
  28. 2024Foundation Models for Music: A Surveycited by 34
  29. 2023Audio-Visual Class-Incremental Learningcited by 33
  30. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  31. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
  32. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  33. 2023MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Trainingcited by 26
  34. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  35. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 25
  36. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  37. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  38. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generationcited by 24

50 most cited of 271 All 271 (JSON)