mlsanityData to 14 Oct 2026

BEATs: Audio Pre-Training with Acoustic Tokenizers

arXiv 2212.09058 ↗First posted 18 Dec 2022eess.AS cs.AI cs.CLcited by 388

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

Key facts. Cited by 388 papers in the map (data through 14 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 18 Dec 2022.

Builds on

56 of 72 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2017Neural Discrete Representation Learningcited by 4,712
38 more
  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  12. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  13. 2021AST: Audio Spectrogram Transformercited by 643
  14. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  15. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  16. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  17. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  18. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  19. 2022Masked Autoencoders that Listencited by 273
  20. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  21. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  22. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  23. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  24. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  25. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  26. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  27. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  28. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  29. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  30. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  31. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  32. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  33. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  34. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  35. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  36. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  37. 2021Interspeech 2021 Deep Noise Suppression Challengecited by 47
  38. 2021Multi-Format Contrastive Learning of Audio Representationscited by 39

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 91; 2025: 137; 2026: 130.

388 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  3. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  7. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  8. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  9. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  10. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  11. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  12. 2025Step-Audio 2 Technical Reportcited by 105
38 more
  1. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  2. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  3. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  4. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  6. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  7. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  8. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  9. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 69
  10. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  11. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  12. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  13. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  14. 2024Gramian Multimodal Representation Learning and Alignmentcited by 54
  15. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  16. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  17. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  18. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  19. 2024Foundation Models for Music: A Surveycited by 37
  20. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  21. 2024BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacousticscited by 36
  22. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 33
  23. 2023Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semanticscited by 32
  24. 2023BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledgecited by 32
  25. 2024Audio-Synchronized Visual Animationcited by 32
  26. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  27. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  28. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  29. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  30. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  31. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  32. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  33. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  34. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  35. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 22
  36. 2023Audio-Visual Instance Segmentationcited by 21
  37. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 21
  38. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 20

50 most cited of 388 All 388 (JSON)