mlsanityData to 14 Oct 2026

HEAR: Holistic Evaluation of Audio Representations

arXiv 2203.03022 ↗First posted 6 Mar 2022cs.SD cs.AI cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each participant submitted an audio embedding model following a common API that is general-purpose, open-source, and freely available to use. Twenty-nine models by thirteen external teams were evaluated on nineteen diverse downstream tasks derived from sixteen datasets. Open evaluation code, submitted models and datasets are key contributions, enabling comprehensive and reproducible evaluation, as well as previously impossible longitudinal studies. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 64 of its 99 references that are in the map. First posted on arXiv 6 Mar 2022.

Builds on

64 of 99 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
38 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  7. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  10. 2021AST: Audio Spectrogram Transformercited by 643
  11. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  13. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  14. 2017Look, Listen and Learncited by 458
  15. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  16. 2021Pay Attention to MLPscited by 368
  17. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  18. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  19. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  20. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  21. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  22. 2018Efficient Neural Audio Synthesiscited by 219
  23. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
  24. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  25. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  26. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  27. 2020DDSP: Differentiable Digital Signal Processingcited by 178
  28. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  29. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 122
  30. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  31. 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 110
  32. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  33. 2017Onsets and Frames: Dual-Objective Piano Transcriptioncited by 94
  34. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  35. 2016Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classificationcited by 85
  36. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  37. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  38. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 17; 2024: 18; 2025: 25; 2026: 12.

78 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  3. 2023Joint Audio and Speech Understandingcited by 100
  4. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  5. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  6. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  7. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  8. 2024Foundation Models for Music: A Surveycited by 37
  9. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  10. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  11. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  12. 2025Perch 2.0: The Bittern Lesson for Bioacousticscited by 26
38 more
  1. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  2. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  3. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  4. 2023BIRB: A Generalization Benchmark for Information Retrieval in Bioacousticscited by 19
  5. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  6. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  7. 2024Benchmarking Representations for Speech, Music, and Acoustic Eventscited by 15
  8. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11
  9. 2025BLAB: Brutally Long Audio Benchcited by 11
  10. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 10
  11. 2023EnCodecMAE: Leveraging neural codecs for universal audio representation learningcited by 9
  12. 2024Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimationcited by 8
  13. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Modelscited by 8
  14. 2025Towards a Unified Representation Evaluation Framework Beyond Downstream Taskscited by 7
  15. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  16. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  17. 2022The Efficacy of Self-Supervised Speech Models for Audio Representationscited by 6
  18. 2023Advancing Natural-Language Based Audio Retrieval with PaSST and Large Audio-Caption Data Setscited by 6
  19. 2022Audio Barlow Twins: Self-Supervised Audio Representation Learningcited by 5
  20. 2023On the Effectiveness of Speech Self-supervised Learning for Musiccited by 5
  21. 2025AVEX: What Matters for Animal Vocalization Encodingcited by 5
  22. 2022Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformerscited by 4
  23. 2024Enhanced Multimodal Content Moderation of Children's Videos using Audiovisual Fusioncited by 4
  24. 2025Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learningcited by 4
  25. 2025TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Modelscited by 4
  26. 2025Revisiting Audio-language Pretraining for Learning General-purpose Audio Representationcited by 4
  27. 2023A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognitioncited by 3
  28. 2024Pre-training with Synthetic Patterns for Audiocited by 3
  29. 2025Universal Music Representations? Evaluating Foundation Models on World Music Corporacited by 3
  30. 2022Efficient Speech Quality Assessment using Self-supervised Framewise Embeddingscited by 2
  31. 2024Predicting Heart Activity from Speech using Data-driven and Knowledge-based featurescited by 2
  32. 2024Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrievalcited by 2
  33. 2024Acoustic identification of individual animals with hierarchical contrastive learningcited by 2
  34. 2024Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Featurescited by 2
  35. 2025FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representationcited by 2
  36. 2026Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddingscited by 2
  37. 2023On the choice of the optimal temporal support for audio classification with Pre-trained embeddingscited by 1
  38. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1

50 most cited of 78 All 78 (JSON)