mlsanityData to 7 Oct 2026

Towards Learning a Universal Non-Semantic Representation of Speech

arXiv 2002.12764 ↗First posted 25 Feb 2020eess.AS cs.LG cs.SDcited by 84

Not on the map. Why?

Abstract (arXiv)

The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks. The visual and language communities have established benchmarks to compare embeddings, but the speech community has yet to do so. This paper proposes a benchmark for comparing speech representations on non-semantic tasks, and proposes a representation based on an unsupervised triplet-loss objective. The proposed representation outperforms other representations on the benchmark, and even exceeds state-of-the-art performance on a number of transfer learning tasks. The embedding is trained on a publicly available dataset, and it is tested on a variety of low-resource downstream tasks, including personalization tasks and medical domain. The benchmark, models, and evaluation code are publicly released.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 22; 2023: 25; 2024: 9; 2025: 5; 2026: 3.

84 papers.

Sort:

By area:

  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  2. 2023MusicLM: Generating Music From Textcited by 442
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  5. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  6. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  7. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  8. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  9. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  10. 2022Audio Self-supervised Learning: A Surveycited by 65
  11. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  12. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
38 more
  1. 2023Transformers in Speech Processing: A Surveycited by 38
  2. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  3. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  4. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  5. 2021Learning Efficient Representations for Keyword Spotting with Triplet Losscited by 24
  6. 2021SERAB: A multi-lingual benchmark for speech emotion recognitioncited by 22
  7. 2021VoxCeleb Enrichment for Age and Gender Recognitioncited by 20
  8. 2023V2Meow: Meowing to the Visual Beat via Video-to-Music Generationcited by 18
  9. 2022TRILLsson: Distilled Universal Paralinguistic Speech Representationscited by 16
  10. 2024HeAR -- Health Acoustic Representationscited by 15
  11. 2021Detecting Dementia from Speech and Transcripts using Transformerscited by 13
  12. 2022ATST: Audio Representation Learning with Teacher-Student Transformercited by 13
  13. 2022A Pre-trained Audio-Visual Transformer for Emotion Recognitioncited by 12
  14. 2022Bias and Fairness on Multimodal Emotion Detection Algorithmscited by 10
  15. 2021Teaching keyword spotters to spot new keywords with limited examplescited by 9
  16. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 9
  17. 2023SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?cited by 9
  18. 2023EnCodecMAE: Leveraging neural codecs for universal audio representation learningcited by 9
  19. 2026Massive Sound Embedding Benchmark (MSEB)cited by 9
  20. 2021Learning De-identified Representations of Prosody from Raw Audiocited by 8
  21. 2023Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settingscited by 8
  22. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  23. 2021DECAR: Deep Clustering for learning general-purpose Audio Representationscited by 7
  24. 2021Self-Supervised Learning from Automatically Separated Sound Scenescited by 6
  25. 2022Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Loadcited by 6
  26. 2022Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learningcited by 6
  27. 2023Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAPcited by 6
  28. 2021Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrasescited by 5
  29. 2022Federated Self-Supervised Learning for Acoustic Event Classificationcited by 5
  30. 2023Performance Conditioning for Diffusion-Based Multi-Instrument Music Synthesiscited by 5
  31. 2021Scenario Aware Speech Recognition: Advancements for Apollo Fearless Steps & CHiME-4 Corporacited by 4
  32. 2021Transfer Learning with Jukebox for Music Source Separationcited by 4
  33. 2022Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learningcited by 4
  34. 2023Label Aware Speech Representation Learning For Language Identificationcited by 4
  35. 2023A Digital Language Coherence Marker for Monitoring Dementiacited by 4
  36. 2023Reformulating NLP tasks to Capture Longitudinal Manifestation of Language Disorders in People with Dementiacited by 4
  37. 2024STAB: Speech Tokenizer Assessment Benchmarkcited by 4
  38. 2024The Unreliability of Acoustic Systems in Alzheimer's Speech Datasets with Heterogeneous Recording Conditionscited by 4

50 most cited of 84 All 84 (JSON)