mlsanityData to 14 Oct 2026

On the Utility of Self-supervised Models for Prosody-related Tasks

arXiv 2210.07185 ↗First posted 13 Oct 2022cs.CL eess.AScited by 42

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) from speech data has produced models that have achieved remarkable performance in many tasks, and that are known to implicitly represent many aspects of information latently present in speech signals. However, relatively little is known about the suitability of such models for prosody-related tasks or the extent to which they encode prosodic information. We present a new evaluation framework, SUPERB-prosody, consisting of three prosody-related downstream tasks and two pseudo tasks. We find that 13 of the 15 SSL models outperformed the baseline on all the prosody-related tasks. We also show good performance on two pseudo tasks: prosody reconstruction and future prosody prediction. We further analyze the layerwise contributions of the SSL models. Overall we conclude that SSL speech models are highly effective for prosody-related tasks.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

26 of 38 references.

References · 26

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  8. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  9. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  11. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
14 more
  1. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  2. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  3. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 174
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  5. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  6. 2020Unsupervised pretraining transfers well across languagescited by 101
  7. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  8. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  9. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  10. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  11. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  12. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  13. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  14. 2021Learning De-identified Representations of Prosody from Raw Audiocited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 17; 2025: 9; 2026: 4.

42 papers.

Sort:

By area:

  1. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  2. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  3. 2023Voice Conversion With Just Nearest Neighborscited by 68
  4. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  5. 2023ProsAudit, a prosodic benchmark for self-supervised speech modelscited by 25
  6. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  7. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 23
  8. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 22
  9. 2023What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Modelcited by 18
  10. 2024A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech modelscited by 18
  11. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  12. 2024Estimating the Completeness of Discrete Speech Unitscited by 9
30 more
  1. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
  2. 2024SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detectioncited by 8
  3. 2023Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Targetcited by 6
  4. 2025The Role of Prosody in Spoken Question Answeringcited by 6
  5. 2024Property Neurons in Self-Supervised Speech Transformerscited by 5
  6. 2023Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognitioncited by 4
  7. 2022Introducing Semantics into Speech Encoderscited by 3
  8. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  9. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  10. 2024A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterancescited by 2
  11. 2024Joint Learning of Context and Feedback Embeddings in Spoken Dialoguecited by 2
  12. 2024WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Modelcited by 2
  13. 2024Which Prosodic Features Matter Most for Pragmatics?cited by 2
  14. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2
  15. 2025DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matchingcited by 2
  16. 2025Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokenscited by 2
  17. 2023Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studiescited by 1
  18. 2024An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesiscited by 1
  19. 2024Improving Voice Quality in Speech Anonymization With Just Perception-Informed Lossescited by 1
  20. 2024QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversioncited by 1
  21. 2025Representation of perceived prosodic similarity of conversational feedbackcited by 1
  22. 2025Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affectcited by 1
  23. 2025TAVID: Text-Driven Audio-Visual Interactive Dialogue Generationcited by 1
  24. 2024Exploring Acoustic Similarity in Emotional Speech and Music via Self-Supervised Representationscited by 0
  25. 2024An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutionscited by 0
  26. 2025Enhancing and Exploring Mild Cognitive Impairment Detection with W2V-BERT-2.0cited by 0
  27. 2025Emphasis Sensitivity in Speech Representationscited by 0
  28. 2026Self-Supervised Speech Representations Track Spoken Language Convergence to Adult Models in Infants and Children Who Are Deaf/Hard-of-Hearingcited by 0
  29. 2026When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Modelscited by 0
  30. 2026Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERTcited by 0