mlsanityData to 14 Oct 2026

Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders

arXiv 1910.12638 ↗First posted 25 Oct 2019eess.AS cs.CL cs.LGcited by 198

Not on the map. Why?

Abstract (arXiv)

We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.

Key facts. Cited by 198 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 55; 2022: 65; 2023: 44; 2024: 23; 2025: 6; 2026: 5.

198 papers.

Sort:

By area:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  7. 2022Masked Autoencoders that Listencited by 273
  8. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  9. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  10. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2022Generative Spoken Dialogue Language Modelingcited by 126
38 more
  1. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  2. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  3. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  4. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  5. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  6. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  7. 2023BrainBERT: Self-supervised representation learning for intracranial recordingscited by 72
  8. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  9. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  10. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 65
  11. 2022Audio Self-supervised Learning: A Surveycited by 65
  12. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  13. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  14. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 53
  15. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  16. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  17. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  18. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  19. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 45
  20. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  21. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  22. 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
  23. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  24. 2023Transformers in Speech Processing: A Surveycited by 39
  25. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  26. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  27. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  28. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 35
  29. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  30. 2022Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learningcited by 29
  31. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  32. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  33. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  34. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  35. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  36. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  37. 2021Don't speak too fast: The impact of data bias on self-supervised speech modelscited by 20
  38. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19

50 most cited of 198 All 198 (JSON)