mlsanityData to 14 Oct 2026

A Quantitative Approach to Understand Self-Supervised Models as Cross-lingual Feature Extractors

arXiv 2311.15954 ↗First posted 27 Nov 2023cs.CL eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

In this work, we study the features extracted by English self-supervised learning (SSL) models in cross-lingual contexts and propose a new metric to predict the quality of feature representations. Using automatic speech recognition (ASR) as a downstream task, we analyze the effect of model size, training objectives, and model architecture on the models' performance as a feature extractor for a set of topologically diverse corpora. We develop a novel metric, the Phonetic-Syntax Ratio (PSR), to measure the phonetic and synthetic information in the extracted representations using deep generalized canonical correlation analysis. Results show the contrastive loss in the wav2vec2.0 objective facilitates more effective cross-lingual feature extraction. There is a positive correlation between PSR scores and ASR performance, suggesting that phonetic information extracted by monolingual SSL models can be used for downstream tasks in cross-lingual settings. The proposed metric is an effective indicator of the quality of the representations and can be useful for model selection.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

42 of 76 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
30 more
  1. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  6. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  7. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  8. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  11. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  12. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  13. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  14. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
  15. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  16. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  17. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  18. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  19. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  20. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 59
  21. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  22. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  23. 2019XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answeringcited by 35
  24. 2020From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformerscited by 35
  25. 2016Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencodercited by 34
  26. 2018Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modelingcited by 34
  27. 2017Deep Generalized Canonical Correlation Analysiscited by 33
  28. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  29. 2020A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognitioncited by 12
  30. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detectioncited by 17
  2. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4