# XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition

arXiv: https://arxiv.org/abs/2103.08207 · HTML: https://mlsanity.com/paper/2103.08207/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling](https://mlsanity.com/paper/1810.03459/) (2018)
- [Large-Scale Domain Adaptation via Teacher-Student Learning](https://mlsanity.com/paper/1708.05466/) (2017)
- [Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2001.01798/) (2020)
- [Sequence-based Multi-lingual Low Resource Speech Recognition](https://mlsanity.com/paper/1802.07420/) (2018)
- [Very Deep Multilingual Convolutional Neural Networks for LVCSR](https://mlsanity.com/paper/1509.08967/) (2015)
- [Analysis of Multilingual Sequence-to-Sequence speech recognition systems](https://mlsanity.com/paper/1811.03451/) (2018)

## Built on by (most cited first, 11 of 11)

- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR](https://mlsanity.com/paper/2110.04484/) (2021)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [A Novel Self-training Approach for Low-resource Speech Recognition](https://mlsanity.com/paper/2308.05269/) (2023)
- [Efficient Utilization of Large Pre-Trained Models for Low Resource ASR](https://mlsanity.com/paper/2210.15445/) (2022)
- [A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers](https://mlsanity.com/paper/2304.07842/) (2023)
- [Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition](https://mlsanity.com/paper/2305.11569/) (2023)
- [Language-universal phonetic encoder for low-resource speech recognition](https://mlsanity.com/paper/2305.11576/) (2023)
- [Unsupervised Pre-Training for Vietnamese Automatic Speech Recognition in the HYKIST Project](https://mlsanity.com/paper/2309.15869/) (2023)
- [Unsupervised ASR via Cross-Lingual Pseudo-Labeling](https://mlsanity.com/paper/2305.13330/) (2023)

## Abstract (arXiv)

In this paper, we propose a weakly supervised multilingual representation learning framework, called cross-lingual self-training (XLST). XLST is able to utilize a small amount of annotated data from high-resource languages to improve the representation learning on multilingual un-annotated data. Specifically, XLST uses a supervised trained model to produce initial representations and another model to learn from them, by maximizing the similarity between output embeddings of these two models. Furthermore, the moving average mechanism and multi-view data augmentation are employed, which are experimentally shown to be crucial to XLST. Comprehensive experiments have been conducted on the CommonVoice corpus to evaluate the effectiveness of XLST. Results on 5 downstream low-resource ASR tasks shows that our multilingual pretrained model achieves relatively 18.6% PER reduction over the state-of-the-art self-supervised method, with leveraging additional 100 hours of annotated English data.