Towards continually learning new languages
Not on the map. Why?
Abstract (arXiv)
Multilingual speech recognition with neural networks is often implemented with batch-learning, when all of the languages are available before training. An ability to add new languages after the prior training sessions can be economically beneficial, but the main challenge is catastrophic forgetting. In this work, we combine the qualities of weight factorization and elastic weight consolidation in order to counter catastrophic forgetting and facilitate learning new languages quickly. Such combination allowed us to eliminate catastrophic forgetting while still achieving performance for the new languages comparable with having all languages at once, in experiments of learning from an initial 10 languages to achieve 26 languages without catastrophic forgetting and a reasonable performance compared to training all languages from scratch.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 21 Nov 2022.
Builds on
26 of 33 references.
References · 26
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
- 2016Progressive Neural Networkscited by 1,361
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
- 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
- 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
14 more
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
- 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
- 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
- 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
- 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 151
- 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
- 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
- 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
- 2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
- 2021Towards Lifelong Learning of End-to-end ASRcited by 21
- 2021Efficient Weight factorization for Multilingual Speech Recognitioncited by 12
- 2022Adaptive multilingual speech recognition with pretrained modelscited by 9
- 2021Stabilizing Elastic Weight Consolidation method in practical ML tasks and using weight importances for neural network pruningcited by 8
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.