mlsanityData to 7 Oct 2026

Efficient Weight factorization for Multilingual Speech Recognition

arXiv 2105.03010 ↗First posted 7 May 2021cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages, resulting in a single neural network to handle transcribing different languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. In this paper we propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions. The key idea of the method is to assign fast weight matrices for each language by decomposing each weight matrix into a shared component and a language dependent component. The latter is then factorized into vectors using rank-1 assumptions to reduce the number of parameters per language. This efficient factorization scheme is proved to be effective in two multilingual settings with 7 and 27 languages, reducing the word error rates by 26% and 27% rel. for two popular architectures LSTM and Transformer, respectively.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

31 of 42 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2017Convolutional Sequence to Sequence Learningcited by 550
  9. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  10. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
19 more
  1. 2020Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscited by 171
  2. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  3. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  4. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  5. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  6. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  7. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  8. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  9. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  10. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  11. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  12. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  13. 2018Contextual Parameter Generation for Universal Neural Machine Translationcited by 40
  14. 2016Very Deep Convolutional Networks for End-to-End Speech Recognitioncited by 34
  15. 2019Massively Multilingual Adversarial Speech Recognitioncited by 28
  16. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26
  17. 2020Relative Positional Encoding for Speech Recognition and Direct Translationcited by 23
  18. 2017Towards Language-Universal End-to-End Speech Recognitioncited by 14
  19. 2018Neural Language Codes for Multilingual Acoustic Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 1; 2024: 3; 2025: 2; 2026: 2.

12 papers.

  1. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18
  2. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  3. 2022Adaptive multilingual speech recognition with pretrained modelscited by 9
  4. 2025Weight Factorization and Centralization for Continual Learning in Speech Recognitioncited by 6
  5. 2024Accent conversion using discrete units with parallel data synthesized from controllable accented TTScited by 5
  6. 2024Continuously Learning New Words in Automatic Speech Recognitioncited by 3
  7. 2022Towards continually learning new languagescited by 2
  8. 2025Bayesian Low-Rank Factorization for Robust Model Adaptationcited by 1
  9. 2022Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Informationcited by 0
  10. 2023Scalable Weight Reparametrization for Efficient Transfer Learningcited by 0
  11. 2026Adding Robust Code-Switching Capabilities to High Performance Multilingual ASRcited by 0
  12. 2026Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026cited by 0