# LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR

arXiv: https://arxiv.org/abs/2309.16178 · HTML: https://mlsanity.com/paper/2309.16178/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognition](https://mlsanity.com/paper/2302.13750/) (2023)
- [On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/1811.00241/) (2018)
- [Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization](https://mlsanity.com/paper/2210.14567/) (2022)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [LAE: Language-Aware Encoder for Monolingual and Multilingual ASR](https://mlsanity.com/paper/2206.02093/) (2022)
- [Data Augmentation for End-to-end Code-switching Speech Recognition](https://mlsanity.com/paper/2011.02160/) (2020)
- [Language-specific Characteristic Assistance for Code-switching Speech Recognition](https://mlsanity.com/paper/2206.14580/) (2022)
- [Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorization](https://mlsanity.com/paper/2111.15016/) (2021)
- [Arabic Code-Switching Speech Recognition using Monolingual Data](https://mlsanity.com/paper/2107.01573/) (2021)
- [End-to-End Code-Switching ASR for Low-Resourced Language Pairs](https://mlsanity.com/paper/1909.12681/) (2019)
- [Towards Zero-Shot Code-Switched Speech Recognition](https://mlsanity.com/paper/2211.01458/) (2022)
- [Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.06812/) (2022)
- [Code-Switching Text Generation and Injection in Mandarin-English ASR](https://mlsanity.com/paper/2303.10949/) (2023)
- [Minimum word error training for non-autoregressive Transformer-based code-switching ASR](https://mlsanity.com/paper/2110.03573/) (2021)
- [Non-autoregressive Mandarin-English Code-switching Speech Recognition](https://mlsanity.com/paper/2104.02258/) (2021)
- [Leveraging Phone Mask Training for Phonetic-Reduction-Robust E2E Uyghur Speech Recognition](https://mlsanity.com/paper/2204.00819/) (2022)
- [Monolingual Data Selection Analysis for English-Mandarin Hybrid Code-switching Speech Recognition](https://mlsanity.com/paper/2006.07094/) (2020)
- [Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation](https://mlsanity.com/paper/2210.12214/) (2022)

## Built on by (most cited first, 6 of 6)

- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languages](https://mlsanity.com/paper/2506.14177/) (2025)
- [Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review](https://mlsanity.com/paper/2507.07741/) (2025)
- [BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR](https://mlsanity.com/paper/2501.12602/) (2025)
- [MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition](https://mlsanity.com/paper/2607.24030/) (2026)

## Abstract (arXiv)

Recently, to mitigate the confusion between different languages in code-switching (CS) automatic speech recognition (ASR), the conditionally factorized models, such as the language-aware encoder (LAE), explicitly disregard the contextual information between different languages. However, this information may be helpful for ASR modeling. To alleviate this issue, we propose the LAE-ST-MoE framework. It incorporates speech translation (ST) tasks into LAE and utilizes ST to learn the contextual information between different languages. It introduces a task-based mixture of expert modules, employing separate feed-forward networks for the ASR and ST tasks. Experimental results on the ASRU 2019 Mandarin-English CS challenge dataset demonstrate that, compared to the LAE-based CTC, the LAE-ST-MoE model achieves a 9.26% mix error reduction on the CS test with the same decoding parameter. Moreover, the well-trained LAE-ST-MoE model can perform ST tasks from CS speech to Mandarin or English text.