# Multilingual Transfer Learning for Code-Switched Language and Speech Neural Modeling

arXiv: https://arxiv.org/abs/2104.06268 · HTML: https://mlsanity.com/paper/2104.06268/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 149 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [Towards Few-Shot Fact-Checking via Perplexity](https://mlsanity.com/paper/2103.09535/) (2021)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [XPersona: Evaluating Multilingual Personalized Chatbot](https://mlsanity.com/paper/2003.07568/) (2020)
- [Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition](https://mlsanity.com/paper/2012.01687/) (2020)
- [Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality](https://mlsanity.com/paper/2010.12730/) (2020)
- [Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling](https://mlsanity.com/paper/1810.10254/) (2018)
- [Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning](https://mlsanity.com/paper/2004.14218/) (2020)
- [Towards End-to-end Automatic Code-Switching Speech Recognition](https://mlsanity.com/paper/1810.12620/) (2018)
- [On the Importance of Word Order Information in Cross-lingual Sequence Labeling](https://mlsanity.com/paper/2001.11164/) (2020)

## Built on by (most cited first, 4 of 4)

- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)

## Abstract (arXiv)

In this thesis, we address the data scarcity and limitations of linguistic theory by proposing language-agnostic multi-task training methods. First, we introduce a meta-learning-based approach, meta-transfer learning, in which information is judiciously extracted from high-resource monolingual speech data to the code-switching domain. The meta-transfer learning quickly adapts the model to the code-switching task from a number of monolingual tasks by learning to learn in a multi-task learning fashion. Second, we propose a novel multilingual meta-embeddings approach to effectively represent code-switching data by acquiring useful knowledge learned in other languages, learning the commonalities of closely related languages and leveraging lexical composition. The method is far more efficient compared to contextualized pre-trained multilingual models. Third, we introduce multi-task learning to integrate syntactic information as a transfer learning strategy to a language model and learn where to code-switch. To further alleviate the aforementioned issues, we propose a data augmentation method using Pointer-Gen, a neural network using a copy mechanism to teach the model the code-switch points from monolingual parallel sentences. We disentangle the need for linguistic theory, and the model captures code-switching points by attending to input words and aligning the parallel words, without requiring any word alignments or constituency parsers. More importantly, the model can be effectively used for languages that are syntactically different, and it outperforms the linguistic theory-based models.