# Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching

arXiv: https://arxiv.org/abs/2111.01231 · HTML: https://mlsanity.com/paper/2111.01231/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters](https://mlsanity.com/paper/2010.10392/) (2020)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [Overview for the Second Shared Task on Language Identification in Code-Switched Data](https://mlsanity.com/paper/1909.13016/) (2019)
- [Revisiting Character-Based Neural Machine Translation with Capacity and Compression](https://mlsanity.com/paper/1808.09943/) (2018)
- [Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task](https://mlsanity.com/paper/1906.04138/) (2019)
- [DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries](https://mlsanity.com/paper/2010.12566/) (2020)
- [Universal Dependency Parsing for Hindi-English Code-switching](https://mlsanity.com/paper/1804.05868/) (2018)
- [From English to Code-Switching: Transfer Learning with Strong Morphological Clues](https://mlsanity.com/paper/1909.05158/) (2019)
- [Code-Switching Language Modeling using Syntax-Aware Multi-Task Learning](https://mlsanity.com/paper/1805.12070/) (2018)
- [Subword-Level Language Identification for Intra-Word Code-Switching](https://mlsanity.com/paper/1904.01989/) (2019)
- [Humor Detection in English-Hindi Code-Mixed Social Media Content : Corpus and Baseline System](https://mlsanity.com/paper/1806.05513/) (2018)
- [Part of speech tagging for code switched data](https://mlsanity.com/paper/1909.13006/) (2019)
- [Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data](https://mlsanity.com/paper/1905.13359/) (2019)

## Built on by (most cited first, 5 of 5)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching](https://mlsanity.com/paper/2203.08979/) (2022)
- [A Survey of Code-switched Arabic NLP: Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2501.13419/) (2025)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Curriculum Script Distillation for Multilingual Visual Question Answering](https://mlsanity.com/paper/2301.07227/) (2023)

## Abstract (arXiv)

Code-switching (CS), a ubiquitous phenomenon due to the ease of communication it offers in multilingual communities still remains an understudied problem in language processing. The primary reasons behind this are: (1) minimal efforts in leveraging large pretrained multilingual models, and (2) the lack of annotated data. The distinguishing case of low performance of multilingual models in CS is the intra-sentence mixing of languages leading to switch points. We first benchmark two sequence labeling tasks -- POS and NER on 4 different language pairs with a suite of pretrained models to identify the problems and select the best performing model, char-BERT, among them (addressing (1)). We then propose a self training method to repurpose the existing pretrained models using a switch-point bias by leveraging unannotated data (addressing (2)). We finally demonstrate that our approach performs well on both tasks by reducing the gap between the switch point performance while retaining the overall performance on two distinct language pairs in both the tasks. Our code is available here: https://github.com/PC09/EMNLP2021-Switch-Point-biased-Self-Training.