# EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching

arXiv: https://arxiv.org/abs/2210.12540 · HTML: https://mlsanity.com/paper/2210.12540/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 69 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models](https://mlsanity.com/paper/2010.06189/) (2020)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text](https://mlsanity.com/paper/2006.00206/) (2020)
- [Zero-Shot Cross-Lingual Transfer with Meta Learning](https://mlsanity.com/paper/2003.02739/) (2020)
- [ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora](https://mlsanity.com/paper/2012.15674/) (2020)
- [Explicit Alignment Objectives for Multilingual Bidirectional Encoders](https://mlsanity.com/paper/2010.07972/) (2020)
- [Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks](https://mlsanity.com/paper/1908.10423/) (2019)
- [ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation](https://mlsanity.com/paper/2112.06223/) (2021)
- [XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization](https://mlsanity.com/paper/2010.06478/) (2020)

## Built on by (most cited first, 8 of 8)

- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [Low-Rank Adaptation for Multilingual Summarization: An Empirical Study](https://mlsanity.com/paper/2311.08572/) (2023)
- [μPLAN: Summarizing using a Content Plan as Cross-Lingual Bridge](https://mlsanity.com/paper/2305.14205/) (2023)
- [Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?](https://mlsanity.com/paper/2507.12838/) (2025)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis](https://mlsanity.com/paper/2411.00691/) (2024)
- [Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognition](https://mlsanity.com/paper/2312.09583/) (2023)

## Abstract (arXiv)

Accurate alignment between languages is fundamental for improving cross-lingual pre-trained language models (XLMs). Motivated by the natural phenomenon of code-switching (CS) in multilingual speakers, CS has been used as an effective data augmentation method that offers language alignment at the word- or phrase-level, in contrast to sentence-level via parallel instances. Existing approaches either use dictionaries or parallel sentences with word alignment to generate CS data by randomly switching words in a sentence. However, such methods can be suboptimal as dictionaries disregard semantics, and syntax might become invalid after random word switching. In this work, we propose EntityCS, a method that focuses on Entity-level Code-Switching to capture fine-grained cross-lingual semantics without corrupting syntax. We use Wikidata and English Wikipedia to construct an entity-centric CS corpus by switching entities to their counterparts in other languages. We further propose entity-oriented masking strategies during intermediate model training on the EntityCS corpus for improving entity prediction. Evaluation of the trained models on four entity-centric downstream tasks shows consistent improvements over the baseline with a notable increase of 10% in Fact Retrieval. We release the corpus and models to assist research on code-switching and enriching XLMs with external knowledge.