# GLUECoS : An Evaluation Benchmark for Code-Switched NLP

arXiv: https://arxiv.org/abs/2004.12376 · HTML: https://mlsanity.com/paper/2004.12376/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2020.

## Built on by (most cited first, 30 of 102)

- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective](https://mlsanity.com/paper/2211.08073/) (2022)
- [NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages](https://mlsanity.com/paper/2205.15960/) (2022)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages](https://mlsanity.com/paper/2303.13592/) (2023)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text](https://mlsanity.com/paper/2107.06483/) (2021)
- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [Predicting the Performance of Multilingual NLP Models](https://mlsanity.com/paper/2110.08875/) (2021)
- [Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots](https://mlsanity.com/paper/2103.09593/) (2021)
- [Unveiling Linguistic Regions in Large Language Models](https://mlsanity.com/paper/2402.14700/) (2024)
- [Can You Traducir This? Machine Translation for Code-Switched Input](https://mlsanity.com/paper/2105.04846/) (2021)
- [Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings](https://mlsanity.com/paper/2110.07385/) (2021)
- [Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes](https://mlsanity.com/paper/2505.14815/) (2025)
- [Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages](https://mlsanity.com/paper/2205.12676/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation](https://mlsanity.com/paper/2310.20470/) (2023)
- [SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification](https://mlsanity.com/paper/2102.01051/) (2021)
- [MINERS: Multilingual Language Models as Semantic Retrievers](https://mlsanity.com/paper/2406.07424/) (2024)

## Abstract (arXiv)

Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.