# Challenges and Considerations with Code-Mixed NLP for Multilingual Societies

arXiv: https://arxiv.org/abs/2106.07823 · HTML: https://mlsanity.com/paper/2106.07823/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 121 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [Fake News Detection on Social Media: A Data Mining Perspective](https://mlsanity.com/paper/1708.01967/) (2017)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [GRAM: Graph-based Attention Model for Healthcare Representation Learning](https://mlsanity.com/paper/1611.07012/) (2016)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [Automated Fact Checking: Task formulations, methods and future directions](https://mlsanity.com/paper/1806.07687/) (2018)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [CoAID: COVID-19 Healthcare Misinformation Dataset](https://mlsanity.com/paper/2006.00885/) (2020)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [A Survey on Natural Language Processing for Fake News Detection](https://mlsanity.com/paper/1811.00770/) (2018)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19](https://mlsanity.com/paper/2006.11343/) (2020)
- [Aggression-annotated Corpus of Hindi-English Code-mixed Data](https://mlsanity.com/paper/1803.09402/) (2018)
- [BanFakeNews: A Dataset for Detecting Fake News in Bangla](https://mlsanity.com/paper/2004.08789/) (2020)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [XFake: Explainable Fake News Detector with Visualizations](https://mlsanity.com/paper/1907.07757/) (2019)
- [Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign](https://mlsanity.com/paper/1802.04291/) (2018)
- [NELA-GT-2020: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles](https://mlsanity.com/paper/2102.04567/) (2021)
- [Towards Automatic Detection of Misinformation in Online Medical Videos](https://mlsanity.com/paper/1909.01543/) (2019)
- [Hate Speech Detection from Code-mixed Hindi-English Tweets Using Deep Learning Models](https://mlsanity.com/paper/1811.05145/) (2018)
- [GeoCoV19: A Dataset of Hundreds of Millions of Multilingual COVID-19 Tweets with Location Information](https://mlsanity.com/paper/2005.11177/) (2020)
- [No Rumours Please! A Multi-Indic-Lingual Approach for COVID Fake-Tweet Detection](https://mlsanity.com/paper/2010.06906/) (2020)
- [Images and Misinformation in Political Groups: Evidence from WhatsApp in India](https://mlsanity.com/paper/2005.09784/) (2020)
- [Analysing the Extent of Misinformation in Cancer Related Tweets](https://mlsanity.com/paper/2003.13657/) (2020)
- [IIT Gandhinagar at SemEval-2020 Task 9: Code-Mixed Sentiment Classification Using Candidate Sentence Generation and Selection](https://mlsanity.com/paper/2006.14465/) (2020)

## Built on by (most cited first, 4 of 4)

- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://mlsanity.com/paper/2402.06619/) (2024)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [Prabhupadavani: A Code-mixed Speech Translation Data for 25 Languages](https://mlsanity.com/paper/2201.11391/) (2022)
- [Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit](https://mlsanity.com/paper/2308.08807/) (2023)

## Abstract (arXiv)

Multilingualism refers to the high degree of proficiency in two or more languages in the written and oral communication modes. It often results in language mixing, a.k.a. code-mixing, when a multilingual speaker switches between multiple languages in a single utterance of a text or speech. This paper discusses the current state of the NLP research, limitations, and foreseeable pitfalls in addressing five real-world applications for social good crisis management, healthcare, political campaigning, fake news, and hate speech for multilingual societies. We also propose futuristic datasets, models, and tools that can significantly advance the current research in multilingual NLP applications for the societal good. As a representative example, we consider English-Hindi code-mixing but draw similar inferences for other language pairs