# Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data

arXiv: https://arxiv.org/abs/2305.15722 · HTML: https://mlsanity.com/paper/2305.15722/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text](https://mlsanity.com/paper/2006.00206/) (2020)
- [A Sentiment Analysis Dataset for Code-Mixed Malayalam-English](https://mlsanity.com/paper/2006.00210/) (2020)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text](https://mlsanity.com/paper/1611.00472/) (2016)
- [Sentiment Analysis of Code-Mixed Indian Languages: An Overview of SAIL_Code-Mixed Shared Task @ICON-2017](https://mlsanity.com/paper/1803.06745/) (2018)
- [Hate Speech Detection from Code-mixed Hindi-English Tweets Using Deep Learning Models](https://mlsanity.com/paper/1811.05145/) (2018)
- [A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection](https://mlsanity.com/paper/1805.11869/) (2018)
- [Humor Detection in English-Hindi Code-Mixed Social Media Content : Corpus and Baseline System](https://mlsanity.com/paper/1806.05513/) (2018)
- [Towards Emotion Recognition in Hindi-English Code-Mixed Data: A Transformer Based Approach](https://mlsanity.com/paper/2102.09943/) (2021)
- [Evaluating Input Representation for Language Identification in Hindi-English Code Mixed Text](https://mlsanity.com/paper/2011.11263/) (2020)

## Built on by (most cited first, 3 of 3)

- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling](https://mlsanity.com/paper/2505.12587/) (2025)
- [Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition](https://mlsanity.com/paper/2509.02514/) (2025)

## Abstract (arXiv)

The term "Code Mixed" refers to the use of more than one language in the same text. This phenomenon is predominantly observed on social media platforms, with an increasing amount of adaptation as time goes on. It is critical to detect foreign elements in a language and process them correctly, as a considerable number of individuals are using code-mixed languages that could not be comprehended by understanding one of those languages. In this work, we focus on low-resource Hindi-English code-mixed language and enhancing the performance of different code-mixed natural language processing tasks such as sentiment analysis, emotion recognition, and hate speech identification. We perform a comparative analysis of different Transformer-based language Models pre-trained using unsupervised approaches. We have included the code-mixed models like HingBERT, HingRoBERTa, HingRoBERTa-Mixed, mBERT, and non-code-mixed models like AlBERT, BERT, and RoBERTa for comparative analysis of code-mixed Hindi-English downstream tasks. We report state-of-the-art results on respective datasets using HingBERT-based models which are specifically pre-trained on real code-mixed text. Our HingBERT-based models provide significant improvements thus highlighting the poor performance of vanilla BERT models on code-mixed text.