# L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models

arXiv: https://arxiv.org/abs/2204.08398 · HTML: https://mlsanity.com/paper/2204.08398/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text](https://mlsanity.com/paper/2107.03760/) (2021)
- [Universal Dependency Parsing for Hindi-English Code-switching](https://mlsanity.com/paper/1804.05868/) (2018)
- [Hate Speech Detection from Code-mixed Hindi-English Tweets Using Deep Learning Models](https://mlsanity.com/paper/1811.05145/) (2018)
- [Evaluating Input Representation for Language Identification in Hindi-English Code Mixed Text](https://mlsanity.com/paper/2011.11263/) (2020)
- [Challenges and Considerations with Code-Mixed NLP for Multilingual Societies](https://mlsanity.com/paper/2106.07823/) (2021)
- [Language Identification of Hindi-English tweets using code-mixed BERT](https://mlsanity.com/paper/2107.01202/) (2021)
- [Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches](https://mlsanity.com/paper/2110.09338/) (2021)

## Built on by (most cited first, 30 of 34)

- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos](https://mlsanity.com/paper/2405.20628/) (2024)
- [Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models](https://mlsanity.com/paper/2403.00794/) (2024)
- [SentMix-3L: A Bangla-English-Hindi Code-Mixed Dataset for Sentiment Analysis](https://mlsanity.com/paper/2310.18023/) (2023)
- [COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing](https://mlsanity.com/paper/2503.21670/) (2025)
- [OffMix-3L: A Novel Code-Mixed Dataset in Bangla-English-Hindi for Offensive Language Identification](https://mlsanity.com/paper/2310.18387/) (2023)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi](https://mlsanity.com/paper/2309.10272/) (2023)
- [Code-mixed Sentiment and Hate-speech Prediction](https://mlsanity.com/paper/2405.12929/) (2024)
- [What talking you?: Translating Code-Mixed Messaging Texts to English](https://mlsanity.com/paper/2411.05253/) (2024)
- [Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data](https://mlsanity.com/paper/2305.15722/) (2023)
- [Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentences](https://mlsanity.com/paper/2410.10580/) (2024)
- [MUTANT: A Multi-sentential Code-mixed Hinglish Dataset](https://mlsanity.com/paper/2302.11766/) (2023)
- [My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks](https://mlsanity.com/paper/2306.14030/) (2023)
- [FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning](https://mlsanity.com/paper/2309.04663/) (2023)
- [CONFLATOR: Incorporating Switching Point based Rotatory Positional Encodings for Code-Mixed Language Modeling](https://mlsanity.com/paper/2309.05270/) (2023)
- [Marathi-English Code-mixed Text Generation](https://mlsanity.com/paper/2309.16202/) (2023)
- [EmoMix-3L: A Code-Mixed Dataset for Bangla-English-Hindi Emotion Detection](https://mlsanity.com/paper/2405.06922/) (2024)
- [CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling](https://mlsanity.com/paper/2505.12587/) (2025)
- [Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification](https://mlsanity.com/paper/2402.03137/) (2024)
- [CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts](https://mlsanity.com/paper/2505.05063/) (2025)
- [PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues](https://mlsanity.com/paper/2601.17277/) (2026)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)
- [The University of Edinburgh's Submission to the WMT22 Code-Mixing Shared Task (MixMT)](https://mlsanity.com/paper/2210.11309/) (2022)
- [Leveraging Language Identification to Enhance Code-Mixed Text Classification](https://mlsanity.com/paper/2306.04964/) (2023)
- [Code-Mixed Text to Speech Synthesis under Low-Resource Constraints](https://mlsanity.com/paper/2312.01103/) (2023)
- [Cross-Linguistic Offensive Language Detection: BERT-Based Analysis of Bengali, Assamese, & Bodo Conversational Hateful Content from Social Media](https://mlsanity.com/paper/2312.10528/) (2023)
- [IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings](https://mlsanity.com/paper/2404.04525/) (2024)
- [Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts](https://mlsanity.com/paper/2412.16614/) (2024)

## Abstract (arXiv)

Code-switching occurs when more than one language is mixed in a given sentence or a conversation. This phenomenon is more prominent on social media platforms and its adoption is increasing over time. Therefore code-mixed NLP has been extensively studied in the literature. As pre-trained transformer-based architectures are gaining popularity, we observe that real code-mixing data are scarce to pre-train large language models. We present L3Cube-HingCorpus, the first large-scale real Hindi-English code mixed data in a Roman script. It consists of 52.93M sentences and 1.04B tokens, scraped from Twitter. We further present HingBERT, HingMBERT, HingRoBERTa, and HingGPT. The BERT models have been pre-trained on codemixed HingCorpus using masked language modelling objectives. We show the effectiveness of these BERT models on the subsequent downstream tasks like code-mixed sentiment analysis, POS tagging, NER, and LID from the GLUECoS benchmark. The HingGPT is a GPT2 based generative transformer model capable of generating full tweets. We also release L3Cube-HingLID Corpus, the largest code-mixed Hindi-English language identification(LID) dataset and HingBERT-LID, a production-quality LID model to facilitate capturing of more code-mixed data using the process outlined in this work. The dataset and models are available at https://github.com/l3cube-pune/code-mixed-nlp .