# HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text

arXiv: https://arxiv.org/abs/2107.03760 · HTML: https://mlsanity.com/paper/2107.03760/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences](https://mlsanity.com/paper/1909.08582/) (2019)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text](https://mlsanity.com/paper/1611.00472/) (2016)
- [Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text](https://mlsanity.com/paper/2106.10123/) (2021)
- [A Dataset for Building Code-Mixed Goal Oriented Conversation Systems](https://mlsanity.com/paper/1806.05997/) (2018)
- [A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection](https://mlsanity.com/paper/1805.11869/) (2018)
- [Cross-Lingual Dependency Parsing Using Code-Mixed TreeBank](https://mlsanity.com/paper/1909.02235/) (2019)
- [Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling](https://mlsanity.com/paper/1810.10254/) (2018)
- [All that is English may be Hindi: Enhancing language identification through automatic ranking of likeliness of word borrowing in social media](https://mlsanity.com/paper/1707.08446/) (2017)
- [Codeswitched Sentence Creation using Dependency Parsing](https://mlsanity.com/paper/2012.02990/) (2020)

## Built on by (most cited first, 28 of 28)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [ArzEn-ST: A Three-way Speech Translation Corpus for Code-Switched Egyptian Arabic - English](https://mlsanity.com/paper/2211.12000/) (2022)
- [MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation](https://mlsanity.com/paper/2107.11534/) (2021)
- [Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text](https://mlsanity.com/paper/2108.01861/) (2021)
- [Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities](https://mlsanity.com/paper/2410.11079/) (2024)
- [COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing](https://mlsanity.com/paper/2503.21670/) (2025)
- [Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models](https://mlsanity.com/paper/2410.22660/) (2024)
- [CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback](https://mlsanity.com/paper/2411.09073/) (2024)
- [Language Agnostic Code-Mixing Data Augmentation by Predicting Linguistic Patterns](https://mlsanity.com/paper/2211.07628/) (2022)
- [Persona-aware Generative Model for Code-mixed Language](https://mlsanity.com/paper/2309.02915/) (2023)
- [Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentences](https://mlsanity.com/paper/2410.10580/) (2024)
- [DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching](https://mlsanity.com/paper/2509.17768/) (2025)
- [MUTANT: A Multi-sentential Code-mixed Hinglish Dataset](https://mlsanity.com/paper/2302.11766/) (2023)
- [My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks](https://mlsanity.com/paper/2306.14030/) (2023)
- [Marathi-English Code-mixed Text Generation](https://mlsanity.com/paper/2309.16202/) (2023)
- [Grammatical Error Correction for Code-Switched Sentences by Learners of English](https://mlsanity.com/paper/2404.12489/) (2024)
- [niksss at HinglishEval: Language-agnostic BERT-based Contextual Embeddings with Catboost for Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text](https://mlsanity.com/paper/2206.08910/) (2022)
- [SIT at MixMT 2022: Fluent Translation Built on Giant Pre-trained Models](https://mlsanity.com/paper/2210.11670/) (2022)
- [Gui at MixMT 2022 : English-Hinglish: An MT approach for translation of code mixed data](https://mlsanity.com/paper/2210.12215/) (2022)
- [Who Speaks Matters: Analysing the Influence of the Speaker's Ethnicity on Hate Classification](https://mlsanity.com/paper/2410.20490/) (2024)
- [Sample-Efficient Language Model for Hinglish Conversational AI](https://mlsanity.com/paper/2504.19070/) (2025)
- [PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues](https://mlsanity.com/paper/2601.17277/) (2026)
- [PreCogIIITH at HinglishEval : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality](https://mlsanity.com/paper/2206.07988/) (2022)
- [JU_NLP at HinglishEval: Quality Evaluation of the Low-Resource Code-Mixed Hinglish Text](https://mlsanity.com/paper/2206.08053/) (2022)
- [BITS Pilani at HinglishEval: Quality Evaluation for Code-Mixed Hinglish Text Using Transformers](https://mlsanity.com/paper/2206.08680/) (2022)
- [The University of Edinburgh's Submission to the WMT22 Code-Mixing Shared Task (MixMT)](https://mlsanity.com/paper/2210.11309/) (2022)
- [Domain Curricula for Code-Switched MT at MixMT 2022](https://mlsanity.com/paper/2210.17463/) (2022)

## Abstract (arXiv)

Text generation is a highly active area of research in the computational linguistic community. The evaluation of the generated text is a challenging task and multiple theories and metrics have been proposed over the years. Unfortunately, text generation and evaluation are relatively understudied due to the scarcity of high-quality resources in code-mixed languages where the words and phrases from multiple languages are mixed in a single utterance of text and speech. To address this challenge, we present a corpus (HinGE) for a widely popular code-mixed language Hinglish (code-mixing of Hindi and English languages). HinGE has Hinglish sentences generated by humans as well as two rule-based algorithms corresponding to the parallel Hindi-English sentences. In addition, we demonstrate the inefficacy of widely-used evaluation metrics on the code-mixed data. The HinGE dataset will facilitate the progress of natural language generation research in code-mixed languages.