# MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation

arXiv: https://arxiv.org/abs/2107.11534 · HTML: https://mlsanity.com/paper/2107.11534/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 24 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Code-switching pre-training for neural machine translation](https://mlsanity.com/paper/2009.08088/) (2020)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [Towards Sub-Word Level Compositions for Sentiment Analysis of Hindi-English Code Mixed Text](https://mlsanity.com/paper/1611.00472/) (2016)
- [A New Dataset for Natural Language Inference from Code-mixed Conversations](https://mlsanity.com/paper/2004.05051/) (2020)
- [HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text](https://mlsanity.com/paper/2107.03760/) (2021)
- [Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text](https://mlsanity.com/paper/2106.10123/) (2021)
- [Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing](https://mlsanity.com/paper/2105.08807/) (2021)
- [Word Level Language Identification in English Telugu Code Mixed Data](https://mlsanity.com/paper/2010.04482/) (2020)
- [Automatic Normalization of Word Variations in Code-Mixed Social Media Text](https://mlsanity.com/paper/1804.00804/) (2018)
- [A Robust Self-Learning Method for Fully Unsupervised Cross-Lingual Mappings of Word Embeddings: Making the Method Robustly Reproducible as Well](https://mlsanity.com/paper/1912.01706/) (2019)

## Built on by (most cited first, 7 of 7)

- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentences](https://mlsanity.com/paper/2410.10580/) (2024)
- [niksss at HinglishEval: Language-agnostic BERT-based Contextual Embeddings with Catboost for Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text](https://mlsanity.com/paper/2206.08910/) (2022)
- [CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts](https://mlsanity.com/paper/2505.05063/) (2025)
- [PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues](https://mlsanity.com/paper/2601.17277/) (2026)
- [BITS Pilani at HinglishEval: Quality Evaluation for Code-Mixed Hinglish Text Using Transformers](https://mlsanity.com/paper/2206.08680/) (2022)

## Abstract (arXiv)

Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric independent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort.