# Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing

arXiv: https://arxiv.org/abs/2105.08807 · HTML: https://mlsanity.com/paper/2105.08807/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://mlsanity.com/paper/1708.00391/) (2017)
- [Code-Switched Language Models Using Neural Based Synthetic Data from Parallel Sentences](https://mlsanity.com/paper/1909.08582/) (2019)
- [PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation](https://mlsanity.com/paper/2004.09447/) (2020)
- [Open Subtitles Paraphrase Corpus for Six Languages](https://mlsanity.com/paper/1809.06142/) (2018)
- [Code-switched Language Models Using Dual RNNs and Same-Source Pretraining](https://mlsanity.com/paper/1809.01962/) (2018)
- [A Deep Generative Model for Code-Switched Text](https://mlsanity.com/paper/1906.08972/) (2019)

## Built on by (most cited first, 16 of 16)

- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages](https://mlsanity.com/paper/2303.13592/) (2023)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [End-to-End Speech Translation for Code Switched Speech](https://mlsanity.com/paper/2204.05076/) (2022)
- [CALCS 2021 Shared Task: Machine Translation for Code-Switched Data](https://mlsanity.com/paper/2202.09625/) (2022)
- [MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation](https://mlsanity.com/paper/2107.11534/) (2021)
- [Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation](https://mlsanity.com/paper/2403.16771/) (2024)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Persona-aware Generative Model for Code-mixed Language](https://mlsanity.com/paper/2309.02915/) (2023)
- [Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models](https://mlsanity.com/paper/2109.14350/) (2021)
- [Marathi-English Code-mixed Text Generation](https://mlsanity.com/paper/2309.16202/) (2023)
- [Improving Similar Language Translation With Transfer Learning](https://mlsanity.com/paper/2108.03533/) (2021)
- [The Who in Code-Switching: A Case Study for Predicting Egyptian Arabic-English Code-Switching Levels based on Character Profiles](https://mlsanity.com/paper/2208.00433/) (2022)
- [SIT at MixMT 2022: Fluent Translation Built on Giant Pre-trained Models](https://mlsanity.com/paper/2210.11670/) (2022)
- [Study of Encoder-Decoder Architectures for Code-Mix Search Query Translation](https://mlsanity.com/paper/2208.03713/) (2022)
- [YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification](https://mlsanity.com/paper/2411.05039/) (2024)

## Abstract (arXiv)

We describe models focused at the understudied problem of translating between monolingual and code-mixed language pairs. More specifically, we offer a wide range of models that convert monolingual English text into Hinglish (code-mixed Hindi and English). Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models (i.e., mT5 and mBART) on the task finding both to work well. Given the paucity of training data for code-mixing, we also propose a dependency-free method for generating code-mixed texts from bilingual distributed representations that we exploit for improving language model performance. In particular, armed with this additional data, we adopt a curriculum learning approach where we first finetune the language models on synthetic data then on gold code-mixed data. We find that, although simple, our synthetic code-mixing method is competitive with (and in some cases is even superior to) several standard methods (backtranslation, method based on equivalence constraint theory) under a diverse set of conditions. Our work shows that the mT5 model, finetuned following the curriculum learning procedure, achieves best translation performance (12.67 BLEU). Our models place first in the overall ranking of the English-Hinglish official shared task.