# Evaluating Input Representation for Language Identification in Hindi-English Code Mixed Text

arXiv: https://arxiv.org/abs/2011.11263 · HTML: https://mlsanity.com/paper/2011.11263/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 6 of 6)

- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing](https://mlsanity.com/paper/2503.21670/) (2025)
- [Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches](https://mlsanity.com/paper/2110.09338/) (2021)
- [Comparative Study of Pre-Trained BERT Models for Code-Mixed Hindi-English Data](https://mlsanity.com/paper/2305.15722/) (2023)
- [Leveraging Language Identification to Enhance Code-Mixed Text Classification](https://mlsanity.com/paper/2306.04964/) (2023)
- [On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi](https://mlsanity.com/paper/2312.01306/) (2023)

## Abstract (arXiv)

Natural language processing (NLP) techniques have become mainstream in the recent decade. Most of these advances are attributed to the processing of a single language. More recently, with the extensive growth of social media platforms focus has shifted to code-mixed text. The code-mixed text comprises text written in more than one language. People naturally tend to combine local language with global languages like English. To process such texts, current NLP techniques are not sufficient. As a first step, the text is processed to identify the language of the words in the text. In this work, we focus on language identification in code-mixed sentences for Hindi-English mixed text. The task of language identification is formulated as a token classification task. In the supervised setting, each word in the sentence has an associated language label. We evaluate different deep learning models and input representation combinations for this task. Mainly, character, sub-word, and word embeddings are considered in combination with CNN and LSTM based models. We show that sub-word representation along with the LSTM model gives the best results. In general sub-word representations perform significantly better than other input representations. We report the best accuracy of 94.52% using a single layer LSTM model on the standard SAIL ICON 2017 test set.