# Language Identification of Hindi-English tweets using code-mixed BERT

arXiv: https://arxiv.org/abs/2107.01202 · HTML: https://mlsanity.com/paper/2107.01202/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [Domain Divergences: a Survey and Empirical Analysis](https://mlsanity.com/paper/2010.12198/) (2020)
- [Comparing Approaches to Dravidian Language Identification](https://mlsanity.com/paper/2103.05552/) (2021)
- [BRUMS at SemEval-2020 Task 12 : Transformer based Multilingual Offensive Language Identification in Social Media](https://mlsanity.com/paper/2010.06278/) (2020)
- [InfoMiner at WNUT-2020 Task 2: Transformer-based Covid-19 Informative Tweet Extraction](https://mlsanity.com/paper/2010.05327/) (2020)
- [Recurrent-Neural-Network for Language Detection on Twitter Code-Switching Corpus](https://mlsanity.com/paper/1412.4314/) (2014)
- [Inferring Political Preferences from Twitter](https://mlsanity.com/paper/2007.10604/) (2020)

## Built on by (most cited first, 4 of 4)

- [L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models](https://mlsanity.com/paper/2204.08398/) (2022)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts](https://mlsanity.com/paper/2211.14459/) (2022)
- [Leveraging Language Identification to Enhance Code-Mixed Text Classification](https://mlsanity.com/paper/2306.04964/) (2023)

## Abstract (arXiv)

Language identification of social media text has been an interesting problem of study in recent years. Social media messages are predominantly in code mixed in non-English speaking states. Prior knowledge by pre-training contextual embeddings have shown state of the art results for a range of downstream tasks. Recently, models such as BERT have shown that using a large amount of unlabeled data, the pretrained language models are even more beneficial for learning common language representations. Extensive experiments exploiting transfer learning and fine-tuning BERT models to identify language on Twitter are presented in this paper. The work utilizes a data collection of Hindi-English-Urdu codemixed text for language pre-training and Hindi-English codemixed for subsequent word-level language classification. The results show that the representations pre-trained over codemixed data produce better results by their monolingual counterpart.