# Comparing Approaches to Dravidian Language Identification

arXiv: https://arxiv.org/abs/2103.05552 · HTML: https://mlsanity.com/paper/2103.05552/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text](https://mlsanity.com/paper/2006.00206/) (2020)
- [Offensive Language Identification in Greek](https://mlsanity.com/paper/2003.07459/) (2020)
- [A Sentiment Analysis Dataset for Code-Mixed Malayalam-English](https://mlsanity.com/paper/2006.00210/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [Overview for the Second Shared Task on Language Identification in Code-Switched Data](https://mlsanity.com/paper/1909.13016/) (2019)
- [MUDES: Multilingual Detection of Offensive Spans](https://mlsanity.com/paper/2102.09665/) (2021)
- [LanideNN: Multilingual Language Identification on Character Window](https://mlsanity.com/paper/1701.03338/) (2017)
- [BRUMS at SemEval-2020 Task 12 : Transformer based Multilingual Offensive Language Identification in Social Media](https://mlsanity.com/paper/2010.06278/) (2020)
- [InfoMiner at WNUT-2020 Task 2: Transformer-based Covid-19 Informative Tweet Extraction](https://mlsanity.com/paper/2010.05327/) (2020)
- [WLV-RIT at HASOC-Dravidian-CodeMix-FIRE2020: Offensive Language Identification in Code-switched YouTube Comments](https://mlsanity.com/paper/2011.00559/) (2020)
- [BRUMS at SemEval-2020 Task 3: Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity](https://mlsanity.com/paper/2010.06269/) (2020)
- [Discriminating Similar Languages: Evaluations and Explorations](https://mlsanity.com/paper/1610.00031/) (2016)
- [Language Model Adaptation for Language and Dialect Identification of Text](https://mlsanity.com/paper/1903.10915/) (2019)

## Built on by (most cited first, 11 of 11)

- [Multilingual Offensive Language Identification for Low-resource Languages](https://mlsanity.com/paper/2105.05996/) (2021)
- [Bhasha-Abhijnaanam: Native-script and romanized Language Identification for 22 Indic languages](https://mlsanity.com/paper/2305.15814/) (2023)
- [WLV-RIT at SemEval-2021 Task 5: A Neural Transformer Framework for Detecting Toxic Spans](https://mlsanity.com/paper/2104.04630/) (2021)
- [LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages](https://mlsanity.com/paper/2305.14263/) (2023)
- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)
- [BERT(s) to Detect Multiword Expressions](https://mlsanity.com/paper/2208.07832/) (2022)
- [Transformers to Fight the COVID-19 Infodemic](https://mlsanity.com/paper/2104.12201/) (2021)
- [Language Identification of Hindi-English tweets using code-mixed BERT](https://mlsanity.com/paper/2107.01202/) (2021)
- [TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation](https://mlsanity.com/paper/2104.04632/) (2021)
- [WLV-RIT at GermEval 2021: Multitask Learning with Transformers to Detect Toxic, Engaging, and Fact-Claiming Comments](https://mlsanity.com/paper/2108.00057/) (2021)
- [From N-grams to Pre-trained Multilingual Models For Language Identification](https://mlsanity.com/paper/2410.08728/) (2024)

## Abstract (arXiv)

This paper describes the submissions by team HWR to the Dravidian Language Identification (DLI) shared task organized at VarDial 2021 workshop. The DLI training set includes 16,674 YouTube comments written in Roman script containing code-mixed text with English and one of the three South Dravidian languages: Kannada, Malayalam, and Tamil. We submitted results generated using two models, a Naive Bayes classifier with adaptive language models, which has shown to obtain competitive performance in many language and dialect identification tasks, and a transformer-based model which is widely regarded as the state-of-the-art in a number of NLP tasks. Our first submission was sent in the closed submission track using only the training set provided by the shared task organisers, whereas the second submission is considered to be open as it used a pretrained model trained with external data. Our team attained shared second position in the shared task with the submission based on Naive Bayes. Our results reinforce the idea that deep learning methods are not as competitive in language identification related tasks as they are in many other text classification tasks.