# SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

arXiv: https://arxiv.org/abs/2102.01051 · HTML: https://mlsanity.com/paper/2102.01051/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [A Survey of Code-switched Speech and Language Processing](https://mlsanity.com/paper/1904.00784/) (2019)
- [Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text](https://mlsanity.com/paper/2006.00206/) (2020)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [A Sentiment Analysis Dataset for Code-Mixed Malayalam-English](https://mlsanity.com/paper/2006.00210/) (2020)
- [DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries](https://mlsanity.com/paper/2010.12566/) (2020)

## Built on by (most cited first, 11 of 11)

- [CodemixedNLP: An Extensible and Open NLP Toolkit for Code-Mixing](https://mlsanity.com/paper/2106.06004/) (2021)
- [Unsupervised Self-Training for Sentiment Analysis of Code-Switched Data](https://mlsanity.com/paper/2103.14797/) (2021)
- [Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling](https://mlsanity.com/paper/2108.12177/) (2021)
- [Task-Specific Pre-Training and Cross Lingual Transfer for Code-Switched Data](https://mlsanity.com/paper/2102.12407/) (2021)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts](https://mlsanity.com/paper/2108.10939/) (2021)
- [Developing Successful Shared Tasks on Offensive Language Identification for Dravidian Languages](https://mlsanity.com/paper/2111.03375/) (2021)
- [AIR-JPMC@SMM4H'22: Classifying Self-Reported Intimate Partner Violence in Tweets with Multiple BERT-based Models](https://mlsanity.com/paper/2209.10763/) (2022)
- [Automatic Identification of Motivation for Code-Switching in Speech Transcripts](https://mlsanity.com/paper/2212.08565/) (2022)
- [Multilingual Text Classification for Dravidian Languages](https://mlsanity.com/paper/2112.01705/) (2021)
- [Sentiment Classification of Code-Switched Text using Pre-trained Multilingual Embeddings and Segmentation](https://mlsanity.com/paper/2210.16461/) (2022)

## Abstract (arXiv)

In this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.