# Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi

arXiv: https://arxiv.org/abs/2204.08669 · HTML: https://mlsanity.com/paper/2204.08669/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources](https://mlsanity.com/paper/2202.01159/) (2022)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models](https://mlsanity.com/paper/2004.06499/) (2020)
- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset](https://mlsanity.com/paper/2103.11408/) (2021)
- [Is Multilingual BERT Fluent in Language Generation?](https://mlsanity.com/paper/1910.03806/) (2019)
- [Deep Learning for Hindi Text Classification: A Comparison](https://mlsanity.com/paper/2001.10340/) (2020)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models](https://mlsanity.com/paper/2203.13778/) (2022)
- [Experimental Evaluation of Deep Learning models for Marathi Text Classification](https://mlsanity.com/paper/2101.04899/) (2021)
- [Hate and Offensive Speech Detection in Hindi and Marathi](https://mlsanity.com/paper/2110.12200/) (2021)
- [Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages](https://mlsanity.com/paper/2112.09301/) (2021)
- [AI-UPV at IberLEF-2021 DETOXIS task: Toxicity Detection in Immigration-Related Web News Comments Using Transformers and Statistical Models](https://mlsanity.com/paper/2111.04530/) (2021)
- [Evaluation of Deep Learning Models for Hostility Detection in Hindi Text](https://mlsanity.com/paper/2101.04144/) (2021)
- [Training dataset and dictionary sizes matter in BERT models: the case of Baltic languages](https://mlsanity.com/paper/2112.10553/) (2021)
- [Monolingual versus Multilingual BERTology for Vietnamese Extractive Multi-Document Summarization](https://mlsanity.com/paper/2108.13741/) (2021)
- [Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition](https://mlsanity.com/paper/2203.12907/) (2022)
- [A Comparative Study on Different Types of Approaches to Bengali document Categorization](https://mlsanity.com/paper/1701.08694/) (2017)

## Built on by (most cited first, 16 of 16)

- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi](https://mlsanity.com/paper/2211.11187/) (2022)
- [L3Cube-MahaSent-MD: A Multi-domain Marathi Sentiment Analysis Dataset and Transformer Models](https://mlsanity.com/paper/2306.13888/) (2023)
- [L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi](https://mlsanity.com/paper/2404.18216/) (2024)
- [User-Aware Multilingual Abusive Content Detection in Social Media](https://mlsanity.com/paper/2410.21321/) (2024)
- [Spread Love Not Hate: Undermining the Importance of Hateful Pre-training for Hate Speech Detection](https://mlsanity.com/paper/2210.04267/) (2022)
- [Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection](https://mlsanity.com/paper/2403.08035/) (2024)
- [A Twitter BERT Approach for Offensive Language Detection in Marathi](https://mlsanity.com/paper/2212.10039/) (2022)
- [Improving code-mixed hate detection by native sample mixing: A case study for Hindi-English code-mixed scenario](https://mlsanity.com/paper/2405.20755/) (2024)
- [A Survey on Automatic Online Hate Speech Detection in Low-Resource Languages](https://mlsanity.com/paper/2411.19017/) (2024)
- [Harnessing Pre-Trained Sentence Transformers for Offensive Language Detection in Indian Languages](https://mlsanity.com/paper/2310.02249/) (2023)
- [Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities](https://mlsanity.com/paper/2501.12980/) (2025)
- [L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models](https://mlsanity.com/paper/2508.21569/) (2025)

## Abstract (arXiv)

Transformers are the most eminent architectures used for a vast range of Natural Language Processing tasks. These models are pre-trained over a large text corpus and are meant to serve state-of-the-art results over tasks like text classification. In this work, we conduct a comparative study between monolingual and multilingual BERT models. We focus on the Marathi language and evaluate the models on the datasets for hate speech detection, sentiment analysis and simple text classification in Marathi. We use standard multilingual models such as mBERT, indicBERT and xlm-RoBERTa and compare with MahaBERT, MahaALBERT and MahaRoBERTa, the monolingual models for Marathi. We further show that Marathi monolingual models outperform the multilingual BERT variants on five different downstream fine-tuning experiments. We also evaluate sentence embeddings from these models by freezing the BERT encoder layers. We show that monolingual MahaBERT based models provide rich representations as compared to sentence embeddings from multi-lingual counterparts. However, we observe that these embeddings are not generic enough and do not work well on out of domain social media datasets. We consider two Marathi hate speech datasets L3Cube-MahaHate, HASOC-2021, a Marathi sentiment classification dataset L3Cube-MahaSent, and Marathi Headline, Articles classification datasets.